Schinckel.net

A recurring feature of #django is that someone asks about settings.py, and using a local_settings.py file. The standard advice is to have the following in your settings.py:

# More settings are above here.

try:
  from local_settings import *
except ImportError:
  pass

This is usually the last (or one of the last) things in the file. This can be used to override settings with sane values for the local environment.

However, this means that local_settings.py must be not in your source control system, or must not be deployed to other servers.

I like keeping everything in my source control system of choice (mercurial), and currently use an hg-based deployment. In my fabfile.py, instead of archiving up the current structure, I use hg to push the main repo, and any sub-repos, and update them to the version that is displayed locally.

This means I want to be able to control the content of production’s local_settings.py equivalent.

The other issue, and this was the one that came up today and gave me the idea of this post, is that someone wanted to add an app to settings.INSTALLED_APPS but only locally. I too have done this (still do, with django-devserver, amongst others).

I came up with the following solution. Instead of having a settings.py and local_settings.py, I have a settings module:

settings/
    __init__.py
    base.py
    development.py
    production.py
    testing.py

base.py contains what was normally in your main settings.py file. That is, settings that are common to all environments.

In development.py, production.py and testing.py, I have the following line at the top:

from base import *

Then, in each of those files, where I need to override or alter a setting, including appending to a list or tuple, I can just modify away. Some things that I do in development.py, for instance:

from base import *

DEBUG = True

DATABASES['default']['HOST'] = '127.0.0.1'

INSTALLED_APPS += (
  'devserver',
  'test_extensions',
  'test_utils' # really only for makfixture.
)

import getpass
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
EMAIL_FILE_PATH =  PROJECT_ROOT / 'log' / 'email-messages-%s' % getpass.getuser()

This shows how you can set a value, alter a value of a dict given a specific key, and append to a tuple. I also have a nice little setup where I use a value set in the base.py file (PROJECT_ROOT) to determine where I want to log email messages to.

Finally, you need some way to say which of these files should be used. This is all done in __init__.py:

servers = {
  'development': [
    'darwin', 'boyd', 'arne'
  ],
  'testing': [
    'testing', 'debian'
  ],
  'production': [
    'staging', 'vps1', 'vps2', 'vps3'
  ]
}

def get_server_type():
  from socket import gethostname
  server_name = gethostname()
  for server_type, names in servers.items():
    if server_name in names:
      return server_type
  
  return 'production' # Or whatever you want the default to be.
                      # I usually have 'testing' here, because I tend to
                      # spin up test servers. If you spun up production
                      # servers lots, you'd use that.

exec("from %s import *" % get_server_type())

This method does require a little bit of maintainence: when you have a new server name, you need to add an entry to this file. If you are often creating testing servers (like I am) then you might want to use testing as the default server type.

Alternatively, you could use some sort of prefix to mean a particular server type.

Anyway, that’s how I do it. The only drawback is that it does mean that your SECRET_KEY, and any passwords you might have defined in settings.py are stored in your repository. We aren’t that fussed about that right now: our project is closed source, and only trusted people have access to the repository.

I mentioned a week or so ago about my django templatetag that will only display a menuitem or link if the logged in user has access to the view that it points to. In passing, I stated how it was rather complicated to do this test.

The complicated bit is finding out all of the decorators that are wrapping a given function. Fortunately, python is a dynamic language, and this type of introspection, whilst not completely simple, is possible.

The key lies in the property that all function objects in python have: func_closure. According to the python docs:

func_closure is None or a tuple of cells that contain binding for the function’s free variables

Depending upon where you read this, it may or may not be writable. Luckily, we don’t need to be able to write to this, only read from it.

If func_closure is not None, then it will be a tuple of <cell> objects. To do anything useful, we’ll need to look at their cell_contents attribute. If that is callable, then it’s a good candidate for a decorator.

Because of the way decorators work, if you have multiple decorators on a function, each one wraps the next one. Thus, we’ll need to have some recursion in there.

At a first draft, we might end up with something like:

def get_decorators(function):
  # If we have no func_closure, it means we are not wrapping any other functions.
  if not function.func_closure:
    return [function]
  decorators = []
  # Otherwise, we want to collect all of the recursive results for every closure we have.
  for closure in function.func_closure:
    decorators.extend(get_decorators(closure.cell_contents))
  return [function] + decorators

It’s important that we return the original function in the base case of the recursive function, as eventually, every closure will fold down to a single callable. It has the side effect that get_decorators will get all of the decorators, and the function they ultimately wrap.

You could probably also do this as a generator.

For basic django function views, this will work fine. For class-based views, we need to do something a little extra.

In the case I was writing this function for, I knew it would only be looking for a get() method on the class-based view, which makes things a little simpler. That, and the dispatch() method were the only places I would need to look on the class for decorators. Also, I only wanted decorators that were callable: since I would actually call a subset of them to test if the user could access the view.

def get_callable_cells(function):
  callables = []
  # Under some circumstances, I wound up with an object that has the name `view_func`: 
  # this is the view function I need to access.
  if not hasattr(function, 'func_closure'):
    if hasattr(function, 'view_func'):
      return get_callable_cells(function.view_func)
  # If we have no more function we are wrapping
  if not function.func_closure:
    return [function]
  for closure in function.func_closure:
    contents = closure.cell_contents
    # Class-based views have a dispatch method
    if hasattr(contents, 'dispatch'):
      callables.extend(get_callable_cells(contents.dispatch.__func__))
      if hasattr(contents, 'get'):
        callables.extend(get_callable_cells(contents.get.__func__))
    callables.extend(get_callable_cells(contents))
  return [function] + callables

The other trick I’m using there is to use the __func__ property of the dispatch and get methods of the view class.

From there, in my case, I wanted those callable cells that look like they take a user as their first argument:

def get_tests(function):
  return [
    x for x in get_callable_cells(function)
    if x.func_code.co_varnames[0] in ["user", "u"]
  ]

So, this works for things that look like:

class ViewClass(ParentClass):
  @login_required
  @permission_required('app_label.permission_name')
  def dispatch(self, *args, **kwargs):
    return super(ViewClass, self).dispatch(*args, **kwargs)
  
  @user_passes_test(lambda u: u.is_staff)
  def get(self, *args, **kwargs):
    # do stuff here
    pass

As well as:

@login_required
@permission_required('app_label.permission_name')
@user_passes_test(lambda u: u.is_staff)
@render('foo/bar.html')
def view_function(request, *args, **kwargs):
  # do stuff here

It will pick up that the @render decorator is not asking for a user.

It works with the @user_passes_test decorator, because that is passed a function that has the first argument of u. It works with the @permission_required decorator for basically the same reason, although that has a function inside a function that actually has the argument of user.

Finally, it works for the @login_required decorator, as that calls user_passes_test.

For future reference:

1 def decode_dreamweaver_password(encoded):
2     output = ""
3     for i in range(0, len(encoded), 2):
4         val = int(data[i:i+1],16) - i/2
5         output += chr(val)
6     return output

Sometimes, the django.contrib.auth User model just doesn’t cut it.

I have bounced around between ways of handling this sorry fact. My production system uses a nasty system of Person–User relationships (where, due to old legacy code, I need to keep the primary keys in sync), to monkey-patching User, using UserProfiles, and subclassing User.

First, a little on the nasty hack I have in place (and how that will affect my choices later on).

My project in work is a rostering system, where not everyone who is a Person in the system needs to be a User. For instance, most managers (who are Users) do not need their staff to be able to log in. However, they themselves must be a Person as well as a User, if they are to be able to log in, but also be rostered on.

Thus, there are many people in the system who are not Users. They don’t have a username, and may not even have an email address. Not that having an email address is that useful in the django User model, as there is no unique constraint upon that.

So, I am currently kind-of using Person as a UserProfile object, but there are Person instances that do not have an associated User, and some of these are required to have an email address, and have first and last names. So, there is lots of duplication across these two tables. Which need to be kept in sync.

The solution I am looking at now moves in the other direction.

A Person is a subclass of User. It has the extra data that we need for our business logic (mobile phone number, company they work for), but I have also monkey-patched User to not require a username. We are moving towards using email addresses for login names anyway, so that isn’t a problem. That has its own concerns (not everyone has a unique email address, but there are workarounds for that).

But not every User will have a Person attached. The admin team’s logins will not (and this will be used to allow them to masquerade as another user for testing and bug-hunting purposes). So, we can’t just ignore the User class altogether and do everything with the Person class.

This is all well and good. I have an authentication backend that will return a Person object instead of a User object (if one matches the credentials). Things are looking good.

Except then I look in the admin interface. And there we have all of the Person objects' related User objects, in the User table. It would be nice if we only had the ‘pure’ Users in here, and all Person objects were just in their category.

So, I needed a way to filter this list.

Luckily, django’s admin has this capability. In my person/admin.py file, I had the following code:

1 from django.contrib import admin
2 from django.contrib import auth
3 
4 class UserAdmin(auth.admin.UserAdmin):
5     def queryset(self, request):
6         return super(UserAdmin, self).queryset(request).filter(person=None)
7 
8 admin.site.unregister(auth.models.User)
9 admin.site.register(auth.models.User, UserAdmin)

And, indeed, this works.

But then I found another User subclass. Now we needed a type of user that is distinct from Person (they are never rostered, are not associated with a given company, but do log into the system).

I wanted the changes to the admin to be isolated within the different apps, so I needed to be able to get the currently installed UserAdmin class, and subclass that to filter the queryset. So the code becomes (in both admin.py files):

 1 from django.contrib import admin
 2 from django.contrib import auth
 3 
 4 BaseUserAdmin = type(admin.site._registry[auth.models.User])
 5 
 6 class UserAdmin(BaseUserAdmin):
 7     def queryset(self, request):
 8         return super(UserAdmin, self).queryset(request).filter(foo=None)
 9 
10 admin.site.unregister(auth.models.User)
11 admin.site.register(auth.models.User, UserAdmin)

The only difference in the two files is the foo. This becomes whatever this sub-class’s name is. Thus, it is person in the person/admin.py file, and orguser in the orguser/admin.py file.

The next step is to change the backend so that it will automatically downcast the logged in user to their child class. Other people have detailed this in the past: mostly the performance issue vanishes here because we are only looking at a single database query for a single object.

TL;DR

1 $ pip install virtualenv
2 $ virtualenv /path/to/django_project/
3 $ . /path/to/django_project/bin/activate
4 $ pip install django

I hang around a fair bit in #django now on IRC. It’s open most of the time I am at work: if I am waiting for something to deploy, I’ll keep an eye out for someone that needs a hand, or whatever. Yesterday, I attempted to help someone out with an issue with django and apache: I ended up having to go home before it got sorted out.

One of the things that came up was how to actually install django. The person was following instructions on how to do so under Ubuntu, but they weren’t exactly ‘best practice’.

One of the things I wish I had been around when I first started developing using python is virtualenv. This tool allows you to isolate a python environment, and install stuff into it that will not affect other virtual environments, or the system python installation.

Unfortunately, it does not come standard with python. If it were part of the standard library, it may reduce the likelihood of someone not using it. The upside of it not being in the standard library is that it gets updated more frequently.

Installing virtualenv

First, see if virtualenv is installed:

1 $ virtualenv --version

If not, you’ll need to install it. You can install it using pip or easy_install, if you have either of those installed. If you are a super-user on your machine (ie, it is your computer), then you may want to use sudo. You can have it installed just in your user account, which you might need to do on a shared computer.

You’ll probably also want to install pip at the system level. I do this first, and use it to install virtualenv, fabric and other packages that I need to use outside of a virtualenv (mercurial springs to mind). Do note that a virtualenv contains an install of pip by default, so this is up to you: once you have virtualenv installed, you can use pip in every virtualenv to install packages.

Setting up a virtual environment

I recommend using virtualenv for both development and deployment.

I think I use virtualenv slightly differently to most other people. My project structure tends to look like:

 1 /home/user/development/<project-name>/
 2     bin/
 3     fabfile.py
 4     include/
 5     lib/python2.6/site-packages/...
 6     project/
 7         # Project-specific stuff goes here
 8     src/
 9         # pip install -e stuff goes here
10     tmp/

Thus, my $VIRTUAL_ENV is actually also my $PROJECT_ROOT. This means that everything is self contained. It has the negative side-effect of meaning if I clone my project, I need to install everything again. This is not such a bad thing, as I use Fabric to automate the setup and deployment processes. It takes a bit of time, but using a local pypi mirror makes if fairly painless.

Obviously, I ignore bin/, lib/ and the other virtualenv created directories in my source control.

However, since we are starting from scratch, we won’t have a fabfile.py to begin with, and we’ll just do stuff manually.

1 $ cd /location/to/develop
2 $ virtualenv my_django_project

That’s it. You now have a virtual environment.

Installing django/other python packages

You’ll want to activate your new virtualenv to install the stuff you will need:

1 $ cd my_django_project
2 $ . bin/activate
3 (my_django_project)$

Notice the prompt changes to show you are in a virtual environment.

Install the packages you need (from now on, I’ll assume your virtualenv is active):

1     $ pip install django

There has been some discussion about having packages like psycopg2 installed at the system level: I tend to install everything into the virtualenv.

So that’s it. You now have django installed in a virtual environment. I plan to write some more later about my deployment process, as well as how I structure my django projects.

I have a use case I am investigating where saving an object should mark the original object as deleted, and save a new copy.

Using django, this is suprisingly easy to achieve.

Here is a simple model that does just that:

 1 from django.db import models
 2 class AutoSave(models.Model):
 3     name = models.CharField(max_length=64)
 4     active = models.BooleanField(default=True)
 5     previous = models.ForeignKey('AutoSave', null=True, blank=True,
 6         related_name='subsequent')
 7 
 8     def save(self, *args, **kwargs):
 9         if self.pk:
10             self.previous_id = self.pk
11             AutoSave.objects.filter(pk=self.pk).update(active=False)
12             self.pk = None
13         super(AutoSave, self).save(*args, **kwargs)

The important bits are that there is a boolean field called active, which can be used to soft-delete, as well as to indicate that this is the most recent revision of an object. There is also a reference to the previous version.

When an object is saved, we look for an existing primary key. Django uses this to determine if we are doing an INSERT or UPDATE. We want to get a reference to the current revision, then update the database version of that revision to be inactive, and then ensure the current data will create a new database record.

This is obviously a bit of a simplification, but the only way I would change it might be to see if no fields have changed, in which case we don’t need to really save a new revision.

Listening to the new podcast by the ever-present Dan Benjamin and the effervescent Merlin Mann today reminded me of the one Mac OS X service I wrote, that I use almost daily.

It allows me to right-click on the filename line in a python traceback, and have the file opened at that line in Textmate. If the file is part of an already open project, it will open in the project window (unless it is open in another window, in which case that may pop to front).

Fairly simple stuff, and should be easy to extend for other traceback/output types.

https://bitbucket.org/schinckel/open-in-textmate-service

bpython is pretty cool. It gives you an improved python shell, with popups of completeable values. About the only drawback is that some command-line editing doesn’t work that well, but I can live with that.

I made a django app that provides a new management command: bshell. This will start a new shell, using bpython, and import all of your models.

You can get it with:

pip install django-bshell

And then install it into your django settings.INSTALLED apps. The app itself is called ‘bshell’. Then you can just use:

django-admin.py bshell

The code can be found on bitbucket.

Using TextMate, it is very easy to get snippets and commands to do things that you often do. However, the python bundle is a bit lacking, and this is a great opportunity to improve that.

I’ve created a Command that will enter a newline, and if not inside a list, function call, dictionary or multi-line string, automatically add a trailing .

I’ve hooked it in to the Enter key, and the other settings can be seen in the screenshot below:

The actual code follows:

 1 #!/usr/bin/env ruby
 2 
 3 scope = ENV['TM_SCOPE'].split
 4 
 5 no_trail = ['punctuation.definition.arguments.end.python',
 6             'meta.structure.list.python',
 7             'meta.structure.dictionary.python',
 8             'string.quoted.single.block.python',
 9             'string.quoted.double.block.python']
10 
11 print (scope & no_trail) == [] ? "\\n" : "\n"

Note for future reference: especially since the data migration script I was running today was running out of memory and crashing.

You can use slice notation, which limits the query:

>>> for p in db.query(Person)[:10]:
...   print p

This rocks.