Schinckel.net

Sometimes, the django.contrib.auth User model just doesn’t cut it.

I have bounced around between ways of handling this sorry fact. My production system uses a nasty system of Person-User relationships (where, due to old legacy code, I need to keep the primary keys in sync), to monkey-patching User, using UserProfiles, and subclassing User.

First, a little on the nasty hack I have in place (and how that will affect my choices later on).

My project in work is a rostering system, where not everyone who is a Person in the system needs to be a User. For instance, most managers (who are Users) do not need their staff to be able to log in. However, they themselves must be a Person as well as a User, if they are to be able to log in, but also be rostered on.

Thus, there are many people in the system who are not Users. They don’t have a username, and may not even have an email address. Not that having an email address is that useful in the django User model, as there is no unique constraint upon that.

So, I am currently kind-of using Person as a UserProfile object, but there are Person instances that do not have an associated User, and some of these are required to have an email address, and have first and last names. So, there is lots of duplication across these two tables. Which need to be kept in sync.

The solution I am looking at now moves in the other direction.

A Person is a subclass of User. It has the extra data that we need for our business logic (mobile phone number, company they work for), but I have also monkey-patched User to not require a username. We are moving towards using email addresses for login names anyway, so that isn’t a problem. That has its own concerns (not everyone has a unique email address, but there are workarounds for that).

But not every User will have a Person attached. The admin team’s logins will not (and this will be used to allow them to masquerade as another user for testing and bug-hunting purposes). So, we can’t just ignore the User class altogether and do everything with the Person class.

This is all well and good. I have an authentication backend that will return a Person object instead of a User object (if one matches the credentials). Things are looking good.

Except then I look in the admin interface. And there we have all of the Person objects’ related User objects, in the User table. It would be nice if we only had the ‘pure’ Users in here, and all Person objects were just in their category.

So, I needed a way to filter this list.

Luckily, django’s admin has this capability. In my person/admin.py file, I had the following code:

1
2
3
4
5
6
7
8
9

from django.contrib import admin
from django.contrib import auth

class UserAdmin(auth.admin.UserAdmin):
    def queryset(self, request):
        return super(UserAdmin, self).queryset(request).filter(person=None)

admin.site.unregister(auth.models.User)
admin.site.register(auth.models.User, UserAdmin)

And, indeed, this works.

But then I found another User subclass. Now we needed a type of user that is distinct from Person (they are never rostered, are not associated with a given company, but do log into the system).

I wanted the changes to the admin to be isolated within the different apps, so I needed to be able to get the currently installed UserAdmin class, and subclass that to filter the queryset. So the code becomes (in both admin.py files):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from django.contrib import admin
from django.contrib import auth

BaseUserAdmin = type(admin.site._registry[auth.models.User])

class UserAdmin(BaseUserAdmin):
    def queryset(self, request):
        return super(UserAdmin, self).queryset(request).filter(foo=None)

admin.site.unregister(auth.models.User)
admin.site.register(auth.models.User, UserAdmin)

The only difference in the two files is the foo. This becomes whatever this sub-class’s name is. Thus, it is person in the person/admin.py file, and orguser in the orguser/admin.py file.

The next step is to change the backend so that it will automatically downcast the logged in user to their child class. Other people have detailed this in the past: mostly the performance issue vanishes here because we are only looking at a single database query for a single object.

TL;DR

1
2
3
4

$ pip install virtualenv
$ virtualenv /path/to/django_project/
$ . /path/to/django_project/bin/activate
$ pip install django

I hang around a fair bit in #django now on IRC. It’s open most of the time I am at work: if I am waiting for something to deploy, I’ll keep an eye out for someone that needs a hand, or whatever. Yesterday, I attempted to help someone out with an issue with django and apache: I ended up having to go home before it got sorted out.

One of the things that came up was how to actually install django. The person was following instructions on how to do so under Ubuntu, but they weren’t exactly ‘best practice’.

One of the things I wish I had been around when I first started developing using python is virtualenv. This tool allows you to isolate a python environment, and install stuff into it that will not affect other virtual environments, or the system python installation.

Unfortunately, it does not come standard with python. If it were part of the standard library, it may reduce the likelihood of someone not using it. The upside of it not being in the standard library is that it gets updated more frequently.

Installing virtualenv

First, see if virtualenv is installed:

1

$ virtualenv --version

If not, you’ll need to install it. You can install it using pip or easy_install, if you have either of those installed. If you are a super-user on your machine (ie, it is your computer), then you may want to use sudo. You can have it installed just in your user account, which you might need to do on a shared computer.

You’ll probably also want to install pip at the system level. I do this first, and use it to install virtualenv, fabric and other packages that I need to use outside of a virtualenv (mercurial springs to mind). Do note that a virtualenv contains an install of pip by default, so this is up to you: once you have virtualenv installed, you can use pip in every virtualenv to install packages.

Setting up a virtual environment

I recommend using virtualenv for both development and deployment.

I think I use virtualenv slightly differently to most other people. My project structure tends to look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

/home/user/development/<project-name>/
    bin/
    fabfile.py
    include/
    lib/python2.6/site-packages/...
    project/
        # Project-specific stuff goes here
    src/
        # pip install -e stuff goes here
    tmp/

Thus, my $VIRTUAL_ENV is actually also my $PROJECT_ROOT. This means that everything is self contained. It has the negative side-effect of meaning if I clone my project, I need to install everything again. This is not such a bad thing, as I use Fabric to automate the setup and deployment processes. It takes a bit of time, but using a local pypi mirror makes if fairly painless.

Obviously, I ignore bin/, lib/ and the other virtualenv created directories in my source control.

However, since we are starting from scratch, we won’t have a fabfile.py to begin with, and we’ll just do stuff manually.

1
2

$ cd /location/to/develop
$ virtualenv my_django_project

That’s it. You now have a virtual environment.

Installing django/other python packages

You’ll want to activate your new virtualenv to install the stuff you will need:

1
2
3

$ cd my_django_project
$ . bin/activate
(my_django_project)$

Notice the prompt changes to show you are in a virtual environment.

Install the packages you need (from now on, I’ll assume your virtualenv is active):

1

    $ pip install django

There has been some discussion about having packages like psycopg2 installed at the system level: I tend to install everything into the virtualenv.

So that’s it. You now have django installed in a virtual environment. I plan to write some more later about my deployment process, as well as how I structure my django projects.

I have a use case I am investigating where saving an object should mark the original object as deleted, and save a new copy.

Using django, this is suprisingly easy to achieve.

Here is a simple model that does just that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from django.db import models
class AutoSave(models.Model):
    name = models.CharField(max_length=64)
    active = models.BooleanField(default=True)
    previous = models.ForeignKey('AutoSave', null=True, blank=True,
        related_name='subsequent')

    def save(self, *args, **kwargs):
        if self.pk:
            self.previous_id = self.pk
            AutoSave.objects.filter(pk=self.pk).update(active=False)
            self.pk = None
        super(AutoSave, self).save(*args, **kwargs)

The important bits are that there is a boolean field called active, which can be used to soft-delete, as well as to indicate that this is the most recent revision of an object. There is also a reference to the previous version.

When an object is saved, we look for an existing primary key. Django uses this to determine if we are doing an INSERT or UPDATE. We want to get a reference to the current revision, then update the database version of that revision to be inactive, and then ensure the current data will create a new database record.

This is obviously a bit of a simplification, but the only way I would change it might be to see if no fields have changed, in which case we don’t need to really save a new revision.

I develop a ‘closed’ system. It isn’t that not just anyone can use it - hell, we’d love to have more customers, but the users of the front-facing site (myROSS) are not the same as our customers, who use the backend indirectly through our client application.

The outcome of this is that we do not allow for registration on myROSS. The only way you can become a user of myROSS is if your company uses The ROSS System, and you have been added by your manager, franchisee or whatever.

However, I have been looking at django-socialregistration a bit lately: it is a very clean way to allow users to register and then login using Facebook, Twitter OAuth, and OpenID. Since one of the reasons that many of our ‘users’ do not use myROSS as much as we would like is because it is another password to remember, and also because most of our users are younger employees of Quick Service Restaurants, we are hoping that allowing them to use their Facebook or Twitter credentials will mean the barrier to use of myROSS is reduced.

So, I would like to be able to use socialregistration to allow currently registered users to connect their myROSS accounts to one or more of their other accounts (Twitter, Facebook, OpenID), and then allow them to use that as a means of authentication. But at the same time, I want to prevent non-current users from registering (or more to the point, prevent a current user from accidentally creating a duplicate account).

The first part is already handled by socialregistration: when a logged in user clicks on a link that connects to Twitter, for instance, and then approves the connection on the Twitter website, their account is then associated. Data is stored in the myROSS database noting which twitter account id matches which myROSS account. They can then log in using Twitter. I assume the same happens with Facebook (which I don’t use), and OpenID (which I haven’t tried yet - although I do use it for other sites).

What I need to do is prevent non-logged in users from registering.

The way socialregistration’s workflow works is that when a non-logged in user authenticates with the outside authentication system, they are redirected to a setup page, which allows them to either create a username, or generates a username for them. We need to just hook in at this time, and instead of asking for a new username/email address, ask them to enter their current myROSS credentials. At that point, we can store the information as is done if they were already authenticated.

Rather than hack the socialregistration source, we can use the hooks that the authors there have kindly left us. Specifically, we only need to override one class, the UserForm. We can do this in the view, by changing the arguments passed in to the ‘setup’ view.

If you copied-and-pasted the contents of the socialregistration urls.py file, then you can just modify the declaration of the ‘socialregistration_setup’ url route, and add in the new form class. Otherwise, you can just override that url route. I included the whole socialregistration.urls, so now the relevant section of my project urls.py looks like:

1
2
3

# routes for socialregistration: note the override of the first one.
(r'^auth/setup/$', 'socialregistration.views.setup', {'form_class':auth_additions.forms.ExistingUserForm}),
(r'^auth/', include('socialregistration.urls')),

There is also an import of my auth_additions.forms module.

Now, the exciting bit: what goes in the ExistingUserForm class?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

from django.contrib.auth.forms import authenticate
from django.utils.translation import gettext as _
from django import forms

class ExistingUserForm(forms.Form):
    username = forms.CharField()
    password = forms.CharField(label=_("Password"), widget=forms.PasswordInput)
    
    def __init__(self, user, profile, *args, **kwargs):
        super(ExistingUserForm, self).__init__(*args, **kwargs)
        self.user = user
        self.profile = profile
    
    def clean(self):
        username = self.cleaned_data.get('username')
        password = self.cleaned_data.get('password')

        if username and password:
            self.user = authenticate(username=username, password=password)
            if self.user is None:
                raise forms.ValidationError(_("Please enter a correct username and password. Note that both fields are case-sensitive."))
            elif not self.user.is_active:
                raise forms.ValidationError(_("This account is inactive."))

        return self.cleaned_data
        
    def save(self, request=None):
        self.profile.user = self.user
        self.profile.save()
        return self.user

Had an annoying issue today after upgrading an installation of django to the trunk. All of a sudden, my admin interface would not load. I had errors stating that the template admin/login.html could not be loaded.

Now, django.contrib.admin was in settings.INSTALLED_APPS, and django.template.loaders.app_directories.Loader was in settings.TEMPLATE_LOADERS.

So, why was django throwing an exception? To find out, I stepped into a shell:

>>> from django.template.loader import find_template_loader
>>> loader = find_template_loader('django.template.loaders.app_directories.Loader')
>>> loader.load_template('admin/login.html')

This was where I realised something was wrong. I don’t get the error now (as I have fixed it), but it complained about being not allowed to open the file. As in a permissions error.

Looking up the location in a new shell, I was able to see that all of the files in the django.contrib.admin.templates directory were only able to be read by root. For some reason, python setup.py install had set the mode of these files to 0600, instead of the expected 0644. A quick sudo chmod -r ag+r templates (from inside the django.contrib.admin directory) fixed it.

bpython is pretty cool. It gives you an improved python shell, with popups of completeable values. About the only drawback is that some command-line editing doesn’t work that well, but I can live with that.

I made a django app that provides a new management command: bshell. This will start a new shell, using bpython, and import all of your models.

You can get it with:

pip install django-bshell

And then install it into your django settings.INSTALLED apps. The app itself is called ‘bshell’. Then you can just use:

django-admin.py bshell

The code can be found on bitbucket.

Did some work on my Django Management.tmbundle last night.

It now handles running tests when (a) The apps are not directly in the project root, but inside another folder, for instance; and (b) the app/tests.py file has been split into seperate files.

The main reason I made this was so that I could run tests and have clickable links in the results window for the location of failing tests.

There is still much to do on this. I am considering re-writing it in python rather than ruby, so I can programmatically find the app name, rather than guess it. I also want to refactor the hell out of it and make it much nicer.

Anyway, if you are interested, you can find the most recent version at http://github.com/schinckel/Django-Management.tmbundle - and I think it also appears in TextMate’s getBundles bundle.

I’ve come to love Django, and now I seem to be spending not only my work time, but also my own time working on one django project or another. Today at work I came across the need to have check constraints in a database. For instance, this can be used to ensure that the start of an event is before the end of the event, or that a certain value has a particular range of values.

For the SQL savvy, it can look like:

CREATE TABLE event (
...more columns here...
start datetime,
finish datetime,
count integer CHECK (count >= 0),
CONSTRAINT event_starts_before_finish CHECK (start < finish)
);

There are two types of constraint here: the column-level constraint, which in this instance can be done with django, and the table-level constraint, which in many systems cannot be a part of a column definition. In PostgreSQL, my DBMS of choice, it is possible to define a constraint that affects more than one column as part of a column, but I prefer not to. Note also that I am using the syntax form that allows me to name the constraint. This means I can alter or drop it later.

As I mentioned, django can do the first type of constraint, as long as it is a > 0 constraint, using one of the field types that subclasses PositiveInteger. However, there is no functionality built into django to do the latter. And I would like to use them.

It is possible to just add the constraints to an already existing database: indeed, that is what I did for work. Having the constraints at the database level means that, since I have more than one interface to my datastore (don’t ask: one of them is for an old SOAP interface I need to keep around), I want to ensure that even if someone accesses it outside of django, they cannot put in data that breaks these constraints. Similarly, if I use an interface other than the admin interface, or heaven forbid, open up the database in raw form, I cannot accidentally put in data that breaks this validation.

But, pure database level constraints don’t give very nice feedback in django. It is nicer to have the pretty red boxes around my field than the traceback. So, I want the validation to occur on the field level as well. As long as I am using django’s forms (and my API for RESTful access will use them for validation), then I will have these errors nicely presented.

So, to that end, I have created some code that allows for the definition and enforcement of both of these types of constraint.

The column form allows for a new keyword argument to a field:

count = models.IntegerField(constraint=">= 0")

Notably, the string that can be passed in must conform to the pattern “cmp value”, where cmp is one of the comparison operators (<, >, <=, >=, =, !=), and value is a valid value for this column type. It will be passed to the to_python() method of this field when comparing. There must be a string between the two parts.

The other form is a new attribute on the Meta class of a model.

check_constraints = ('start < finish',)

This must be a tuple of strings, where a string is of the form “column cmp column”. Again, there must be a space either side of cmp, and each column name must be a valid column. Not meeting these criteria will result in a validation error.

From these definitions, the database constraints will be applied, and validation of forms will also occur.

I have deliberately made the constraints simple (ie, not callable objects) so that they can easily be converted to database constraints. For instance, they can effectively be transferred straight through to the database (with the addition of the column name in the case of the column constraint).

I am going to create a ticket in the django trac, and submit a patch. Guessing I should write up some test cases, though!

I am loving Django for web development. I didn’t have it set up to serve my (development) projects automatically until just now.

I had installed the Passenger Prefpane, which greatly simplifies the management of VirtualHost-based serving of sites, at least for Rails and other ruby-based frameworks. With a little work, you can use the same setup to serve Django projects.

Rather than re-detail the setup, I’ll just point you to the mod_passenger setup, and the Passenger Prefpane setup pages.

Now, to set up a Django project: obviously you need a django project. Create one, and note where it is located. I stick all of mine in ~/Sites.

Add a file to the root of this project, called passenger_wsgi.py. It needs to contain the following data:

1
2
3
4
5

import os, sys
sys.path.append('/Users/matt/Sites') # Replace with your directory
os.environ['DJANGO_SETTINGS_MODULE'] = 'testing.settings' # replace with your projectname.
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Now, add the site to the Passenger prefpane. My site is the testing.local site:

Now visit the address and ensure that it works. You should get the basic you need to set up django message.

To get the admin media served by the standard apache setup, I created a link inside the /Library/WebServer/Documents directory to /Library/Python/2.5/site-packages/django/contrib/admin/media. This can be done inside the Terminal:

sudo ln -s /Library/Python/2.5/site-packages/django/contrib/admin/media /Library/WebServer/Documents/admin-media/

Then, change the ADMIN_MEDIA setting in your django projects to http://localhost/admin-media/. This is probably the weakest point in the setup, as it will only work for pages served to your machine, not others on your network.

1
2
3
4

# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = 'http://localhost/admin-media/'

I had some issues with mod_passenger serving the data for localhost (and arne, my actual machine’s name) from the first installed VirtualHost. To overcome this, I put in a new file into /etc/apache2/other/localhost.conf, which looks like:

1
2
3
4
5
6
7

<VirtualHost *:80>
  DocumentRoot "/Library/WebServer/Documents"
  <directory "/Library/WebServer/Documents">
    Order allow,deny
    Allow from all
  </directory>
</VirtualHost>

This forces unnamed, or other sites to work as intended. Including the /User/*/Sites directories.

For some time, the git repository I am using to track changes for my Django-Management TextMate bundle was not working correctly in Redmine. I was able to connect it up using the settings, but was getting errors about the files not being found.

It turns out that git was not in my path. Putting a link from /usr/local/bin/git to /usr/bin/git fixed that all up.