Schinckel.net

Sometimes, the django.contrib.auth User model just doesn’t cut it.

I have bounced around between ways of handling this sorry fact. My production system uses a nasty system of Person-User relationships (where, due to old legacy code, I need to keep the primary keys in sync), to monkey-patching User, using UserProfiles, and subclassing User.

First, a little on the nasty hack I have in place (and how that will affect my choices later on).

My project in work is a rostering system, where not everyone who is a Person in the system needs to be a User. For instance, most managers (who are Users) do not need their staff to be able to log in. However, they themselves must be a Person as well as a User, if they are to be able to log in, but also be rostered on.

Thus, there are many people in the system who are not Users. They don’t have a username, and may not even have an email address. Not that having an email address is that useful in the django User model, as there is no unique constraint upon that.

So, I am currently kind-of using Person as a UserProfile object, but there are Person instances that do not have an associated User, and some of these are required to have an email address, and have first and last names. So, there is lots of duplication across these two tables. Which need to be kept in sync.

The solution I am looking at now moves in the other direction.

A Person is a subclass of User. It has the extra data that we need for our business logic (mobile phone number, company they work for), but I have also monkey-patched User to not require a username. We are moving towards using email addresses for login names anyway, so that isn’t a problem. That has its own concerns (not everyone has a unique email address, but there are workarounds for that).

But not every User will have a Person attached. The admin team’s logins will not (and this will be used to allow them to masquerade as another user for testing and bug-hunting purposes). So, we can’t just ignore the User class altogether and do everything with the Person class.

This is all well and good. I have an authentication backend that will return a Person object instead of a User object (if one matches the credentials). Things are looking good.

Except then I look in the admin interface. And there we have all of the Person objects’ related User objects, in the User table. It would be nice if we only had the ‘pure’ Users in here, and all Person objects were just in their category.

So, I needed a way to filter this list.

Luckily, django’s admin has this capability. In my person/admin.py file, I had the following code:

1
2
3
4
5
6
7
8
9

from django.contrib import admin
from django.contrib import auth

class UserAdmin(auth.admin.UserAdmin):
    def queryset(self, request):
        return super(UserAdmin, self).queryset(request).filter(person=None)

admin.site.unregister(auth.models.User)
admin.site.register(auth.models.User, UserAdmin)

And, indeed, this works.

But then I found another User subclass. Now we needed a type of user that is distinct from Person (they are never rostered, are not associated with a given company, but do log into the system).

I wanted the changes to the admin to be isolated within the different apps, so I needed to be able to get the currently installed UserAdmin class, and subclass that to filter the queryset. So the code becomes (in both admin.py files):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from django.contrib import admin
from django.contrib import auth

BaseUserAdmin = type(admin.site._registry[auth.models.User])

class UserAdmin(BaseUserAdmin):
    def queryset(self, request):
        return super(UserAdmin, self).queryset(request).filter(foo=None)

admin.site.unregister(auth.models.User)
admin.site.register(auth.models.User, UserAdmin)

The only difference in the two files is the foo. This becomes whatever this sub-class’s name is. Thus, it is person in the person/admin.py file, and orguser in the orguser/admin.py file.

The next step is to change the backend so that it will automatically downcast the logged in user to their child class. Other people have detailed this in the past: mostly the performance issue vanishes here because we are only looking at a single database query for a single object.

TL;DR

1
2
3
4

$ pip install virtualenv
$ virtualenv /path/to/django_project/
$ . /path/to/django_project/bin/activate
$ pip install django

I hang around a fair bit in #django now on IRC. It’s open most of the time I am at work: if I am waiting for something to deploy, I’ll keep an eye out for someone that needs a hand, or whatever. Yesterday, I attempted to help someone out with an issue with django and apache: I ended up having to go home before it got sorted out.

One of the things that came up was how to actually install django. The person was following instructions on how to do so under Ubuntu, but they weren’t exactly ‘best practice’.

One of the things I wish I had been around when I first started developing using python is virtualenv. This tool allows you to isolate a python environment, and install stuff into it that will not affect other virtual environments, or the system python installation.

Unfortunately, it does not come standard with python. If it were part of the standard library, it may reduce the likelihood of someone not using it. The upside of it not being in the standard library is that it gets updated more frequently.

Installing virtualenv

First, see if virtualenv is installed:

1

$ virtualenv --version

If not, you’ll need to install it. You can install it using pip or easy_install, if you have either of those installed. If you are a super-user on your machine (ie, it is your computer), then you may want to use sudo. You can have it installed just in your user account, which you might need to do on a shared computer.

You’ll probably also want to install pip at the system level. I do this first, and use it to install virtualenv, fabric and other packages that I need to use outside of a virtualenv (mercurial springs to mind). Do note that a virtualenv contains an install of pip by default, so this is up to you: once you have virtualenv installed, you can use pip in every virtualenv to install packages.

Setting up a virtual environment

I recommend using virtualenv for both development and deployment.

I think I use virtualenv slightly differently to most other people. My project structure tends to look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

/home/user/development/<project-name>/
    bin/
    fabfile.py
    include/
    lib/python2.6/site-packages/...
    project/
        # Project-specific stuff goes here
    src/
        # pip install -e stuff goes here
    tmp/

Thus, my $VIRTUAL_ENV is actually also my $PROJECT_ROOT. This means that everything is self contained. It has the negative side-effect of meaning if I clone my project, I need to install everything again. This is not such a bad thing, as I use Fabric to automate the setup and deployment processes. It takes a bit of time, but using a local pypi mirror makes if fairly painless.

Obviously, I ignore bin/, lib/ and the other virtualenv created directories in my source control.

However, since we are starting from scratch, we won’t have a fabfile.py to begin with, and we’ll just do stuff manually.

1
2

$ cd /location/to/develop
$ virtualenv my_django_project

That’s it. You now have a virtual environment.

Installing django/other python packages

You’ll want to activate your new virtualenv to install the stuff you will need:

1
2
3

$ cd my_django_project
$ . bin/activate
(my_django_project)$

Notice the prompt changes to show you are in a virtual environment.

Install the packages you need (from now on, I’ll assume your virtualenv is active):

1

    $ pip install django

There has been some discussion about having packages like psycopg2 installed at the system level: I tend to install everything into the virtualenv.

So that’s it. You now have django installed in a virtual environment. I plan to write some more later about my deployment process, as well as how I structure my django projects.

I have a use case I am investigating where saving an object should mark the original object as deleted, and save a new copy.

Using django, this is suprisingly easy to achieve.

Here is a simple model that does just that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from django.db import models
class AutoSave(models.Model):
    name = models.CharField(max_length=64)
    active = models.BooleanField(default=True)
    previous = models.ForeignKey('AutoSave', null=True, blank=True,
        related_name='subsequent')

    def save(self, *args, **kwargs):
        if self.pk:
            self.previous_id = self.pk
            AutoSave.objects.filter(pk=self.pk).update(active=False)
            self.pk = None
        super(AutoSave, self).save(*args, **kwargs)

The important bits are that there is a boolean field called active, which can be used to soft-delete, as well as to indicate that this is the most recent revision of an object. There is also a reference to the previous version.

When an object is saved, we look for an existing primary key. Django uses this to determine if we are doing an INSERT or UPDATE. We want to get a reference to the current revision, then update the database version of that revision to be inactive, and then ensure the current data will create a new database record.

This is obviously a bit of a simplification, but the only way I would change it might be to see if no fields have changed, in which case we don’t need to really save a new revision.

Listening to the new podcast by the ever-present Dan Benjamin and the effervescent Merlin Mann today reminded me of the one Mac OS X service I wrote, that I use almost daily.

It allows me to right-click on the filename line in a python traceback, and have the file opened at that line in Textmate. If the file is part of an already open project, it will open in the project window (unless it is open in another window, in which case that may pop to front).

Fairly simple stuff, and should be easy to extend for other traceback/output types.

https://bitbucket.org/schinckel/open-in-textmate-service

bpython is pretty cool. It gives you an improved python shell, with popups of completeable values. About the only drawback is that some command-line editing doesn’t work that well, but I can live with that.

I made a django app that provides a new management command: bshell. This will start a new shell, using bpython, and import all of your models.

You can get it with:

pip install django-bshell

And then install it into your django settings.INSTALLED apps. The app itself is called ‘bshell’. Then you can just use:

django-admin.py bshell

The code can be found on bitbucket.

Using TextMate, it is very easy to get snippets and commands to do things that you often do. However, the python bundle is a bit lacking, and this is a great opportunity to improve that.

I’ve created a Command that will enter a newline, and if not inside a list, function call, dictionary or multi-line string, automatically add a trailing .

I’ve hooked it in to the Enter key, and the other settings can be seen in the screenshot below:

The actual code follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/usr/bin/env ruby

scope = ENV['TM_SCOPE'].split

no_trail = ['punctuation.definition.arguments.end.python',
            'meta.structure.list.python',
            'meta.structure.dictionary.python',
            'string.quoted.single.block.python',
            'string.quoted.double.block.python']

print (scope & no_trail) == [] ? "\\\n" : "\n"

Note for future reference: especially since the data migration script I was running today was running out of memory and crashing.

You can use slice notation, which limits the query:

>>> for p in db.query(Person)[:10]:
...   print p

This rocks.

I’ve had cause to transfer a whole stack of data from an old sqlite database to a PostgreSQL database, and since on the new database I am using SQL Alchemy, I used SA to do the transfer.

Because of some of the relations, I have had to keep IDs constant across some of the columns. This is fine, but because they have Sequence objects associated with them, these are not kept up to date with the ‘custom’ created IDs. Thus, when attempting to create a new row in the table, it often fails, since it is trying to add a primary key that already exists.

After a little bit of research, I discovered it is possible to force the sequence object to increment, using the command:

nextID = engine.execute(Sequence('sequence_name'))

Using a little bit of magic, we can find out the largest index currently in use:

maxID = db.query(Object).order_by('-id').first().id

And a simple while loop will keep incrementing until we have reached the correct id.

Three languages. Three different Object Relational Mapping systems. One operating system.

Over the past couple of days, I’ve been madly learning how to create programs in Core Data, using Objective C and Cocoa. It’s given me plenty of food for thought, and made me perhaps think that it’s not that SQL Alchemy rocks my world, it’s just that J2EE/EJB is just ORM done wrong.

I actually got exposure to SQL Alchemy in detail before J2EE - I first came across them at the same time, but using SQL Alchemy was at work, and I basically had to learn in 2 days what a full semester worth of J2EE taught me. Perhaps that was just because learning the python stuff was so damn easy.

Now that I understand just how cool an Object Relational Mapping is, getting into Core Data was easy. Creating the same schema in SQL Alchemy (actually, using Elixir, so it was just object creation), and then in a Core Data xcdatamodel - basically the same process, was even simpler in Core Data: simply because it is a GUI tool, and you can see the whole model in one go, instead of having to scroll through a text file examining classes.

But doing GUI programming using the ORM is where Core Data really shines. Using Cocoa Bindings, you can just plonk down an NSTableView, and tell the object where it gets it’s data from. If you have two GUI widgets using the same data model, and you change selection in one, it even changes the selection in the other!

Core Data also helps look after Undo, Saving and all other sorts of goodness I haven’t even come across yet.

The only thing that Core Data isn’t good for is a multi-user system - or more precisely, a system where multiple users are accessing the data at the same time. I’ve used SOAP as the messenger format, where I had a rich client accessing services provided by my server, but this is cumbersome. I’ll just use Pylons, or perhaps Django to do a web application - where the interface is largely a Web Browser. In this instance, it will probably be best to just stick to a python-based approach. I’m tempted by WebObjects, but that would still require me to use Java.

If only Apple would release a distributed Core Data. I might be able to do something kind of cool with Distributed Objects, for a rich client, at least, but for Web, I may as well do the whole thing in python.

One of the criticisms of many languages is that they are so complex, that people do not use all of the features of them. Meaning that one person might write a program in a language that another, fluent reader/writer of that language may not be able to understand. I think this is just rubbish, at least to some respect. If you see some code in a language that doesn’t make sense, then hopefully you should be able to either figure it out, or look up in the documentation (what, your language doesn’t have complete online, searchable documentation?) what is going on.

A similar thing happened to me the other day - and I didn’t even need to look up in the Python documentation to see what the story was.

People often criticise python because it lacks the ability to mark a variable as private, or protected. All attributes of a class are automatically public, and it is only really a guideline that if you prefix a variable with one or two underscores, you are marking it as protected or private. Namespace mangling means that this becomes slightly more than just a guideline, IIRC, although I’m not really sure of how this all works. Basically, the theory is that if you prefix an attribute, that is a marker to users of your API that you really shouldn’t use this.

I’ve often been frustrated when writing Java programs that attributes aren’t accessible, but I can see the value in only allowing protected access, through getters and setters. You can do this to prevent user classes/objects from putting garbage data into your attributes. This becomes even more important in Python, as its dynamic (but still strong) typing means I could put a “GOO” in where you expected an integer.

I discovered yesterday, as I was implementing a database in SQL Alchemy, that python has the ability to only allow protected access, through getters and setters, using the property function.

So, for this project, I have a database, where Sheep objects are stored, along with a variety of attributes. Sheep can be either Rams or Ewes, but the data that is stored about a Ram is virtually identical as that stored about a Ewe. However, using polymorphic mapping, I can have Ram and Ewe objects both stored in the Sheep table, since they are both sub-classes. Furthermore, I can use the gender attribute to determine if the object retrieved from the database should be a Ram or a Ewe. This is much better than anything I was able to do with J2EE, at least as far as I could find.

Because I am using a composite primary key (again, not something that is easy to do with J2EE, without defining a Primary Key class, ugh), and a Sheep can have a .sire and a .dam, which internally is stored with three columns in the table each, and that a sire/dam must be older than the sheep in question (and of particular gender), then some form of restricting which data can be assigned to the sire and dam attributes (which, using SQL Alchemy are references to other objects in the table).

Thus, I can have the following lines in the class definition for Sheep:

1
2

        sire = property(_getSire, _setSire)
        dam = property(_getDam, _setDam)

The crux of this post is that now, if you attempt to set a sire, using: sheep.sire = otherSheep; then it will pass the contents of otherSheep to the _setSire method of the Sheep class.

This goes even further - it now allows me to either use a Ram object, or a Sheep object (and uses some checking to ensure that it is a Gender=M), or even just a string that is the string representation of a sheep - the value that I am using for the three parts of the primary key. It will then look up the sheep in question, or create a new one if it doesn’t exist in the database.

Thus, a sheep is identified to the real world by three things: it’s flock number, such as 160188, the year it was born, like 2004, and the tag number, such as 040001, which is not unique between flocks, or necessarily years. For the flock this system is designed for, the tag number is unique for sheep of a given century, which would generally suffice, but in a situation where there were more than 9999 sheep born every year in a flock, then you wouldn’t just be able to use the last two digits of the year for the start of the tag number.

Thus a sheep might be declared as 160188-2007-070001, which is a unique identifier of that sheep in the world (or at least Australia!), but having this as a primary key on it’s own means more jigglery would be required to get the flock number and year. By storing as three seperate parts it is possible to keep an easy reference to the flock, and the year. There is no chance of a data integrity error as there might be if I stored the id, then the year and flock number. It would be possible then to change one without the others being udpated.

I’ve also done something quite clever with the mapping of the sire/dam relationship. By having different classes for Ram and Ewe, and mapping sire/dam to these, it is possible to use the same backreference term, offspring, to generate a list of sheep who are offspring of the Ram or Ewe in question. If you don’t use subclasses, then it isn’t nearly as neat.

•

If you can’t tell, I’m loving SQL Alchemy. I had some issues figuring out what the hell was going wrong with my relations between sheep and sire, but it turned out that you have to use the remote_side argument to the relation function to do what I wanted. Funnily, it was working with dam, but not with sire, but now it’s much more solid. No more circular reference constraints either.

By the way, this project is going to be an online database for livestock information, allowing potential purchasers to research pedigrees online. If anyone else is interested in purchasing it, send me an email. At this stage, it is sheep only, but will be easy to change to, say cattle if they are your thing…