Multi-table Inheritance and the Django Admin

Django’s admin interface is a great way to be able to interact with your models without having to write any view code, and, within limits, it’s useful in production too. However, it can quickly get very crowded when you register lots of models.

Consider the situation where you are using Django’s multi-table inheritance:

from django.db import models

from model_utils.managers import InheritanceManager

class Sheep(models.Model):
    sheep_id = models.AutoField(primary_key=True)
    tag_id = models.CharField(max_length=32)
    date_of_birth = models.DateField()
    sire = models.ForeignKey('sheep.Ram', blank=True, null=True, related_name='progeny')
    dam = models.ForeignKey('sheep.Ewe', blank=True, null=True, related_name='progeny')

    objects = InheritanceManager()

    class Meta:
        verbose_name_plural = 'sheep'

    def __str__(self):
        return '{}: {}'.format(self._meta.verbose_name, self.tag_id)


class Ram(Sheep):
    sheep = models.OneToOneField(parent_link=True)

    class Meta:
        verbose_name = 'ram'
        verbose_name_plural = 'rams'


class Ewe(Sheep):
    sheep = models.OneToOneField(parent_link=True)

    class Meta:
        verbose_name = 'ewe'
        verbose_name_plural = 'ewes'

Ignore the fact there is no specialisation on those child models: in practice you’d normally have some.

Also note that I’ve manually included the primary key, and the parent link fields. This has been done so that the actual columns in the database match, and in this case will all be sheep_id. This will make writing joins slightly simpler, and avoids the (not specific to Django) ORM anti-pattern of “always have a column named id”.

We can use the models like this, but it might be nice to have all sheep in the one admin changelist, and just allow filtering by subclass model.

First, we’ll put some extra stuff onto the parent model, to make obtaining the subclasses simpler. Some of these will use a new decorator, which creates a class version of the @property decorator.

class classproperty(property):
    def __get__(self, cls, owner):
      return self.fget.__get__(None, owner)()


class Sheep(models.Model):
    # Fields, etc. defined as above.

    @classproperty
    @classmethod
    def SUBCLASS_OBJECT_CHOICES(cls):
        "All known subclasses, keyed by a unique name per class."
        return {
          rel.name: rel.related_model
          for rel in cls._meta.related_objects
          if rel.parent_link
        }

    @classproperty
    @classmethod
    def SUBCLASS_CHOICES(cls):
        "Available subclass choices, with nice names."
        return [
            (name, model._meta.verbose_name)
            for name, model in cls.SUBCLASS_OBJECT_CHOICES.items()
        ]

    @classmethod
    def SUBCLASS(cls, name):
        "Given a subclass name, return the subclass."
        return cls.SUBCLASS_OBJECT_CHOICES.get(name, cls)

Note that we don’t need to enumerate the subclasses: adding a new subclass later in development will automatically add it to these properties, even though in this case it would be unlikely to happen.

From these, we can write some nice neat stuff to enable using these in the admin.

from django import forms
from django.conf.urls import url
from django.contrib import admin
from django.utils.translation import ugettext as _

from .models import Sheep


class SubclassFilter(admin.SimpleListFilter):
    title = _('gender')
    parameter_name = 'gender'

    def lookups(self, request, model_admin):
      return Sheep.SUBCLASS_CHOICES

    def queryset(self, request, queryset):
      if self.value():
        return queryset.exclude(**{self.value(): None})
      return queryset


@admin.register(Sheep)
class SheepAdmin(admin.ModelAdmin):
    list_display = [
        'tag_id',
        'date_of_birth',
        'gender'
    ]
    list_filter = [SubclassFilter]

    def get_queryset(self, request):
      return super(SheepAdmin, self).get_queryset(request).select_subclasses()

    def gender(self, obj):
        return obj._meta.verbose_name

    def get_form(self, request, obj=None, **kwargs):
        if obj is None:
            Model = Sheep.SUBCLASS(request.GET.get('gender'))
        else:
            Model = obj.__class__

        # When we change the selected gender in the create form, we want to reload the page.
        RELOAD_PAGE = "window.location.search='?gender=' + this.value"
        # We should also grab all existing field values, and pass them as query string values.

        class ModelForm(forms.ModelForm):
            if not obj:
                gender = forms.ChoiceField(
                    choices=[('', _('Please select...'))] + Sheep.SUBCLASS_CHOICES,
                    widget=forms.Select(attrs={'onchange': RELOAD_PAGE})
                )

            class Meta:
                model = Model
                exclude = ()

        return ModelForm

    def get_fields(self, request, obj=None):
        # We want gender to be the first field.
        fields = super(SheepAdmin, self).get_fields(request, obj)

        if 'gender' in fields:
            fields.remove('gender')
            fields = ['gender'] + fields

        return fields

    def get_urls(self):
        # We want to install named urls that match the subclass ones, but bounce to the relevant
        # superclass ones (since they should be able to handle rendering the correct form).
        urls = super(SheepAdmin, self).get_urls()
        existing = '{}_{}_'.format(self.model._meta.app_label, self.model._meta.model_name)
        subclass_urls = []
        for name, model in Sheep.SUBCLASS_OBJECT_CHOICES.items():
            opts = model._meta
            replace = '{}_{}_'.format(opts.app_label, opts.model_name)
            subclass_urls.extend([
                url(pattern.regex.pattern, pattern.callback, name=pattern.name.replace(existing, replace))
                for pattern in urls if pattern.name
            ])

        return urls + subclass_urls

Wow. There’s quite a lot going on there, but the summary is:

  • We create a custom filter that filters according to subclass.
  • The .select_subclasses() means that objects are downcast to their subclass when fetched.
  • There is a custom form, that, when in create mode, has a selector for the desired subclass.
  • When the subclass is changed (only on the create form), the page is reloaded. This is required in a situation where there are different fields on each of the subclass models.
  • We register the subclass admin url paths, but use the superclass admin views.

I’ve had ideas about this for some time, and have just started using something like this in development: in my situation, there will be an arbitrary number of subclasses, all of which will have several new fields. The code in this page is extracted (and changed) from those ideas, so may not be completely correct. Corrections welcome.

Adding data to admin templates

It came up in the #django IRC channel the other day about how to extend a django admin template to show other information, possibly related to an object, but not necessarily editable.

I use this in production: we have a Company object, which has Location objects associated with it. The django validation is stricter than the data may have been created for these objects, so from time to time a field is missing, and the django admin will not allow saving it.

So, I wanted to be able to display some information about each related object, with links to various bits and pieces. Having the inline Location data is great, except for when it is missing something, that we may not have received from the customer yet.

The trick is that you’ll need to override the admin template for that model.

In this case, our class is in app_name.ModelName, so we need to put the following structure into our template directory:

    templates/
      admin/
        app_name/
          modelname/
            change_form.html

Within that file, I have the content (spaces between % and {,} are there because I can’t remember how to escape them in Liquid Templates…):

{ % extends "admin/change_form.html" % }
{ % block after_related_objects % }
  ... the extra stuff is here ...
{ % endblock % }

In my case, I have the following html structure, and it looks nice:

<div class="inline-group">
  <h2>Units</h2>
  <table width="100%">
    <thead>
      <tr>
        <th>Name</th>
        ...
      </tr>
    </thead>
    <tbody>
      ... loop through stuff here ...
    </tbody>
  </table>
</div>

The other trick is that the admin change view gives us an object, called original, which we can use to do lookups on related objects and the like.

The django admin is awesomesauce, and does most of what I need an administration interface to do. There are lots of places where you do need to extend it, and this is just one way of doing that.

Filtering querysets in django.contrib.admin forms

I make extensive use of the django admin interface. It is the primary tool for our support team to look at user data for our product, and I have stretched it in many ways to suit my needs.

One problem I often come back to is a need to filter querysets in forms and formsets. Specifically, the objects that should be presented to the admin user in a relationship to the currently viewed object should be filtered. In most cases, this is something as simple as making sure the Person and the Units they work at are within the same company.

There is a simple bit of boilerplate that can do this. You need to create a custom form, and attach this to the ModelAdmin for the parent object:

 1 from django.contrib import admin
 2 from django import forms
 3 from models import Person, Unit
 4 
 5 class PersonAdminForm(forms.ModelForm):
 6     class Meta:
 7         model = Person
 8     
 9     def __init__(self, *args, **kwargs):
10         super(PersonAdminForm, self).__init__(*args, **kwargs)
11         # This is the bit that matters:
12         self.fields['units'].queryset = self.instance.company.units
13 
14 class PersonAdmin(admin.ModelAdmin):
15     form = PersonAdminForm

In actuality, it is a little more complicated than this: you need to test if the selected object has a company, and really, if the user has changed the company (or selected it on a new person), you should use that instead. So the code looks a bit more like:

 1 company = None
 2 if self.data.get('company', None):
 3     try:
 4         company = Company.objects.get(pk=self.data['company'])
 5     except Company.DoesNotExist:
 6         pass
 7 else:
 8     try:
 9         company = self.instance.company
10     except Company.DoesNotExist:
11         pass
12 if company:
13     self.fields['units'].queryset = company.units.all()

Now, having to write all of that every time you have to filter the choices available wears rather thin. And wait until you need to do it to a formset instead: you need to also do stuff to the empty_form, so that when you dynamically add an inline form, it has the correct choices.

Enter FilteringForm, and her niece FilteringFormSet:

 1 from django import forms
 2 from django.core.exceptions import ObjectDoesNotExist
 3 
 4 class FilterMixin(object):
 5     filters = {}
 6     instance_filters = {}
 7     def apply_filters(self, forms=None):
 8         # If we didn't get a forms argument, we apply to ourself.
 9         if forms is None:
10             forms = [self]
11         # We need to apply instance filters first, as they allow us to
12         # select an attribute on our instance to be the queryset, and
13         # then apply a filter onto that with filters.
14         for field, attr in self.instance_filters.iteritems():
15             # It may be using a related attribute. person.company.units
16             tokens = attr.split('.')
17             
18             source = None
19             # See if there is any incoming data first.
20             if self.data.get(tokens[0], ''):
21                 try:
22                     source = self.instance._meta.get_field_by_name(tokens[0])[0].rel.to.objects.get(pk=self.data[tokens[0]])
23                 except ObjectDoesNotExist:
24                     pass
25             # Else, look for a match on the object we already have stored
26             if not source:
27                 try:
28                     source = getattr(self.instance, tokens[0])
29                 except ObjectDoesNotExist:
30                     pass
31             
32             # Now, look for child attributes.
33             if source:
34                 for segment in tokens[1:]:
35                     source = getattr(source, segment)
36                 if forms:
37                     for form in forms:
38                         form.fields[field].queryset = source
39         
40         # We can now apply any simple filters to the queryset.
41         for field, q_filter in self.filters.iteritems():
42             for form in forms:
43                 form.fields[field].queryset = form.fields[field].queryset.filter(q_filter)
44     
45 
46 class FilteringForm(forms.ModelForm, FilterMixin):
47     def __init__(self, *args, **kwargs):
48         super(FilteringForm, self).__init__(*args, **kwargs)
49         self.apply_filters()
50 
51 class FilteringFormSet(forms.models.BaseInlineFormSet, FilterMixin):
52     filters = {}
53     instance_filters = {}
54     
55     def __init__(self, *args, **kwargs):
56         super(FilteringFormSet, self).__init__(*args, **kwargs)
57         self.apply_filters(self.forms)
58     
59     def _get_empty_form(self, **kwargs):
60         form = super(FilteringFormSet, self)._get_empty_form(**kwargs)
61         self.apply_filters([form])
62         return form
63     empty_form = property(_get_empty_form)

Now, to use all of this, you still need to subclass, but you can declare the filters:

1 class PersonAdminForm(FilteringForm):
2     class Meta:
3         model = Person
4     
5     instance_filters = {
6         'units': 'company.units'
7     }

You can also have non-instance filters, and they will be applied after the instance_filters:

 1 from django.db import models
 2 
 3 class PersonAdminForm(FilteringForm):
 4     class Meta:
 5         model = Person
 6     
 7     instance_filters = {
 8         'units': 'company.units'
 9     }
10     filters = {
11         'units': models.Q(is_active=True)
12     }

I think it might be nice to be able to add an extra set of filtering for the empty form in a formset, so you could make it that only choices that hadn’t already been selected, for instance, were the only ones available. But that isn’t an issue for me right now.

Displaying only objects without subclasses

Sometimes, the django.contrib.auth User model just doesn’t cut it.

I have bounced around between ways of handling this sorry fact. My production system uses a nasty system of Person-User relationships (where, due to old legacy code, I need to keep the primary keys in sync), to monkey-patching User, using UserProfiles, and subclassing User.

First, a little on the nasty hack I have in place (and how that will affect my choices later on).

My project in work is a rostering system, where not everyone who is a Person in the system needs to be a User. For instance, most managers (who are Users) do not need their staff to be able to log in. However, they themselves must be a Person as well as a User, if they are to be able to log in, but also be rostered on.

Thus, there are many people in the system who are not Users. They don’t have a username, and may not even have an email address. Not that having an email address is that useful in the django User model, as there is no unique constraint upon that.

So, I am currently kind-of using Person as a UserProfile object, but there are Person instances that do not have an associated User, and some of these are required to have an email address, and have first and last names. So, there is lots of duplication across these two tables. Which need to be kept in sync.

The solution I am looking at now moves in the other direction.

A Person is a subclass of User. It has the extra data that we need for our business logic (mobile phone number, company they work for), but I have also monkey-patched User to not require a username. We are moving towards using email addresses for login names anyway, so that isn’t a problem. That has its own concerns (not everyone has a unique email address, but there are workarounds for that).

But not every User will have a Person attached. The admin team’s logins will not (and this will be used to allow them to masquerade as another user for testing and bug-hunting purposes). So, we can’t just ignore the User class altogether and do everything with the Person class.

This is all well and good. I have an authentication backend that will return a Person object instead of a User object (if one matches the credentials). Things are looking good.

Except then I look in the admin interface. And there we have all of the Person objects’ related User objects, in the User table. It would be nice if we only had the ‘pure’ Users in here, and all Person objects were just in their category.

So, I needed a way to filter this list.

Luckily, django’s admin has this capability. In my person/admin.py file, I had the following code:

1 from django.contrib import admin
2 from django.contrib import auth
3 
4 class UserAdmin(auth.admin.UserAdmin):
5     def queryset(self, request):
6         return super(UserAdmin, self).queryset(request).filter(person=None)
7 
8 admin.site.unregister(auth.models.User)
9 admin.site.register(auth.models.User, UserAdmin)

And, indeed, this works.

But then I found another User subclass. Now we needed a type of user that is distinct from Person (they are never rostered, are not associated with a given company, but do log into the system).

I wanted the changes to the admin to be isolated within the different apps, so I needed to be able to get the currently installed UserAdmin class, and subclass that to filter the queryset. So the code becomes (in both admin.py files):

 1 from django.contrib import admin
 2 from django.contrib import auth
 3 
 4 BaseUserAdmin = type(admin.site._registry[auth.models.User])
 5 
 6 class UserAdmin(BaseUserAdmin):
 7     def queryset(self, request):
 8         return super(UserAdmin, self).queryset(request).filter(foo=None)
 9 
10 admin.site.unregister(auth.models.User)
11 admin.site.register(auth.models.User, UserAdmin)

The only difference in the two files is the foo. This becomes whatever this sub-class’s name is. Thus, it is person in the person/admin.py file, and orguser in the orguser/admin.py file.

The next step is to change the backend so that it will automatically downcast the logged in user to their child class. Other people have detailed this in the past: mostly the performance issue vanishes here because we are only looking at a single database query for a single object.