Django AJAX Edit Mode

I can’t even remember where I saw this, but the suggestion was that viewing and editing data are different operations, and should be different modes.

For instance, when viewing some data, you would need to explicitly decide to enter the edit mode. In a web page, this would be by following a link to a page that had a form that allowed for editing. Attempting to submit the form would result in either the same page being displayed, along with any validation errors, or being redirected back to the viewing page.

This is the pattern I’ve been working on implementing with a module for our work system. However, we can reduce the amount of data sent, and the amount of page redraw, by using dynamic element replacement. So, we can have some AJAX that loads in the edit form in-place, and replaces the view form. Saving it then either returns the edit form with validation errors, or the view form again.

We can do all of this with one view. Depending upon how it is accessed, it will return either a different template, or render the template differently.

Solution One: render a different template.

# views.py
from django.views import generic

class AjaxEditView(generic.UpdateView):
  model = MyModel
  
  def get_template_names(self):
    if self.request.is_ajax():
      if 'edit' in self.request.GET:
        return 'partial/edit.html'
      return 'partial/display.html'
    return 'detail.html'
  
  def get_success_url(self):
    return reverse('mymodel_detail', kwargs={'pk': 1})

The disadvantages of this are that we need to name the url route, and use it here in this view, and that we have different templates. The templates are almost identical, however:

<!-- partial/display.html -->
{% load url from future %}

<form method="GET" action="{% url 'mymodel_detail' object.pk %}">
  <button>EDIT</button>
  <input type="hidden" name="edit" value="1">
  
  <table>
    {% for field in form %}
      <tr>
        <th>{{ field.label }}</th>
        <td>{{ field.value }}</td>
      </tr>
    {% endfor %}
  </table>
</form>
<!-- partial/edit.html -->
{% load url from future %}

<form method="POST" action="{% url 'mymodel_detail' object.pk %}">
  <button type="submit">SAVE</button>
  <a class="cancel" href="{% url 'mymodel_detail' object.pk %}">CANCEL</a>
  
  {% csrf_token %}
  
  <table>
    {% for field in form %}
    <tr>
      <th>{{ field.label }}</th>
      <td>{{ field.errors }}{{ field }}</td>
      <td>{{ field.help_text }}</t>
    </tr>
    {% endfor %}
  </table>
  
</form>

The partial/display.html template has a few things of note: it has a method="GET", and a single <input> element, which tells the view to render the response on a GET request ready for editing.

The partial/edit.html template has a link/button for cancelling. In this case, we could look at the form for the location we should load, but this is a bit more explicit.

Solution Two: context_data variable

The other solution uses just one AJAX template, but adds to the context of the view if it should be rendered in editable form or not.

# views.py
from django.views import generic

class AjaxEditView(generic.UpdateView):
  model = MyModel
  
  def get_context_data(self, **context):
    context['edit'] = self.request.GET.get('edit', False)
    return super(AjaxEditView, self).get_context_data(**context)
    
  def get_template_names(self):
    if self.request.is_ajax():
      return 'partial/form.html'
    return 'detail.html'
  
  def get_success_url(self):
    return reverse('mymodel_detail', kwargs={'pk': 1})

And our template looks like:

<!-- partial/form.html -->
{% load url from future %}

<form method="{% if edit %}POST{% else %}GET{% endif %}" 
      action="{% url 'user_detail' object.pk %}">
  
  <button type="submit">
    {% if edit %}SAVE{% else %}EDIT{% endif %}
  </button>
  
  {% if edit %}
    <a class="cancel" href="{% url 'user_detail' object.pk %}">
      CANCEL
    </a>
    {% csrf_token %}
  {% else %}
    <input name="edit" type="hidden" value="1">
  {% endif %}
  
  <table>
    {% for field in form %}
    <tr>
      <th>{{ field.label }}</th>
      <td>
        {% if edit %}
          {{ field.errors }}{{ field }}
        {% else %}
          {{ field.value }}
        {% endif %}
      </td>
      {% if edit %}
      <td>{{ field.help_text }}</td>
      {% endif %}
    </tr>
    {% endfor %}
  </table>
  
</form>

The downside of this one is that the template is much more complicated. I’ve been using the latter in a work project, but I may switch later.

The last part that ties all of this together is the Javascript. It’s fairly simple, written using jQuery:

$(function() {
  // Submit handler. Simply submit the form, and replace the
  // form in it's entirety with the response from the server.
  $(document).on('submit', 'form', function(evt) {
    var form = evt.target;
    var $form = $(form);
    evt.preventDefault();
    $.ajax({
      url: form.action,
      type: form.method,
      data: $form.serialize(),
      success: function(data){
        $form.replaceWith($.parseHTML(data));
      }
    });
  });
  
  // Cancel editing button handler. Do an ajax fetch on the
  // url of the button, and replace the parent form with
  // the response from the server.
  $(document).on('click', '.cancel', function(evt) {
    evt.preventDefault();
    $.ajax({
      url: evt.href,
      success: function(data){ 
        $(evt.target).closest('form').replaceWith($.parseHTML(data));
      }
    });
  });
});

The other thing that it is possible to do is make it so that the edit button will only display if the logged in user is permitted to edit that object.

In practice, I’m combining multiple display/edit views within one view (for related concepts: for instance Bank account details, Tax File Number and Superannuation details in the one page, but they have seperate models). I have some ideas about a nice way to handle this, but that’s for another post.

There is a project available on bitbucket that demonstrates this: dynamic-form-demo. There is a seperate branch for each solution outlined above.

jQuery dynamic forms

Some javascript, currently using jQuery, that will convert a form into an ajax form.

First up, a shim to enable FormData for non-compliant browsers. Based on formdata.js, but with some changes. Man, callback based code is a bitch when all you need to do is get some value, and have to work around that.

(function(w, $) {
  // Don't override if it is native.
  if (w.FormData)
    return;
      
  function FormData(form) {
    var fd = this;
    this.fake = true;
    this.boundary = "--------FormData" + Math.random();
    this._fields = [];
    this.contentType = 'multipart/form-data; boundary=' + this.boundary;
          
    if (form) {
      var $form = $(form);
      $.each($form.serializeArray(), function(i, obj){
        fd.append(obj.name, obj.value);
      });
      $form.find('[type=file]').each(function(i, file) {
        fd.append(file.name, file.files[0]);
      });
    }
  }
    
  // A listener to automatically add the binary version of the data. This may suck.
  $('[type=file]').change(function updateData(change) {
    var reader = new FileReader();
    reader.onload = function(load) {
      $(change.target.files[0]).data('binary-file-data', load.target.result);
    }
    reader.readAsBinaryString(change.target.files[0]);
  });
    
  // The interface FormData provides...
  FormData.prototype.append = function(key, value) {
    this._fields.push([key, value]);
  }
    
  // But, we will actually look more like a string to XMLHttpRequest.
  FormData.prototype.toString = function() {
    var boundary = this.boundary;
    var body = '';
    $.each(this._fields, function(i, field) {
      body += '--' + boundary + '\r\n';
      if (field[1].name) {
        var file = field[1];
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\"; filename=\""+ file.name +"\"\r\n";
        body += "Content-Type: "+ file.type +"\r\n\r\n";
        body += $(file).data('binary-file-data') + "\r\n";
      } else {
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\";\r\n\r\n";
        body += field[1] + "\r\n";
      }
    });
          
    body += "--" + boundary +"--";
    return body;
  }
  w.FormData = FormData;
})(window, jQuery);

Now, for the jQuery code. This will override the submit event on all forms. If there are any <input type=file> elements, then it will use a FormData object, else it will just .serialize() the object. The response will override the .html() content of the form, but not the form itself.

$('form').submit(function(evt) {
  evt.preventDefault();
  var form = evt.target, $form = $(form), data;
  var options = {
    cache: false,
    type: form.method,
    url: form.action,
    done: function(data) {
      $form.html(data);
    },
    fail: function(xhr, status, error) {
      console.log(status, error, xhr);
    }
  };
    
  if ($form.find('[type=file]').length) {
    data = new FormData(form);
    // Native FormData objects set this automatically (how???), but we need to manually do it.
    options.contentType = data.contentType || false;
    options.processData = false;
  } else {
    data = $form.serialize();
  }
  options.data = data;
  $.ajax(options);
});

Django urlpattern nested regex groups

Had one of those annoying things that I could not figure out why it was not working; learned something about how django’s url routing works along the way.

I created a new view for within the admin, that provides a summary of the permissions associated with a group, or set of groups. For our purposes, a company can have a number of groups associated with it, so I wanted to be able to optionally provide a company id: if it was provided, it would only show the groups+permissions for that company; if not provided it should show all of the groups and their permissions.

So, I had the urlpattern like:

# Included under '/company/...'
url(r'^((?P<company>\d+)?/)?groups/$', 'group_perms', name='company_group_permissions'),

This resolves fine. All of these variations work as expected:

http://example.com/company/10/groups/
http://example.com/company/groups/
http://example.com/company//groups/

However, I wanted to put a link in the admin change page for the company class, but was getting resolution errors, so I tried reverse directly:

reverse('group_perms', kwargs={'company': 10})
# -> NoReverseMatch: Reverse for 'group_perms' with 
#    arguments '()' and keyword arguments '{}' not found.

That’s odd. Maybe I was getting the name or something wrong:

resolve('/company/10/groups/')
# Result:
ResolverMatch(
  func=<function group_permissions at 0x104b96de8>, 
  args=(), kwargs={'company': '10'}, 
  url_name='company_group_permissions', 
  app_name='None', 
  namespace=''
)

Then, I removed the extra grouping in the regex:

url(r'^(?P<company>\d+)?/groups/$', 'group_perms', name='company_group_permissions'),

And it all works as expected. However, this slightly limits the available urls:

http://example.com/company/10/groups/
http://example.com/company//groups/

This one no longer works:

http://example.com/company/groups/

I can live with that.

I can’t find anything in the django docs that details this, although I kind-of remember reading that there are limits as to the ability of reverse() to generate urls.

Trust your tools, or how django's ORM bested me

Within my system, there is a complicated set of rules for determining if a person is “inactive”.

They may have been explicitly marked as inactive, or their company may have been marked as inactive. These are simple to discover and filter to only get active people:

Person.objects.filter(active=True, company__active=True)

The other clause for inactive users is if they only work at locations that have been marked as inactive. This means we can disable a location (within a company that remains active), and not have to manually deactivate the staff who only work at that location; it also means when we reactivate a location, staff will automatically be restored to an active state.

I’ve written the code several times that determines the activity status, but have never really been that happy with it. It generally degenerates into something that uses N+1 queries to discover the activity status of N people, or requires using django’s queryset.extra() method to run queries within the database.

Now, I have a cause to fetch all active staff, from the entire system. Which I had written a query to do, but it was mistakenly including staff who are only active at inactive units. I tried playing around with .extra(select={...}), but was not able to filter on the pseudo-fields that were generated.

Then, I had the idea to do the following:

active = Location.objects.active()
inactive = Location.objects.inactive()
Person.objects.filter(
  Q(locations__in=active) | ~Q(locations__in=inactive)
)

As long as the objects active and inactive are querysets, they will be lazily evaluated, and the SQL that is generated is relatively concise:

SELECT ... 
FROM "people" 
LEFT OUTER JOIN "people_locations" 
ON ("people"."id" = "people_locations"."person_id") 
WHERE (
  "people_locations"."location_id" IN (
    SELECT U0."id" FROM "location" U0 WHERE U0."status" = 0
  )
  OR NOT ((
    "people"."id" IN (
      SELECT U1."person_id" FROM "people_locations" U1 WHERE (
        U1."location_id" IN (
          SELECT U0."id" FROM "location" U0 WHERE U0."status" = 1
        )
        AND U1."person_id" IS NOT NULL
      )
    ) 
    AND "people"."id" IS NOT NULL)
  )
)
ORDER BY "..." ASC

This is much better than how I had previously done it, and has the bonus of being db-agnostic: wheras my previous solution used Postgres ARRAY types to aggregate the statuses of locations into a list.

The moral of the story: trust your high-level abstraction tools, and use them first. If you still have performance issues, then look at optimising.

Sorting dates in DataTables

If you have tabular data, then semantically, you’ll want to put it into an HTML table. It makes sense, and is certainly easier than trying to post-style nested divs as a table.

The other really nice thing is that it’s fairly easy to use DataTables to then make that table dynamic. Especially useful if your table is large: I use it on a report of all customers in my work, and have just started using it in some user-facing pages. In essence, it is as simple as doing:

$('#table-id').dataTable();

With this, you get sortable columns, pagination, and searching.

But sorting of dates sucks, unless they are in ISO8601 format. ISO8601 is fantastic, by the way. Not only do you get dates/datetimes that are inherently no longer ambiguous, but they sort alphabetically, as you would expect. Because every field is larger than all of the fields following it, and all fields are zero-padded, eveny date or datetime will be correctly sorted.

However, the general public does not understand these two reasons for a ‘one true date format’, so we are generally forced to display it in a more readable format. Which doesn’t sort alphabetically.

There is a trick you can use to get sorting, and nice dates using DataTables, though. For example, the following (rendered) html will sort correctly, both ascending and desencding, but also only display a nice format:

<td>
  <span style="display: none;">2012-06-07</span>
  Thursday, June 7th, 2012
</td>

In django, you can use the following snippet:

<td>
  <span style="display: none;">{{ value|date:"Y-m-d" }}</span>
  {{ value|date:"l, F jS, Y" }}
</td>

Recently, DataTables also had a blog post about how to use it with Twitter Bootstrap 2. I think it looks rather nice. And with this tip, it is so much more useful.

You can also use this way of thinking on other things that should be sorted differently to how they are printed.

Spurious CORS Errors from Sentry

I realised the other day that Sentry, the awesome system we have been using for a while to track our error logs from our Django project, can also be used to track exceptions from other systems. Like Javascript. In fact, there is a client available: raven.js.

So, we have a server set up for work, but I have a side-project I have been working on, Workout Builder. So, I thought I’d set up a server in Heroku to act as my sentry server. And I found a nice simple way to get up and running: Daniel Watkins has a nice post over at Odd_Blog, Deploying Sentry on Heroku.

It’s pretty straightforward, and extremely simple. I got it up and running in no time, and then attempted to set up an email service. Rather than use my actual account for sending, I thought I’d set up a sending-only account at my domain, hosted as a Gmail Apps domain. So, I set it up, and set about testing.

All of a sudden, I’m getting errors, that didn’t appear for 30 seconds, that my test domain is not permitted to send a request due to CORS. But, I had been sending them previously.

After lots of dicking around, I discovered it was because I did not have the gmail settings quite right. Instead of telling me what the problem was, something was masking the issue (that the server was timing out because the server/port combination was not correct), and jQuery thought it was a CORS issue.

So, fixing up the email sending settings, and it’s all gravy:

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

EMAIL_HOST = 'smtp.gmail.com'
EMAIL_HOST_PASSWORD = '<oh no you don\'t>'
EMAIL_HOST_USER = 'noreply@schinckel.net'
EMAIL_PORT = 587
EMAIL_USE_TLS = True

Initially, I had used mail.google.com, port 25, and EMAIL_USE_TLS = False. Eventually, I got it all right.

Developing RESTful Web APIs with Python, ...

This week’s Python Weekly has a link to a presentation by Nicola Iarocci, called Developing RESTful Web APIs with Python, Flask and MongoDB.

I have a few minor concerns with some aspects of the content.

No form validation

This concerns me. I’ve recently started using django’s forms for my validation layer for API calls, and also for the generation of the serialised output. It’s not completely flawless, but it seems to be working quite well. It certainly is more robust than my hand-rolled scheme of validating data, and using code that is better tested than my own is always a bonus.

Instead, as we see later, there is a data validation layer. It has basically the same goal as django’s forms, but is somewhat more nested, rather than using classes. Also, using classes makes it easier to have inheritance, a great way to have shared rules. You could do this using the same function in your custom validation, but this feels disconnected.

MongoDB
scalable, high-performance, …

The integrity of my data is important to me. It’s very rare that the db is the limiting factor in my system’s performance, and having stuff written to disk as soon as it is ‘real’ is kind-of critical.

http://api.example.com/v1/contacts/

Okay, this is where I jump on my high horse: “versioning should happen in the media-type”. Or even better, resources should be forwards and backwards compatible, and clients should be written to handle (or ignore) changes to schemata.

@mimerender( ... )

A decorator that has 5 arguments? That will be applied to every view function? Surely there’s a way to do this without having to decorate every function. Django CBV FTW here.

“Thu, 1 Mar 2012 10:00:49 UTC”

Egad. I can’t think of a reason to have machine readable dates in any format other than ISO 8601. Purely for the reason of being able to sort dates whilst they are still strings.

PATCH
Why not PUT?

Why not POST?

This is something that has been debated for ages. I think I kind-of agree with the author: PATCH is more explicitly a partial update. It does make me think about using some type of diff, but I guess using concurrency control covers the same ground.

"<link rel='parent' ... />"

Okay, HTML/XML inside a JSON object?

Why not have:

{
  "rel": "parent",
  "title": "...",
  "href": "..."
}

At least that way you’ll be able to parse the data out natively.

“updated”: “…”,
“etag”: “…”

I’m not sure if it is necessary/warranted/desired to have the etag as part of the representation. Especially if the etag is generated from the content: that would kind-of preclude it.

Personally, I generate etags from a higher resolution timestamp (possibly hashed with the object id, class or whatever). Whilst etags are opaque, having them as human readable helps with troubleshooting.

To me, this seems to be metadata, and should not be part of the object. I think you could argue a case that within Collection+JSON you could add this in, for convenience. It certainly would make it easier not to have to store the etag in a seperate variable on the client, for one.

The discussion about Concurrency Control is quite good. Which reminds me: I enjoyed most of this presentation. I have some minor nitpicks, but some of those I understand the author’s choices. Some I don’t (date format). It’s certainly better than the REST API Design Rulebook, which is a load of junk.

Metaclass magic registry pattern

The Registry Pattern is something I use relatively frequently. In django, for instance, we see it used for the admin interface, and I used very derivative code for my first API generation tool: django-rest-api. For our integration with external POS and other systems, we need to register importers, so that the automated stats fetching is able to look for units that need to fetch data from an external system’s website, or parse incoming email headers for matching delivered data.

I had been using something similar to:

from base import BaseStatsImporter, register

class FooStatsImporter(BaseStatsImporter):
    # ...

register(FooStatsImporter)

This is all well and good, but it is annoying. I need to remember to register each class after I declare it.

Then I discovered the magic of __metaclass_, used with __new__:

class RegistryMetaClass(type):
    def __new__(cls, clsname, bases, attrs):
        new_class = super(cls, RegistryMetaClass).__new__(cls, clsname, bases, attrs)
        register(new_class)
        return new_class
        
class BaseStatsImporter(object):
    __metaclass__ = RegistryMetaClass
    
    # ...

As long as your subclasses don’t override __metaclass__, then every new subclass will be added to the registry.

Obviously, this is magic, and in some cases the explicit way would be better.

The Organism Application

I had an email from a self-confessed django beginner, asking for some assistance. Here is my solution, as I worked through it.

The Application

The application is designed to allow tracking information related to identifying various organisms. An organism may have many identifying features, such as on a tree, the height, and the leaf morphology, or on a bird, the colour of the feathers, size of the egg and so on. To make it simpler for the users, it would be useful to classify organisms as belonging to a type, which can then be used to limit the available choices of identifying features: if an organism is a bird, then we only show those features that make sense for a bird.

To do all of this, we can have a class structure that looks somewhat like:

# models.py
from django.db import models

class OrganismType(models.Model):
    description = models.CharField(max_length=200)

class IdentificationField(models.Model):
    type = models.ForeignKey(OrganismType, related_name='id_fields')
    name = models.CharField(max_length=200)
    
    class Meta:
        unique_together = ('type', 'name')

class Organism(models.Model):
    common_name = models.CharField(max_length=200)
    latin_name = models.CharField(max_length=200, unique=True)
    type = models.ForeignKey(OrganismType, related_name='organisms')

class IdentificationDetail(models.Model):
    organism = models.ForeignKey(Organism, related_name="id_details")
    field = models.ForeignKey(IdentificationField)
    description = models.CharField(max_length=250)
    
    class Meta:
        unique_together = ('organism', 'field')

You’ll see I’ve also included a couple of unique_together constraints: I’ve assumed that each field for a given organism should only appear once.

Bending the admin to our will

Next, we can put all of this into the admin. This is really quite simple, but, as we will see, has it’s limits.

# admin.py
from django.contrib import admin

from models import OrganismType, Organism, IdentificationField, IdentificationDetail

class IdentificationFieldInline(admin.TabularInline):
    model = IdentificationField
    extra = 0

class OrganismTypeAdmin(admin.ModelAdmin):
    inlines = [IdentificationFieldInline]

class IdentificationDetailInline(admin.TabularInline):
    model = IdentificationDetail
    extra = 0

class OrganismAdmin(admin.ModelAdmin):
    inlines = [IdentificationDetailInline]    
    list_display = ('common_name', 'latin_name', 'type')
    list_filter = ('type',)

admin.site.register(OrganismType, OrganismTypeAdmin)
admin.site.register(Organism, OrganismAdmin)

I’ve removed the extra empty forms on the formsets, it looks much cleaner. I’ve also used a couple of the nice features of the admin to make display of stuff better.

At this point, thanks to the magic of django, you now have an administrative interface. But, it doesn’t quite do what we want: that is, we haven’t limited which identification fields will be available in the organism’s inlines.

To do that, we need to fiddle with the formset.

# forms.py
from django import forms

from models import IdentificationDetail, Organism

class IdentificationDetailFormSet(forms.models.BaseInlineFormSet):
    def __init__(self, *args, **kwargs):
        super(IdentificationDetailFormSet, self).__init__(*args, **kwargs)
        for form in self.forms:
            self.update_choices(form)
    
    # We need to override the constructor (and the associated property) for the
    # empty form, so dynamic forms work.
    def _get_empty_form(self, **kwargs):
        form = super(IdentificationDetailFormSet, self)._get_empty_form(**kwargs)
        self.update_choices(form)
        return form
    empty_form = property(_get_empty_form)
    
    # This updates one form's 'field' field queryset, if there is an organism with type
    # associated with the formset. Otherwise, make the choice list empty.
    def update_choices(self, form):
        if 'type' in self.data:
            id_fields = OrganismType.objects.get(pk=self.data['type']).id_fields.all()
        elif self.instance.pk and self.instance.type:
            id_fields = self.instance.type.id_fields.all()
        else:
            id_fields = IdentificationDetail.objects.none()
        
        form.fields['field'].queryset = id_fields

This process is something I’ve talked about before (and finding that post was what pointed the questioner in my direction), but I’ll discuss it again anyway. This is perhaps a more concrete example anyway.

We want to change the queryset available to a given field (in this case, confusingly called field), based on the value of a related object. In this case, we want to set the queryset of an identification detail’s field to all of the available identification fields on the related organism’s type. Whew!

As it turns out, it’s easier to see this in the code. Note also that if there is no selected organism type (as would be the case when an empty form is presented), no fields can be selected.

This alone would work: except that changing the organism’s type should change the available list of field types. There are two approaches that can be used: have all of the data available in the page somewhere, and use JavaScript to filter the available list of field types, or fetch the data dynamically from the server (again, using JavaScript) at the time the type is changed. If I were using something like KnockoutJS, then the former would be easier, and improve the responsiveness: the change would be immediate. Since I’m not using anything that doesn’t come with django, I’ll fetch the data on each change.

So, we are going to need some JavaScript. When we do the end-user page, it’s easy to see how to put that in, but we need to understand how to override django’s admin templates in order to inject it in this case.

The django documentation has some nice detail about how to do this: Overriding admin templates. In this case, we need to create a file within our app at templates/admin/organisms/organism/change_form.html. We want to just add data to the regular template, so we just inherit from it.

{% extends 'admin/change_form.html' %}

{% block after_related_objects %}
{{ block.super }}
<script>
django.jQuery(function($){
  $('#id_type').change(function(evt){
    $.ajax({
      url: "/admin/organisms/organismtype/" + this.value + '/fields/',
      type: 'get',
      success: function(data) {
        $('tr.form-row td.field-field select').html(data);
      }
    });
  });
});
</script>
{% endblock %}

The script here adds a change event handler to the organism type <select> element, that hits the server, and gets the list of fields for that type. It then sets the content of the inline identification detail field fields to the data the server returned. This clears whatever had been stored there previously, but that is probably the behaviour we want in this case. Note that I am hard-coding the URL for now: we’ll see a way to handle that in a better way later.

Only one thing remains: to actually write the view that returns the desired content of the <select> element. For now, we will put this into the admin class of the organism type. Again, later we’ll move this to a proper seperate view, but doing it this way shows how easy it is to extend the admin interface.

Back in our admin.py file, we want to change the OrganismTypeAdmin class:

# admin.py

from django.contrib import admin
from django.conf.urls import patterns, url
from django.http import HttpResponse

# [snip]

class OrganismTypeAdmin(admin.ModelAdmin):
    inlines = [IdentificationFieldInline]
    
    def get_urls(self, **kwargs):
        urls = super(OrganismTypeAdmin, self).get_urls(**kwargs)
        urls = patterns('', 
            url(r'^(.*)/fields/$', self.get_fields, name='organisms_organismtype_fields'),
        ) + urls
        return urls
    urls = property(get_urls)
    
    def get_fields(self, request, *args, **kwargs):
        data = "<option value>---------</option>"
        if args[0]:
            data += "".join([
                "<option value='%(id)s'>%(name)s</option>" % x 
                for x in OrganismType.objects.get(pk=args[0]).id_fields.values()
            ])
        return HttpResponse(data)

We can use the fact that the admin model object provides its own urls, and we can override the method that generates them. We need to put our fields view before the existing ones (and allow empty strings where we want the primary key), else it will be matched by another route.

Finally, we write the view itself. If there was no primary key provided, we return a “null” option, otherwise we include that and the actual list of choices.

Doing it for real

Of course, in a real environment, we probably don’t want to give access to the admin interface to anyone but trusted users. And even then, limit that to as few as possible. In this case, I would suggest that the admin users would be creating the OrganismType objects, but creating Organism objects would be done by regular users. Which means we really only have a couple of pages that need to be written for the outside world:

  • View a list of organisms.
    • Filter the list of organisms by OrganismType
    • Search for an organism by common name or latin name
    • Search for an organism by some other means (feather colour, etc)
  • Create a new organism
  • Edit an existing organism
  • Fetch a list of field types for a given organism type (the get_fields view above.)

This may come in a future post: I had forgotten about this and need some time to get back into it.

Hypermedia APIs in Django: Leveraging Class Based Views

It seems that I keep rewriting code that generates APIs from django. I think I’m getting closer to actually getting it right, though :)

I’m rather keen on Collection+JSON at the moment, and spent some time over easter writing an almost complete Collection+JSON client, using KnockoutJS. It loads up a root API url, and then allows navigation around the API using links. While working on this, it occurred to me that Collection+JSON really encodes the same information as a web page:

  • every <link> or <a href=...></a> element is either in links or queries.
  • form-based queries map nicely to queries elements that have a data attribute.
  • items encapsulates the actual data that should be presented.
  • template contains data that can be used to render a form for creating/updating an object.

Ideally, what feels best from my perspective is to have a pure HTML representation of the API, which can be rendered by browsers with JS disabled, and then all of the same urls could also be fetched as Collection+JSON. Then, you are sharing the code, right up to the point where the output is generated.

To handle this, I’ve come up with a protocol for developing django Class Based Views that can be represented as Collection+JSON or plain old HTML. Basically, your view needs to be able to provide links, queries, items. template comes from a form object (called form), and by default items is the queryset attribute.

leveraging views

I subscribe the idea that the less code that is written the better, and I believe that the API wrapper should (a) have minimal code itself, and (b) allow the end developer to write as little code as possible. Django is a great framework, we should leverage as much as is possible of that well written (and well tested) package.

The part of a hypermedia API that is sometimes ignored by web developers is handling the media type selection. I believe this is the domain of the “Accept” and “Content-Type” headers, not anything to do with the URL. Thus, I have a mixin that allows for selecting the output format based on the Accept header. It uses the inbuilt render_to_response method that a django View class has, and handles choosing how to render the response. As it should.

The other trick is how to get the links, queries, items and template into the context. For this, we can use get_context_data. We can call self.get_FOO(**kwargs) for FOO in each of those items. It is then up to the View class to handle those methods.

By default, a Model-based Resource is likely to have a form class, and a model class or a queryset. These can be used to get the items, and in the case of the form, the template. Even in the instance of the queryset (or model), we use the form class to turn the objects into something that can be rendered.

Finally, so it’s super-easy to use the same pattern as with django’s Views (generic.CreateView, for instance), I have a couple of classes: ListResource and DetailResource, which map directly onto CreateView and UpdateView. In the simplest case, you can just use:

urlpatterns = patterns('',
    url(r'^foo/$', ListResource.as_view(model=Foo)),
    url(r'^foo/(<?P<pk>\d+)/$', DetailResource.as_view(model=Foo))
)

There is also a Resource, which just combines the resource-level bits with generic.TemplateView. You can use ResourceMixin with any other class-based-view, but make sure it appears earlier than the django view class, to make sure we get the correct method resolution order.

There is still the matter of the links attribute. Knowing what to put into this can be a bit tricky. I’ve come to realise that this should contain a list of the valid states that can be accessed when you are in a given state. You will want to use django’s reverse function to populate the href attribute:

class Root(Resource):
    template_name = 'base.html'
    
    def get_links(self):
        return [
            {"rel": "root", "href": reverse('root'), "prompt": "Home"},
            {"rel": "user", "href": reverse('user'), "prompt": "You"},
            {"rel": "links", "href": reverse('tasks:list'), "prompt": "Task List"},
        ]

Note that you actually need to provide the view names (and namespaces, if appropriate) to reverse. Similarly, for any queries, you would want to use reverse, to make it easier to change the URL later. Also, django will complain if you have not installed something you reference, meaning your links and queries should never 404.

I’m still toying with the feature of having an automatic list of links that should be used for every view. Obviously, this should only contain a list of states that can be moved to from any state within the system.

For rendering HTML, you may need to change your templates: actually, you should change your templates. Instead of using:

<a href="{% url 'foo'  %}">Foo Link</a>

You would reference that items in your links array:

<a href="{{ links.foo.href }}">{{ links.foo.prompt }}</a>

I have used a little bit of magic here too: in order to be able to access links items according to their rel attribute, when rendering HTML, we use a sub-class of list that allows for __getattr__ to look through the items and find the first one that matches by rel type.

enter django-hypermedia

As you may surmise from the above text: I’ve already written a big chunk of this. It’s not complete (see below), but you can see where it is at now: django-hypermedia.

There is a demo/test project included, that has some functionality. It shows how you still need to do things “the django way”, and then you get the nice hypermedia stuff automatically.

what is still to come?

I’ve never really been happy with Collection+JSON’s error object, so I haven’t started handling that yet. I want to be able to reference where the error lies, similar to how django’s forms can display their own errors.

I want to flesh out the demo/test project. It has some nice bits already, but I want to have it so that it also uses my nice KnockoutJS client. Pretty helps. :)