Hypermedia APIs in Django: Leveraging Class Based Views

It seems that I keep rewriting code that generates APIs from django. I think I’m getting closer to actually getting it right, though :)

I’m rather keen on Collection+JSON at the moment, and spent some time over easter writing an almost complete Collection+JSON client, using KnockoutJS. It loads up a root API url, and then allows navigation around the API using links. While working on this, it occurred to me that Collection+JSON really encodes the same information as a web page:

  • every <link> or <a href=...></a> element is either in links or queries.
  • form-based queries map nicely to queries elements that have a data attribute.
  • items encapsulates the actual data that should be presented.
  • template contains data that can be used to render a form for creating/updating an object.

Ideally, what feels best from my perspective is to have a pure HTML representation of the API, which can be rendered by browsers with JS disabled, and then all of the same urls could also be fetched as Collection+JSON. Then, you are sharing the code, right up to the point where the output is generated.

To handle this, I’ve come up with a protocol for developing django Class Based Views that can be represented as Collection+JSON or plain old HTML. Basically, your view needs to be able to provide links, queries, items. template comes from a form object (called form), and by default items is the queryset attribute.

leveraging views

I subscribe the idea that the less code that is written the better, and I believe that the API wrapper should (a) have minimal code itself, and (b) allow the end developer to write as little code as possible. Django is a great framework, we should leverage as much as is possible of that well written (and well tested) package.

The part of a hypermedia API that is sometimes ignored by web developers is handling the media type selection. I believe this is the domain of the “Accept” and “Content-Type” headers, not anything to do with the URL. Thus, I have a mixin that allows for selecting the output format based on the Accept header. It uses the inbuilt render_to_response method that a django View class has, and handles choosing how to render the response. As it should.

The other trick is how to get the links, queries, items and template into the context. For this, we can use get_context_data. We can call self.get_FOO(**kwargs) for FOO in each of those items. It is then up to the View class to handle those methods.

By default, a Model-based Resource is likely to have a form class, and a model class or a queryset. These can be used to get the items, and in the case of the form, the template. Even in the instance of the queryset (or model), we use the form class to turn the objects into something that can be rendered.

Finally, so it’s super-easy to use the same pattern as with django’s Views (generic.CreateView, for instance), I have a couple of classes: ListResource and DetailResource, which map directly onto CreateView and UpdateView. In the simplest case, you can just use:

urlpatterns = patterns('',
    url(r'^foo/$', ListResource.as_view(model=Foo)),
    url(r'^foo/(<?P<pk>\d+)/$', DetailResource.as_view(model=Foo))
)

There is also a Resource, which just combines the resource-level bits with generic.TemplateView. You can use ResourceMixin with any other class-based-view, but make sure it appears earlier than the django view class, to make sure we get the correct method resolution order.

There is still the matter of the links attribute. Knowing what to put into this can be a bit tricky. I’ve come to realise that this should contain a list of the valid states that can be accessed when you are in a given state. You will want to use django’s reverse function to populate the href attribute:

class Root(Resource):
    template_name = 'base.html'
    
    def get_links(self):
        return [
            {"rel": "root", "href": reverse('root'), "prompt": "Home"},
            {"rel": "user", "href": reverse('user'), "prompt": "You"},
            {"rel": "links", "href": reverse('tasks:list'), "prompt": "Task List"},
        ]

Note that you actually need to provide the view names (and namespaces, if appropriate) to reverse. Similarly, for any queries, you would want to use reverse, to make it easier to change the URL later. Also, django will complain if you have not installed something you reference, meaning your links and queries should never 404.

I’m still toying with the feature of having an automatic list of links that should be used for every view. Obviously, this should only contain a list of states that can be moved to from any state within the system.

For rendering HTML, you may need to change your templates: actually, you should change your templates. Instead of using:

<a href="{% url 'foo'  %}">Foo Link</a>

You would reference that items in your links array:

<a href="{{ links.foo.href }}">{{ links.foo.prompt }}</a>

I have used a little bit of magic here too: in order to be able to access links items according to their rel attribute, when rendering HTML, we use a sub-class of list that allows for __getattr__ to look through the items and find the first one that matches by rel type.

enter django-hypermedia

As you may surmise from the above text: I’ve already written a big chunk of this. It’s not complete (see below), but you can see where it is at now: django-hypermedia.

There is a demo/test project included, that has some functionality. It shows how you still need to do things “the django way”, and then you get the nice hypermedia stuff automatically.

what is still to come?

I’ve never really been happy with Collection+JSON’s error object, so I haven’t started handling that yet. I want to be able to reference where the error lies, similar to how django’s forms can display their own errors.

I want to flesh out the demo/test project. It has some nice bits already, but I want to have it so that it also uses my nice KnockoutJS client. Pretty helps. :)

Collection+JSON Primer (and comments)

Collection+JSON, created by Mike Amundsen, is a standard way of creating hypermedia APIs. There were a few things I didn’t pick up correctly reading through his great book, or the spec.

First, let us look at a partial example document.

{
  collection: {
    version: "1.0",
    href: "http://api.example.com/",
    links: [],
    items: [],
    queries: [],
    template: {},
    error: {}
  }
}

I’m not so keen on having the version number in the document itself, as this refers to the version of Collection+JSON, rather than the version of the document. In my mind, the version of Collection+JSON should be contained within the media-type (Content-Type: application/vnd.collection+json;version=1.0), just as the version of the document is contained within the Etag header (Etag: 026e10f644ba4b06). Anyway, I’ll let that slide for now.

Secondly, having the href of the collection seems a little superfluous. I’m assuming there will always be an entry in links that has a rel=self, which should give you the same value. Again, not a big issue.

What I was a little unclear on was the difference between links and queries. We can have a look at a couple of examples:

links: [
  {href: "http://api.example.com", rel: "self", prompt: "Home", name: "home", render: "link"},
  {href: "http://api.example.com", rel: "users", prompt: "Users", name: "users", render: "link"}
],
queries: [
  {href: "http://api.example.com", rel: "search", prompt: "Enter search string", data: [
    {name: "search", value: ""}
  ]}
]

The difference between links and queries to me seems somewhat artificial. Sure, in this case, my query has data fields, but it seems that this is not always necessary. The example Mike uses in his book:

queries: [
  {href: "...", rel: "all", prompt: "All tasks"},
  {href: "...", rel: "open", prompt: "Open tasks"},
  {href: "...", rel: "closed", prompt: "Closed tasks"},
  {href: "...", rel: "date-due", prompt: "Date Due", data: [
    {name: "dateStart", value: "", prompt: "Start Date"},
    {name: "dateStop", value: "", prompt: "Stop Date"}
  ]},
]

I’m not quite sure when I should be using a link, and when I should be using a query? In this example, it looks like a query is a filter on the collection: maybe that is the difference?

The other sticking point I have is that both queries and lists are GET requests: the data attribute is simply the query string applied to the URL. Before I continue, we need to look at the items and template attributes of the collection object. In this case, we have a single-object collection, including a write template for it.

items: [
  {
    href: "...",
    data: [
      {name: "first_name", value: "Matthew", prompt: "First name"},
      {name: "last_name", value: "Schinckel", prompt: "Last name"},
      {name: "email": value: "matt@schinckel.net", prompt: "Email address"},
      {name: "gender", value: "male", prompt: "Gender"}
    ]
  }
],
template: {
  data: [
    {name: "first_name", value: "Matthew", prompt: "First name"},
    {name: "last_name", value: "Schinckel", prompt: "Last name"},
    {name: "email", value: "matt@schinckel.net", prompt: "Email address", regexp: "^[^@]+@[^@]+\.[^@]+"},
    {name: "gender", value: "male", prompt: "Gender", options: [
      {value: "male", text: "Male"}, 
      {value: "female", text: "Female"}
    ]}
  ]
}

Again, we see duplicate information. In this case, the template is populated: if it were a ‘proper’ collection rather than a single object, the template would be used for creating new objects in the collection, so I’m prepared to let this one go. You’ll also notice that I’m using a couple of undocumented features: regexp and options. These enable us to either present a list of choices to the user, or have client-side validation based on a regular expression.

To update an object, we can use a PUT (or POST) to the object’s href, and we send the name/value parts of the updated template data:

{
  template: {
    data: [
      {name: "first_name", value: "Matthew"},
      {name: "last_name", value: "Schinckel"},
      {name: "gender", value: "male"},
      {name: "email", value: "matt@schinckel.net"}
    ]
  }
}

To create a new object, we send the same type of data in a POST request to a collection’s href. To delete an object, we can send a DELETE request to the object’s href.

Finally, we come to the error property. I wrote last night how I think this is a little limiting: Collection+JSON error objects. Anyway, an error looks like:

error: {
  title: "Error saving your details",
  code: "409",
  message: "Your date of birth is invalid (19977-11-30 is not a valid date)"
}

After writing most of this, I did come across Collection+JSON - Examples, but I may have described it in a slightly different manner. It still doesn’t elaborate on the difference between links and queries, however.

Collection+JSON error objects

I’m still keen on the idea of implementing a rich hypermedia API based on django’s forms.

One of the nicest things about the django forms is that they handle the validation of incoming data. Each form field has a .clean() method, which will clean the data. On a form, it is then possible to have extra methods, .clean_FIELDNAME(), which will process the incoming data again, meaning you don’t need to subclass a field to add simple cleaning functionality. Finally, the form itself has a .clean() method, that can be used to clean composite data, say, ensuring that start is before finish.

The form validation code will create an errors property on the form, that will contain the fields that have errors, and any non-field errors (such as the last example above). When rendering an HTML page, and displaying a form that has errors, these are marked up with CSS classes that enable you to show which fields have invalid or missing data, and also display relatively friendly messages (which you can customise).

But Collection+JSON has a fairly simple error property on the collection object:

{
  "collection": {
    "error": {
      "title": "Error saving your details",
      "code": "409",
      "message": "Your date of birth is invalid (19777-11-30)"
    }
  }
}

Compare this to the format I have been using for JSON responses:

{
  "message": "Error saving your details",
  "detail": {
    "date_of_birth": "The value '19777-11-30' is not a valid date."
  }
}

Programmatically, this allows me to attach the error messages to where they belong: the message value is shown in the main messages area of the client, the detail values for each field are attached to the fields for which they apply.