Schinckel.net

Hypermedia APIs in Django: Leveraging Class Based Views

2012-04-26T00:13:24+09:30

It seems that I keep rewriting code that generates APIs from django. I think I’m getting closer to actually getting it right, though :)

I’m rather keen on Collection+JSON at the moment, and spent some time over easter writing an almost complete Collection+JSON client, using KnockoutJS. It loads up a root API url, and then allows navigation around the API using links. While working on this, it occurred to me that Collection+JSON really encodes the same information as a web page:

every <link> or <a href=...></a> element is either in links or queries.
form-based queries map nicely to queries elements that have a data attribute.
items encapsulates the actual data that should be presented.
template contains data that can be used to render a form for creating/updating an object.

Ideally, what feels best from my perspective is to have a pure HTML representation of the API, which can be rendered by browsers with JS disabled, and then all of the same urls could also be fetched as Collection+JSON. Then, you are sharing the code, right up to the point where the output is generated.

To handle this, I’ve come up with a protocol for developing django Class Based Views that can be represented as Collection+JSON or plain old HTML. Basically, your view needs to be able to provide links, queries, items. template comes from a form object (called form), and by default items is the queryset attribute.

leveraging views

I subscribe the idea that the less code that is written the better, and I believe that the API wrapper should (a) have minimal code itself, and (b) allow the end developer to write as little code as possible. Django is a great framework, we should leverage as much as is possible of that well written (and well tested) package.

The part of a hypermedia API that is sometimes ignored by web developers is handling the media type selection. I believe this is the domain of the “Accept” and “Content-Type” headers, not anything to do with the URL. Thus, I have a mixin that allows for selecting the output format based on the Accept header. It uses the inbuilt render_to_response method that a django View class has, and handles choosing how to render the response. As it should.

The other trick is how to get the links, queries, items and template into the context. For this, we can use get_context_data. We can call self.get_FOO(**kwargs) for FOO in each of those items. It is then up to the View class to handle those methods.

By default, a Model-based Resource is likely to have a form class, and a model class or a queryset. These can be used to get the items, and in the case of the form, the template. Even in the instance of the queryset (or model), we use the form class to turn the objects into something that can be rendered.

Finally, so it’s super-easy to use the same pattern as with django’s Views (generic.CreateView, for instance), I have a couple of classes: ListResource and DetailResource, which map directly onto CreateView and UpdateView. In the simplest case, you can just use:

urlpatterns = patterns('',
    url(r'^foo/$', ListResource.as_view(model=Foo)),
    url(r'^foo/(<?P<pk>\d+)/$', DetailResource.as_view(model=Foo))
)

There is also a Resource, which just combines the resource-level bits with generic.TemplateView. You can use ResourceMixin with any other class-based-view, but make sure it appears earlier than the django view class, to make sure we get the correct method resolution order.

links

There is still the matter of the links attribute. Knowing what to put into this can be a bit tricky. I’ve come to realise that this should contain a list of the valid states that can be accessed when you are in a given state. You will want to use django’s reverse function to populate the href attribute:

class Root(Resource):
    template_name = 'base.html'
    
    def get_links(self):
        return [
            {"rel": "root", "href": reverse('root'), "prompt": "Home"},
            {"rel": "user", "href": reverse('user'), "prompt": "You"},
            {"rel": "links", "href": reverse('tasks:list'), "prompt": "Task List"},
        ]

Note that you actually need to provide the view names (and namespaces, if appropriate) to reverse. Similarly, for any queries, you would want to use reverse, to make it easier to change the URL later. Also, django will complain if you have not installed something you reference, meaning your links and queries should never 404.

I’m still toying with the feature of having an automatic list of links that should be used for every view. Obviously, this should only contain a list of states that can be moved to from any state within the system.

For rendering HTML, you may need to change your templates: actually, you should change your templates. Instead of using:

<a href="{% url 'foo'  %}">Foo Link</a>

You would reference that items in your links array:

<a href="{{ links.foo.href }}">{{ links.foo.prompt }}</a>

I have used a little bit of magic here too: in order to be able to access links items according to their rel attribute, when rendering HTML, we use a sub-class of list that allows for __getattr__ to look through the items and find the first one that matches by rel type.

enter django-hypermedia

As you may surmise from the above text: I’ve already written a big chunk of this. It’s not complete (see below), but you can see where it is at now: django-hypermedia.

There is a demo/test project included, that has some functionality. It shows how you still need to do things “the django way”, and then you get the nice hypermedia stuff automatically.

what is still to come?

I’ve never really been happy with Collection+JSON’s error object, so I haven’t started handling that yet. I want to be able to reference where the error lies, similar to how django’s forms can display their own errors.

I want to flesh out the demo/test project. It has some nice bits already, but I want to have it so that it also uses my nice KnockoutJS client. Pretty helps. :)

Belair Hill Climb

2012-04-25T22:56:03+09:30

Since I bought my Garmin Forerunner 405cx in November 2010, I’ve gradually gotten more and more into running. Having a computer that tracks when, where, and how fast I run appeals to me, and has probably been the biggest motivator to me running as much as I do now.

I was tracking my running using Garmin Connect, but recently, thanks to my good friend Travis, got hooked on Strava. The key feature for me is the segments, and how competitive they enable me to be. Mostly against myself (my hard running occurs on the same track each week, which no-one else using Strava seems to have discovered).

Technically, I’m in training for this year’s City to Bay, although that is a long way away, so I’m spending the time working on getting faster over that type of distance. Last year I finished in 1490th place, with a time of 00:54:43. My target time had been 00:54:00, so I was a little disappointed to miss that by such a short margin. I did speed up a little too early (2km out), which nearly killed me, and I needed to back off. Also, I was absolutely exhausted by the end.

In fact, if I look at my performance, at the 47 minute mark I sped up, increasing my pace from 4:30 to 4:00 min/km, and promptly had to slow to a walk. There may have been some dry retching going on there too.

So, I set a target time for this year’s race of 00:48:00, with the possibility of reducing that to 00:44:00 if I could get to my target time 12 weeks before the race. Now, it does occur to me that my target pace is the one that forced me to stop early last year, but I think I’m already in much better shape now than I was then.

To improve my speed, I determined that first I needed to reduce my heart-rate. In last year’s City to Bay, my heartrate was basically in the Threshold zone (Z4) for the entire race. It did get a little higher when I sped up, but otherwise was fairly constant, which means I probably did run that race as fast as I could have then.

So, my training plan of late has been to run a lot more with my HR in zone 3, and see if I can get my speed up. So far, it seems to be working. I’ve been doing lots of 30 and 60 minute easy runs, where my Forerunner will beep at me if I get my HR above Z3. I’m finding that lately I’ve been running at around the same pace, but find my HR sometimes dips below Z3, so perhaps I can speed up a little.

Tonight, I ran faster up the Belair Hill Climb than I had ever done so before. Not only that, but each Strava segment was faster, too. More importantly, my Strava “Suffer Score” was only 44, as compared to a 62 on my previous PB up the hill. When done, I was all primed to then run some more hill climbs (6x600m uphill), but it started to rain, and it was time for dinner.

Perhaps the only thing that’s missing from Strava for me was the ability to track my weight: Garmin Connect has it’s “Health” tab, which enables you to enter a weight manually, or accepts data from a supported scale. This information is useful to me, as I can see that my weight increased significantly over the leadup to NTL, where I was training much harder, but obviously bulking up a bit too. Lots more speed and strength work there: I do recall having pants that no longer fit my thighs. I’m now down to 78.1kg, after a high of 84.6, and I’d love to be able to import this data into Strava too.

Something that even Garmin Connect doesn’t do, and which I need to keep Garmin Training Center around for is the advanced workouts. Oh, you can enter them into Garmin Connect, but the interface is slow and clunky, and I was never able to get more than one to upload to my watch at a time. Not that useful when I was in a more free-form mode, and would pick workouts based on how I felt. Now, I have a pre-programmed schedule for the next 20-odd weeks, all stored in there. I think I’ll look at a web-app that improves on the process though, as GTC is a bit rubbish.

Oh, and I have my eye on the Garmin Forerunner 610. Not sure when I will get around to upgrading. The 405cx still works really well for my needs, but there are a few nice new features in the 610.

Collection+JSON Primer (and comments)

2012-03-10T17:46:59+10:30

Collection+JSON, created by Mike Amundsen, is a standard way of creating hypermedia APIs. There were a few things I didn’t pick up correctly reading through his great book, or the spec.

First, let us look at a partial example document.

{
  collection: {
    version: "1.0",
    href: "http://api.example.com/",
    links: [],
    items: [],
    queries: [],
    template: {},
    error: {}
  }
}

I’m not so keen on having the version number in the document itself, as this refers to the version of Collection+JSON, rather than the version of the document. In my mind, the version of Collection+JSON should be contained within the media-type (Content-Type: application/vnd.collection+json;version=1.0), just as the version of the document is contained within the Etag header (Etag: 026e10f644ba4b06). Anyway, I’ll let that slide for now.

Secondly, having the href of the collection seems a little superfluous. I’m assuming there will always be an entry in links that has a rel=self, which should give you the same value. Again, not a big issue.

What I was a little unclear on was the difference between links and queries. We can have a look at a couple of examples:

links: [
  {href: "http://api.example.com", rel: "self", prompt: "Home", name: "home", render: "link"},
  {href: "http://api.example.com", rel: "users", prompt: "Users", name: "users", render: "link"}
],
queries: [
  {href: "http://api.example.com", rel: "search", prompt: "Enter search string", data: [
    {name: "search", value: ""}
  ]}
]

The difference between links and queries to me seems somewhat artificial. Sure, in this case, my query has data fields, but it seems that this is not always necessary. The example Mike uses in his book:

queries: [
  {href: "...", rel: "all", prompt: "All tasks"},
  {href: "...", rel: "open", prompt: "Open tasks"},
  {href: "...", rel: "closed", prompt: "Closed tasks"},
  {href: "...", rel: "date-due", prompt: "Date Due", data: [
    {name: "dateStart", value: "", prompt: "Start Date"},
    {name: "dateStop", value: "", prompt: "Stop Date"}
  ]},
]

I’m not quite sure when I should be using a link, and when I should be using a query? In this example, it looks like a query is a filter on the collection: maybe that is the difference?

The other sticking point I have is that both queries and lists are GET requests: the data attribute is simply the query string applied to the URL. Before I continue, we need to look at the items and template attributes of the collection object. In this case, we have a single-object collection, including a write template for it.

items: [
  {
    href: "...",
    data: [
      {name: "first_name", value: "Matthew", prompt: "First name"},
      {name: "last_name", value: "Schinckel", prompt: "Last name"},
      {name: "email": value: "matt@schinckel.net", prompt: "Email address"},
      {name: "gender", value: "male", prompt: "Gender"}
    ]
  }
],
template: {
  data: [
    {name: "first_name", value: "Matthew", prompt: "First name"},
    {name: "last_name", value: "Schinckel", prompt: "Last name"},
    {name: "email", value: "matt@schinckel.net", prompt: "Email address", regexp: "^[^@]+@[^@]+\.[^@]+"},
    {name: "gender", value: "male", prompt: "Gender", options: [
      {value: "male", text: "Male"}, 
      {value: "female", text: "Female"}
    ]}
  ]
}

Again, we see duplicate information. In this case, the template is populated: if it were a ‘proper’ collection rather than a single object, the template would be used for creating new objects in the collection, so I’m prepared to let this one go. You’ll also notice that I’m using a couple of undocumented features: regexp and options. These enable us to either present a list of choices to the user, or have client-side validation based on a regular expression.

To update an object, we can use a PUT (or POST) to the object’s href, and we send the name/value parts of the updated template data:

{
  template: {
    data: [
      {name: "first_name", value: "Matthew"},
      {name: "last_name", value: "Schinckel"},
      {name: "gender", value: "male"},
      {name: "email", value: "matt@schinckel.net"}
    ]
  }
}

To create a new object, we send the same type of data in a POST request to a collection’s href. To delete an object, we can send a DELETE request to the object’s href.

Finally, we come to the error property. I wrote last night how I think this is a little limiting: Collection+JSON error objects. Anyway, an error looks like:

error: {
  title: "Error saving your details",
  code: "409",
  message: "Your date of birth is invalid (19977-11-30 is not a valid date)"
}

After writing most of this, I did come across Collection+JSON – Examples, but I may have described it in a slightly different manner. It still doesn’t elaborate on the difference between links and queries, however.

Collection+JSON error objects

2012-03-09T17:14:48+10:30

I’m still keen on the idea of implementing a rich hypermedia API based on django’s forms.

One of the nicest things about the django forms is that they handle the validation of incoming data. Each form field has a .clean() method, which will clean the data. On a form, it is then possible to have extra methods, .clean_FIELDNAME(), which will process the incoming data again, meaning you don’t need to subclass a field to add simple cleaning functionality. Finally, the form itself has a .clean() method, that can be used to clean composite data, say, ensuring that start is before finish.

The form validation code will create an errors property on the form, that will contain the fields that have errors, and any non-field errors (such as the last example above). When rendering an HTML page, and displaying a form that has errors, these are marked up with CSS classes that enable you to show which fields have invalid or missing data, and also display relatively friendly messages (which you can customise).

But Collection+JSON has a fairly simple error property on the collection object:

{
  "collection": {
    "error": {
      "title": "Error saving your details",
      "code": "409",
      "message": "Your date of birth is invalid (19777-11-30)"
    }
  }
}

Compare this to the format I have been using for JSON responses:

{
  "message": "Error saving your details",
  "detail": {
    "date_of_birth": "The value '19777-11-30' is not a valid date."
  }
}

Programmatically, this allows me to attach the error messages to where they belong: the message value is shown in the main messages area of the client, the detail values for each field are attached to the fields for which they apply.

Django and Collection+JSON

2012-03-09T13:16:52+10:30

Recently, I have been reading (and re-reading) Building Hypermedia APIs with HTML5 and Node. There’s lots to like about this book, especially after reading (and mostly discarding) REST API Design Rulebook.

There is one thing that bugs me, and that is the way that templates are used to generate the JSON. As I said to Mike Amundsen:

Something just feels wrong about using templates to generate JSON, even collection+JSON /cc @mamund
— Matthew Schinckel (@schinckel) March 8, 2012

His response was that he sometimes used JSON.stringify, at other times templates. But it got me thinking. I have written lots of code that serialises Django models, or more recently forms into JSON and other formats. Getting a nice Collection+JSON representation actually maps quite nicely onto these django constructs, as we often have the metadata that is required for the fields.

Consider the following (simple) django model:

class Article(models.Model):
    title = models.CharField('Title of Article', max_length=128)
    content = models.TextField('Content of Article')
    author = models.ForeignKey('auth.User', verbose_name='Author of Article')
    
    @permalink
    def get_absolute_url(self):
        return reverse('article_detail', kwargs={'pk', self.pk})

I don’t normally supply verbose_names, but I have in this case. We’ll see why in a minute.

Now, what I would declare is the obvious JSON representation of this is something like:

{
  "title": "Title goes here",
  "content": "Content goes here",
  "author": 1,
  "href": "…"
}

But, I’m quite interested in Collection+JSON. We might see something like:

{
  "collection": {
    "version": "1.0",
    "href": "…",
    "links": [
      {"href":"…", "rel":"…", "prompt":"…", "name":"…", "render":"string"}
    ],
    "items": [
      {
        "href": "…",
        "data": [
          {"name":"title", "value":"Title goes here", "prompt":"Title of Article"},
          {"name":"content", "value":"Content goes here", "prompt":"Content of Article"},
          {"name":"author", "value":"1", "prompt":"Author of Article"},
        ],
        "links": []
      }
    ]
  }
}

From a django ModelForm, we should be able to easily generate each member of items:

links = getattr(form, 'links', [])
return {
    "data": [
        {"name":f.name, "prompt":f.label, "value":f.value()} for f in form
    ],
    "href": ,
    "links": links
}

The only bit that we are missing out of the form/field data is data type, or more specifically in this case, the available choices that are permitted for the author field. Now, this is missing from the Collection+JSON spec, so I’m not sure how to handle that.

I think this is actually an important problem: if we have a discoverable/hypermedia API, how do we indicate to the client what are valid values that can be entered for a given field?

For those not familiar with django: the verbose_name on a model field is used for the default label on a form field. If you were not using a model, you could just supply a label in the form class.

The other part that is a little hard to think about now are the other attributes: href, and links. Now, these may actually coalesce into one, as links.self.href should give us href. Perhaps we have to look on the form object for a links property. But, in django, it’s not really the domain of the form to contain information about that. For now, I’m going to have a links property on my forms, but that feels dirty too.

Steve Jobs, Enid Blyton and my mother

2012-02-13T20:25:11+10:30

We watched the movie Enid the other week. I read lots of Enid Blyton books as a child, and really enjoyed them. This movie really pushed home how, whilst she had an amazing impact on, and connection with, millions of children, she really didn’t have a very good connection with her own. My tip is, if you don’t know much about her life, but enjoyed her work, don’t watch the movie. Whilst it was excellent, it really sours the memory of her books.

Similarly, I read Walter Isaacson’s biography Steve Jobs recently. Unlike what seems like everyone else, I actually quite enjoyed it. Sure, there may have been some factual errors, and maybe it could have been a much better book, but I felt it did give me a lot of insight into the man that I never had up until that point. I would like to have known more about the NeXT years, but it still contained a lot of what was to me new information.

Interestingly, my mind drew a lot of parallels between these two people, lots of them coming after the fact as I finally got around to listening to all of the 5by5 podcasts discussing the book. The main similarity for me was that these two people had huge impacts on lots of people, but failed to connect effectively with their own children.

Which brings me to my mother. It was her 60th birthday on the weekend, and I gave a short, crappy speech. What I really wanted to say really only crystalised in my mind after a couple of other people had spoken, and I had some time to think about it.

My mum worked for many years running the child day care centre in Naracoorte, and whilst she didn’t touch quite as many childrens' lives as Jobs and Blyton, the number of children she had a significant impact on was by no means small.

The difference was, she still managed to have a great connection with her children.

Adding data to admin templates

2012-02-10T11:41:09+10:30

It came up in the #django IRC channel the other day about how to extend a django admin template to show other information, possibly related to an object, but not necessarily editable.

I use this in production: we have a Company object, which has Location objects associated with it. The django validation is stricter than the data may have been created for these objects, so from time to time a field is missing, and the django admin will not allow saving it.

So, I wanted to be able to display some information about each related object, with links to various bits and pieces. Having the inline Location data is great, except for when it is missing something, that we may not have received from the customer yet.

The trick is that you’ll need to override the admin template for that model.

In this case, our class is in app_name.ModelName, so we need to put the following structure into our template directory:

    templates/
      admin/
        app_name/
          modelname/
            change_form.html

Within that file, I have the content (spaces between % and {,} are there because I can’t remember how to escape them in Liquid Templates…):

{ % extends "admin/change_form.html" % }
{ % block after_related_objects % }
  ... the extra stuff is here ...
{ % endblock % }

In my case, I have the following html structure, and it looks nice:

<div class="inline-group">
  <h2>Units</h2>
  <table width="100%">
    <thead>
      <tr>
        <th>Name</th>
        ...
      </tr>
    </thead>
    <tbody>
      ... loop through stuff here ...
    </tbody>
  </table>
</div>

The other trick is that the admin change view gives us an object, called original, which we can use to do lookups on related objects and the like.

The django admin is awesomesauce, and does most of what I need an administration interface to do. There are lots of places where you do need to extend it, and this is just one way of doing that.

Bing Boy website deployed

2012-02-07T00:32:38+10:30

Just deployed the Bing Boy website: www.bingboy.com.au. This is actually something that I’ve been working on for a long time.

Jaq did the design, and I put together the site using jekyll. Actually, Jaq did the design for everything, right from the logo and all of the print materials, right up to the shop layout and fitout. I believe she even named some of the menu items!

So, pop along, have a look. If you live here in Adelaide, then make sure you check them out, they are super tasty. My favourite is Pretty n Peking. There are two stores so far, and more opening soon: Myer Centre and Southern Cross.

In case the site has changed, here is what it looked like when deployed:

Pre-validating Many to Many fields.

2012-02-06T23:18:56+10:30

Django’s form validation is great. You can rely on it to parse data that you got from the user, and ensure that the rules you have implemented are all applied. Model validation is similar, and I tend to use that in preference, as I often make changes from outside of the request-response cycle. Indeed, I’ve started to rewrite my API framework around using forms for serialisation as well as parsing.

One aspect of validation that is a little hard to grok is changes to many-to-many fields. For instance, the part of the system I am working on right now has Tags that are applied to Units, but a change to business requirements is that these tags need to be grouped, and a unit may only have one tag from a given TagGroup.

Preventing units from being saved with an invalid combination of Tags is simple if you use the django.db.models.signals.m2m_changed signal.

from django.db.models.signals import m2m_changed
from django.dispatch import receiver

@receiver(m2m_changed, sender=Tag.units.through)
def prevent_duplicate_tags_from_group(sender, instance, action, reverse, model, pk_set, **kwargs):
  if action != 'pre_add':
    return
  
  if reverse:
    # At this point, we know we are adding Tags to a Unit.
    tags = Tag.objects.filter(pk__in=pk_set).select_related('group')
    existing_groups = TagGroup.objects.filter(tags__units=instance).distinct()
    invalid_tags = set()
    for tag in tags:
      if tag.group in existing_groups:
        invalid_tags.add(tag)
      group_count = 0
      for other_tag in tags:
        if other_tag.group == tag.group:
          group_count += 1
      if group_count > 1:
        invalid_tags.add(tag)
    if invalid_tags:
      raise ValidationError(_(u'A unit may only have one Tag from a given Tag Group'))
  else:
    # At this point, we know we are adding Units to a Tag.
    units = Unit.objects.filter(pk__in=pk_set)
    group = instance.group
    invalid_units = []
    for unit in units:
      if unit.tags.exclude(pk=instance.pk).filter(group=group).exists():
        invalid_units.append(unit.name)
    if invalid_units:
      raise ValidationError(_(u'The unit%s "%s" already ha%s a Tag from group "%s"' % (
        "s" if len(invalid_units) > 1 else "",
        ", ".join(invalid_units),
        "ve" if len(invalid_units) > 1 else "s",
        group.name
      )))

Now, this on it’s own is nice enough. However, if you try to save invalid data from within the admin interface, then you will get an ugly trackback. If only there was a way to get this validation code to run during the validation phase of a form. i.e., when you are cleaning it…

So, we can create a form:

from django import forms
from models import Tag, prevent_duplicate_tags_from_group
class TagForm(forms.ModelForm):
  class Meta:
    model = Tag
    
  def clean_units(self):
    units = self.cleaned_data.get('units', [])
    if units:
      prevent_duplicate_tags_from_group(
        sender=self.instance.units,
        instance=self.instance,
        action="pre_add",
        reverse=False,
        model=self.instance.units.model,
        pk_set=units
      )
    return self.cleaned_data

You can create a complementary form on the other end (or, if you already have one, then just hook this into the field validator). The bonus here is that the validation errors will be put on the field with errors, in this case units.

Modular django settings

2012-02-01T21:21:15+10:30

A recurring feature of #django is that someone asks about settings.py, and using a local_settings.py file. The standard advice is to have the following in your settings.py:

# More settings are above here.

try:
  from local_settings import *
except ImportError:
  pass

This is usually the last (or one of the last) things in the file. This can be used to override settings with sane values for the local environment.

However, this means that local_settings.py must be not in your source control system, or must not be deployed to other servers.

I like keeping everything in my source control system of choice (mercurial), and currently use an hg-based deployment. In my fabfile.py, instead of archiving up the current structure, I use hg to push the main repo, and any sub-repos, and update them to the version that is displayed locally.

This means I want to be able to control the content of production’s local_settings.py equivalent.

The other issue, and this was the one that came up today and gave me the idea of this post, is that someone wanted to add an app to settings.INSTALLED_APPS but only locally. I too have done this (still do, with django-devserver, amongst others).

I came up with the following solution. Instead of having a settings.py and local_settings.py, I have a settings module:

settings/
    __init__.py
    base.py
    development.py
    production.py
    testing.py

base.py contains what was normally in your main settings.py file. That is, settings that are common to all environments.

In development.py, production.py and testing.py, I have the following line at the top:

from base import *

Then, in each of those files, where I need to override or alter a setting, including appending to a list or tuple, I can just modify away. Some things that I do in development.py, for instance:

from base import *

DEBUG = True

DATABASES['default']['HOST'] = '127.0.0.1'

INSTALLED_APPS += (
  'devserver',
  'test_extensions',
  'test_utils' # really only for makfixture.
)

import getpass
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
EMAIL_FILE_PATH =  PROJECT_ROOT / 'log' / 'email-messages-%s' % getpass.getuser()

This shows how you can set a value, alter a value of a dict given a specific key, and append to a tuple. I also have a nice little setup where I use a value set in the base.py file (PROJECT_ROOT) to determine where I want to log email messages to.

Finally, you need some way to say which of these files should be used. This is all done in __init__.py:

servers = {
  'development': [
    'darwin', 'boyd', 'arne'
  ],
  'testing': [
    'testing', 'debian'
  ],
  'production': [
    'staging', 'vps1', 'vps2', 'vps3'
  ]
}

def get_server_type():
  from socket import gethostname
  server_name = gethostname()
  for server_type, names in servers.items():
    if server_name in names:
      return server_type
  
  return 'production' # Or whatever you want the default to be.
                      # I usually have 'testing' here, because I tend to
                      # spin up test servers. If you spun up production
                      # servers lots, you'd use that.

exec("from %s import *" % get_server_type())

This method does require a little bit of maintainence: when you have a new server name, you need to add an entry to this file. If you are often creating testing servers (like I am) then you might want to use testing as the default server type.

Alternatively, you could use some sort of prefix to mean a particular server type.

Anyway, that’s how I do it. The only drawback is that it does mean that your SECRET_KEY, and any passwords you might have defined in settings.py are stored in your repository. We aren’t that fussed about that right now: our project is closed source, and only trusted people have access to the repository.

Re: HTML is not XAML

2012-01-26T22:37:57+10:30

This is a response to MVVM on MVC: HTML is not XAML. I attempted to post a comment, first from my iPad, but after a few characters the textarea became unresponsive. Then, from my iMac, I was able to enter a comment, but not post it. It seems like something weird is happening when it shows me the CAPTCHA, and it dismisses the dialog before I can do anything. Disabling JavaScript prevents me from commenting. (But to be honest, using DISQUS does the same).

I’ll start by saying that I’m not a Silverlight developer, indeed, I do nothing with any part of the Microsoft development stack. I don’t think I even have any Microsoft software installed on either my work or home machines. I have been doing a lot of stuff with KnockoutJS lately, though.

Jeremy makes some valid points about designer ↔ developer interactions. Maybe I’m (un)lucky, but my interactions with a designer seem to be that (s)he provides me with an image, and I make the HTML match. Either that, or I do the design work myself. In that case, I design in the browser. Safari’s inspector allows you to alter CSS rules and view their impact live. This also means that my HTML is always as sparse as I can possibly make it.

Before I get to the main point, regarding using bindings inside the HTML, there is one thing I just need to point out. Jeremy has the code:

<div id="menuContainer">
  <ul id="menu">
    <li data-bind="foreach: menuItem">
      <span data-bind="text: name">Name</span>
    </li>
  </ul>
</div>

This would only create one <li> element, with multiple <span> elements. In addition, the Name text is superfluous, and would be replaced when the bindings were applied. To match the intent of the previously listed code, I think he meant (and I’m spelling some stuff differently):

<nav class="main-menu">
  <ul data-bind="foreach: menuItems">
    <li>
      <a data-bind="text: name, attr: {href: url}"></a>
    </li>
  </ul>
</nav>

Jeremy then goes on to discuss a way to have all of these bindings applied using code.

Personally, being able to bind the data declaratively is one thing that really draws me to KnockoutJS. It’s easy to see the HTML structure, and what data is bound to them. In fact, in some ways it reminds me lots of Cocoa Bindings.

One of his beefs is that designers may muck with the code. I think this could be easily remedied by a little education: don’t touch anything that has data-bind="...". This really isn’t that different to don’t touch anything’s id.

But a deeper problem is that by adding the bindings in code means that you can’t see from the HTML what flow control will be used to handle the layout. Assuming you are still able to apply the foreach binding to the ul in the example above if it had an id instead, it’s not obvious that there may be multiple items. Maybe that’s not the greatest example, as it is a list, so there probably will, but foreach can be used anywhere.

And there are more, too: if allows you to have bits that are not rendered (significantly different to just making them invisible). Plus, if you use the with binding, then you would need to keep in your head the nested structure of what you are accessing inside the with block. Do it in HTML, and you can see (assuming you have reasonable indenting).

Jeremy seems to come to the agreement (in one of his comments), that having the application of the bindings in the code makes things even more complicated, but I propose that it even makes them more brittle. No longer are you relying just on the names of the ViewModel’s attributes, but you are also relying on the ids of the HTML elements. And this is the kicker: a binding to a name that no longer exists in the ViewModel will fail when you try to view the page, meaning nothing will work (and you’ll see a nice red message in your console). What will the following code do if there is no matching HTML element?

$('#spelled-this-id-worng').attr('data-bind', 'value: theValue');

It does nothing.

But it does it silently.

Styling radio buttons like a boss segmented button

2012-01-23T21:39:03+10:30

I quite like the concept of segmented buttons, where you have a list of related buttons, and can select one or more of them. In MacOS X, and iOS, the ones that are selected have a nice indented look.

I’m currently working on a GUI framework for KnockoutJS, and today I had reason to use this type of control. Initially, I had the following markup that I intended to style:

<nav>
  <ul class="segmented">
    <li><a>Organisation</a></li>
    <li><a>Users</a></li>
    <li><a>Units</a></li>
    <li><a>Tags</a></li>
  </ul>
</nav>

But, then it occurred to me that HTML radio buttons are a good fit for this use case. They can be set so that only one of them will be selected, which means you can actually get them to work without using any JavaScript to keep selected status in sync. And the bonus is that the labels will be clickable, so we don’t need JavaScript for associating them with the radio buttons.

<nav class="segmented">
  <input type="radio" name="seg-1" value="Organisation" id="seg-Organisation">
  <label for="seg-Organisation">Organisation</label>
  <input type="radio" name="seg-1" value="Users" id="seg-Users">
  <label for="seg-Users">Users</label>
  <input type="radio" name="seg-1" value="Units" id="seg-Units">
  <label for="seg-Units">Units</label>
  <input type="radio" name="seg-1" value="Tags" id="seg-Tags">
  <label for="seg-Tags">Tags</label>
</nav>

Now, there’s slightly more markup, but that’s okay. So, what does that look like?

Hmm, not quite what we want. We’ll actually want to hide the radio button widgets, and style the labels. Rather than try to have this in the page, here’s one I prepared earlier:

As you can see, this is with no JavaScript.

Obviously, this is fairly crappy styling: it just looks like some Windows buttons. Let’s tart it up a bit. This is the default styling for koui:

As a bonus, I’ve disabled one of the elements.

From the perspective of KnockoutJS, we can use the checked binding applied to the radio buttons to see which one is selected. If you were submitting this inside a form, you may want to not use display: none; on them, as they may not submit under certain browsers. For ko, however, it’s fine.

I’m going to be using this technique within koui: for segmented buttons, which I haven’t worked out a nice way to define bindings for, and for the tab_view binding, which associates these buttons with a view below, containing a choice of data based on the selection.

Update: There is one drawback with the technique I used here. It is detailed at How to fix the broken iPad form label issue.

It’s actually a rather simple solution: All you need to do is stop the propagation of the event. I had played around with re-firing the event onto the input element, but that might fire twice in some browsers, which might be bad (especially if you were using checkboxes, rather than radio buttons!)

jQuery long-poll for CouchDB changes.

2012-01-22T22:02:56+10:30

I spent a bit of time this weekend playing with CouchDB. I think I have almost figured out how to do some stuff that had been bugging me. Coming to a document-based store after so much time and effort on relational DBMS does mean you really need to approach things from a different direction.

Our project is basically a multi-tenanted hosted application. Each customer has their own data, and within a relational model you basically have a field in each table that references the customer the data belongs to. Either that, or you have a seperate database and installation per-customer, but that doesn’t scale well on a system like ours that get intermittent use throughout the week.

I’m going to talk more about CouchDB and segmenting the data later, but the best solution is to have a single database per-customer. This makes more sense when you know how CouchDB works: a CouchDB database is a container for documents. Grouping these by customer means you can easily replicate one customer’s data, or move it to a different node.

One really nice feature of CouchDB is the changes feed. From this, you can subscribe to all of the changes that occur in a database, and this could be filtered (so a user would only be notified of changes to documents that they have read-access on, for instance).

This could potentially solve lots of problems that we have with different users from the same company working on the same roster at the same time, and those changes automatically appearing in everyone’s browser.

There are three types of change feeds that are interesting:

The list of changes since I last checked (I send in a sequence number).
The same, but handled using long-polling (I keep the request open until a change occurs).
A continuous polling approach, where changes are sent to my open connection as they occur.

I really liked the sound of the last one, but $.stream, the only library I could find that did that for jQuery had some issues: like the fact is sent the request by POST, and that I couldn’t actually get it to see any data that was coming back in.

To solve the problem using #1, you could write some code that keeps track of the sequence number, and runs a request every X seconds. But I liked the idea of long-polling.

The idea I had was to run a request, and in the success handler, recursively call the function. To actually handle the incoming data, I thought that it might be a good solution to use jQuery events. I’m having them triggered on $(document), as I haven’t seen a standard way to do this. The other option might be on $(window), or a passed-in object.

Fetching iTunes Store Data

2012-01-20T22:07:31+10:30

Or, quite possible the best AppleScript I have ever written.

So, I have a new iMac, and I was moving over a heap of ‘legacy’ media from another machine, and some of it was DVD rips of Scrubs, Mad Men and Big Love. Now, these were mostly well tagged, but often I didn’t have all of the nice descriptions that were in iTunes.

Typing in all of this data is fairly annoying.

AppleScript, jQuery to the rescue:

tell application "iTunes"
    repeat with theTrack in selection
        tell theTrack
            set trackNo to episode number
        end tell
        tell application "Safari"
            set theDesc to do JavaScript "$('span.index span').filter(function() { return parseInt($(this).text(),10) == " & trackNo & "; }).closest('tr').find('.description .text').text();" in document 1
            set theName to do JavaScript "$('span.index span').filter(function() { return parseInt($(this).text(),10) == " & trackNo & "; }).closest('tr').find('.name .text').text();" in document 1
        end tell
        tell theTrack
            set description to theDesc
            set name to theName
            set episode ID to (my replace(show, " ", ".")) & ".S" & my zero_pad(season number, 2) & "E" & my zero_pad(episode number, 2)
        end tell
    end repeat
end tell

on zero_pad(value, string_length)
    set tmp_string to "000000000" & (value as string)
    set padded_value to characters ((length of tmp_string) - string_length + 1) thru -1 of tmp_string as string
    return padded_value
end zero_pad

on replace(content, sub, repl)
    set delims to AppleScript's text item delimiters
    set AppleScript's text item delimiters to sub
    set content to every word of content
    set AppleScript's text item delimiters to repl
    set content to content as text
    set AppleScript's text item delimiters to delims
    return content
end replace

You just need to have the relevant TV Series Season page in the frontmost window in Safari, and select all of the episodes for that season in iTunes. Then, run the script, and watch the magic.

Get decorators wrapping a function

2012-01-20T22:00:18+10:30

I mentioned a week or so ago about my django templatetag that will only display a menuitem or link if the logged in user has access to the view that it points to. In passing, I stated how it was rather complicated to do this test.

The complicated bit is finding out all of the decorators that are wrapping a given function. Fortunately, python is a dynamic language, and this type of introspection, whilst not completely simple, is possible.

The key lies in the property that all function objects in python have: func_closure. According to the python docs:

func_closure is None or a tuple of cells that contain binding for the function’s free variables

Depending upon where you read this, it may or may not be writable. Luckily, we don’t need to be able to write to this, only read from it.

If func_closure is not None, then it will be a tuple of <cell> objects. To do anything useful, we’ll need to look at their cell_contents attribute. If that is callable, then it’s a good candidate for a decorator.

Because of the way decorators work, if you have multiple decorators on a function, each one wraps the next one. Thus, we’ll need to have some recursion in there.

At a first draft, we might end up with something like:

def get_decorators(function):
  # If we have no func_closure, it means we are not wrapping any other functions.
  if not function.func_closure:
    return [function]
  decorators = []
  # Otherwise, we want to collect all of the recursive results for every closure we have.
  for closure in function.func_closure:
    decorators.extend(get_decorators(closure.cell_contents))
  return [function] + decorators

It’s important that we return the original function in the base case of the recursive function, as eventually, every closure will fold down to a single callable. It has the side effect that get_decorators will get all of the decorators, and the function they ultimately wrap.

You could probably also do this as a generator.

For basic django function views, this will work fine. For class-based views, we need to do something a little extra.

In the case I was writing this function for, I knew it would only be looking for a get() method on the class-based view, which makes things a little simpler. That, and the dispatch() method were the only places I would need to look on the class for decorators. Also, I only wanted decorators that were callable: since I would actually call a subset of them to test if the user could access the view.

def get_callable_cells(function):
  callables = []
  # Under some circumstances, I wound up with an object that has the name `view_func`: 
  # this is the view function I need to access.
  if not hasattr(function, 'func_closure'):
    if hasattr(function, 'view_func'):
      return get_callable_cells(function.view_func)
  # If we have no more function we are wrapping
  if not function.func_closure:
    return [function]
  for closure in function.func_closure:
    contents = closure.cell_contents
    # Class-based views have a dispatch method
    if hasattr(contents, 'dispatch'):
      callables.extend(get_callable_cells(contents.dispatch.__func__))
      if hasattr(contents, 'get'):
        callables.extend(get_callable_cells(contents.get.__func__))
    callables.extend(get_callable_cells(contents))
  return [function] + callables

The other trick I’m using there is to use the __func__ property of the dispatch and get methods of the view class.

From there, in my case, I wanted those callable cells that look like they take a user as their first argument:

def get_tests(function):
  return [
    x for x in get_callable_cells(function)
    if x.func_code.co_varnames[0] in ["user", "u"]
  ]

So, this works for things that look like:

class ViewClass(ParentClass):
  @login_required
  @permission_required('app_label.permission_name')
  def dispatch(self, *args, **kwargs):
    return super(ViewClass, self).dispatch(*args, **kwargs)
  
  @user_passes_test(lambda u: u.is_staff)
  def get(self, *args, **kwargs):
    # do stuff here
    pass

As well as:

@login_required
@permission_required('app_label.permission_name')
@user_passes_test(lambda u: u.is_staff)
@render('foo/bar.html')
def view_function(request, *args, **kwargs):
  # do stuff here

It will pick up that the @render decorator is not asking for a user.

It works with the @user_passes_test decorator, because that is passed a function that has the first argument of u. It works with the @permission_required decorator for basically the same reason, although that has a function inside a function that actually has the argument of user.

Finally, it works for the @login_required decorator, as that calls user_passes_test.

KnockoutJS dirty extender.

2012-01-14T20:14:51+10:30

Ryan Niemeyer is the man with respect to most things KnockoutJS, and I had been using a version of his smart dirty flag in some projects. I recall making it so it didn’t have to bind to a secondary property, but I may be mistaken.

Anyway, with Knockout 2.0, we get extenders. Now, it’s possible to do things like:

var thing = ko.observable(null).extend({dirty: true});

It will then look for ko.extenders.dirty, and call that function with two arguments: the observable, and the argument (in this case, true).

Thus, it is possible to re-implement his dirty flag as an extender:

ko.extenders.dirty = function(target, startDirty) {
  var cleanValue = ko.observable(ko.mapping.toJSON(target));
  var dirtyOverride = ko.observable(ko.utils.unwrapObservable(startDirty));
  
  target.isDirty = ko.computed(function(){
    return dirtyOverride() || ko.mapping.toJSON(target) !== cleanValue();
  });
  
  target.markClean = function(){
    cleanValue(ko.mapping.toJSON(target));
    dirtyOverride(false);
  };
  target.markDirty = function(){
    dirtyOverride(true);
  };
  
  return target;
};

The advantage I think mine has over Ryan’s is that you can mark an observable as dirty (thing.markDirty()), and it will stay dirty until you explicitly mark it as clean (thing.markClean()).

Otherwise, it’s just: thing.isDirty() and you are all good.

Alternatively, you could remove the two helper functions, and implement .isDirty() as a writeable observable, that tests the incoming value and sets the internal cleanValue if it needs to.

More thoughts on HATEOAS

2012-01-14T13:21:49+10:30

Last night I wrote some stuff about REST, user access and HATEOAS. After I wrote it, I started planning out what our application would look like if it were structured this way. This morning, as I prepared Weetbix for my little boy, I had a bit of an epiphany.

You could use HATEOAS with a non-javascript HTML web-site. It becomes less ‘dynamic’ in the sense that it is page → click → next page. But each hyperlink represents a change of state: either of what the user is viewing (on a GET request), or what the system is storing (on a PUT/POST/DELETE).

I had been thinking about it from a ‘rich client’ perpective, whether that was in a browser (and loading pages using $.ajax() or similar), or what we have now (a wxWidgets application). But everything makes sense from a pure HTML perspective too. In fact, you can automatically generate links and everything, if you want.

This made me think a bit more about how to define what the system does (or more specifically, what the system can and should do when). One of the problems we have now is that it is such a big system that it’s hard to know what is happening when. I had thought that the whole ‘Use Case’ concept taught in Software Engineering courses was mostly bollocks, but I can see some value in parts of it.

Flow diagrams suddenly make sense, because you need to define what possible things can happen from any given state: what other system states are available to that user at that time.

For instance, a user has logged in, and is viewing their own user details. The things they can do are:

update their own details
change their password
exit viewing their own details

This maps nicely onto link relations, and the resource they are viewing becomes:

  {
    "_links": {
      "self": {"href": "..."},
      "edit": {"href": "...", "title": "Save changes"},
      "change-password": {"href": "...", "title": "Change Password"},
      "up": {"href": "...", "title": "Exit"}
    },
    "formats": {
      "date": "%Y-%m-%d",
      "time": "%H:%M:%S",
      "name": "%(first_name)s %(last_name)s"
    }
  }

If they are not permitted to perform an action: say they may not edit their own options, then the edit link is simply not there. The client could then use this to know that the data is not editable, and only represent a static text view of it, instead of editable fields. I am a big fan of stopping the user from performing an action, rather than failing.

Now, a (naïve) pure html web page could look like (I’ve only included the interesting parts):

<form method="put" action="...">
  <input name="formats:date" value="%Y-%m-%d">
  <input name="formats:time" value="%H:%M:%S">
  <input name="formats:date" value="%(first_name)s %(last_name)s">
  <input name="edit" type="submit">Save Changes</input>
</form>
<a href="...">Change Password</a>
<a href="...">Exit</a>

A more sophisticated client would probably want to have limited choices for those three formats. If it were still pure HTML (ie, generated by the server that will be handling it), then it would know about the available choices for those fields, and you could have something like:

<form method="put" action="...">
  <select name="formats:date">
    <option value="%Y-%m-%d" selected>ISO 8601 (2012-01-26)</option>
    <option value="%b %d, %Y">Long (January 26, 2012)</option>
    <option value="%m/%d/%Y">US (26/01/2012)</option>
    <option value="%d/%m/%Y">Australian (01/26/2012)</option>
    ...
  </select>
  ...
  <input name="edit" type="submit">Save Changes</input>
</form>
<a href="...">Change Password</a>
<a href="...">Exit</a>

But how do we provide these choices to a non-pure HTML client? Surely we don’t want to embed them in the resource: that seems wasteful, especially if they are unlikely to change often. Why, we can have a resource that contains the choices:

[
  {
    "value": "%Y-%m-%d",
    "title": "ISO 8601"
  },
  {
    "value": "%b %d, %Y",
    "title": "Long"
  },
  {
    "value": "%m/%d/%Y",
    "title": "US"
  }
]

Note here that I am relying on the rich client to provide the example, based on today’s date, probably. That means we can cache this more.

But where do we link to this?

I’d probably extend it so that all of the formats appear in one resource:

{
  "date": [...],
  "time": [...],
  "name": [...]
}

And then have a link:

{
  "_links": {
    "self": {"href": "..."},
    "edit": {"href": "...", "title": "Save changes"},
    "change-password": {"href": "...", "title": "Change Password"},
    "up": {"href": "...", "title": "Exit"},
    "choices:formats": {"href": "..."}
  },
  "formats": {
    "date": "%Y-%m-%d",
    "time": "%H:%M:%S",
    "name": "%(first_name)s %(last_name)s"
  }
}

Indeed, it might be even more complicated than that: with the name format, for instance, and even with the date and time, we could provide some canned types, and “Custom”, allowing the user to use the valid formatting token to make their own. But that’s another story.

link reltypes

What is still part of the same story, however, is how to know what verb types should be used for which links, and what data should be sent in the case of PUT/POST requests. I’ve been sitting down working some stuff out about this, and come up with some conventions I think I’ll look at using.

rel=self

"self": {"href": "...", "title": "User Details"}

rel=self means that this is the address of the resource that is currently being used. We could look at using the title attribute to mean that ‘this is a good title for the current place of interaction within the system’. That also means we can have a different title for the same resource, depending upon other stuff in the system, for instance.

rel=edit

"edit": {"href": "...", "title": "Save changes", "data": {
    "editable-field-name": {
      "format": "string",
      "required": true,
      "multiple": false
    }, ...
  }
}

rel=edit means that this is the URI that should be PUT to, with what the client desires the resource should be. Note that I have played around a bit and added in an optional attribute: data, which contains an object with every editable field on the source object. If this field is missing, then the implication is that all fields are editable.

This also can be used to provide the client with information as to the data type, and if it is required and/or must contain an array of objects. The formats would be string, number, boolean. I still haven’t worked out how you might include an object here: the format could be object, but then you might need to define further what should go there.

Requesting this link with a PUT does not ‘move’ the application location: essentially you are still viewing the same object. A client may choose to follow the rel=up link after a successful update, however.

rel=delete

"delete": {"href": "...", "title": "Delete"}

This link can be used to delete an object. I’m not sure what exactly should happen after a deletion: we could have some method of undo that only works until the user visits another link:

{
  "_links": {
    "undelete": {"href": "...", "title": "Restore"}
    "up": {"href": "...", "title": "..."}
  }
}

rel=up, rel=root

"up": {"href": "...", "title": "Back to <...>"}

This link type would be used to move up a level in the application ‘hierarchy’. I think it would need to be present in every resource, except the root resource.

Perhaps we could also have:

"root": {"href": "...", "title": "Home"}

Then we could always jump quickly back to the root resource, without having to navigate through many layers.

rel=[other]

Every other link might be a domain-specific link. For instance, the change-password link in the example above. But we need a way to handle what http verb should be used.

Here is an example of how I think you could indicate that this should be a POST link:

"change-password": {
  "href": "...", 
  "title": "Change password",
  "data": {
    "old-password": {"format": "string", "required": true},
    "new-password": {"format": "string", "required": true}
  }
}

By having the data attribute, we prevent the need for having a GET request to a form resource, and then a POST from that resource to its rel=edit. However, there may be some value in having that step (it matches up with web page navigation, for instance).

The way I would handle it is that every link with a data attribute is essentially defining its form data, which could be used to construct multiple HTML forms in the one page.

It would be domain specific what happens after this request has been executed. In this case, you would probably want to stay on the user details page, but just flash a message that the password had been changed (or not).

Which brings up how to reply with messages in this context. I am tempted to have a “messages” attribute at the same level as the “links” attribute, but I’m not totally sold on that just yet.

In the cases where you want the link to be followed with a GET, simply omit the data attribute. However, if you want to be able to have query parameters, you could use:

"posts": {
  "href": "...",
  "title": "Posts",
  "param": {
    "date": {"format": "date"},
    "tag": {"format": "string", "multiple": true}
  }
}

In this case, neither of those parameters are required, but tag can be supplied multiple times. The IANA Link Relations document seems to imply that rel=search should be used for all search stuff, but I have tried to avoid having multiple links of the same reltype. Having that means that instead of being able to use _links.reltype.uri, you would need to use _links.reltype.uri || _links.reltype[0].uri, or something like that.

versioning resources

There are also some nice bits and bobs in the IANA Link Relations document about how versions of resources can refer to one another. Having a working copy is a nice idea, and would allow you to aggregate a series of changes together, while still allowing you to have business logic (and essentially saves) happening. You can then commit a working copy to create a new version.

I’m still a bit anti-locking of resources, as the danger is that a user might lock a resource from one machine, and the commit/discard request may never arrive, resulting in the version remaining locked, even when the same user attempts to change it again. I have done some stuff with merging conflicts on a 412 response.

The two options you have if you discard locking are:

There may be multiple working-copy versions of a resource. Essentially each user (or each process they are using) would have a seperate working-copy. You would then need to merge conflicts when a subsequent working-copy is committed to the resource.
There may only be one working-copy of a resource. Multiple users may interact with it, and conflicts must be resolved before the working-copy will be updated with that user’s changes.

I think I actually prefer #2. It does mean that two users working on the same object would never be able to independently create their own version: it’s probably a bit more svn than git, but for the business domain I am thinking in, it makes a bit more sense. And it seems to be simpler to merge conflicts as soon as possible.

Again, this just makes me want to write code.

REST, permissions, hypermedia and DRY.

2012-01-13T23:30:57+10:30

I’ve been working through the whole concept of REST over the past couple of years. I really like the fact that in theory the API should be clean, well designed, and resource-based.

One of the first takeaways I got from this whole school of thought is that it becomes the verbs in HTTP that become important. That is, given a resource, say at http://api.example.com/foo/1/, we can use GET on that resource to view it, DELETE to remove it, and PUT (or POST) to update it.

Since I prefer JSON to XML, I’ll use that for the resource representation examples.

We might get the following back when we do a GET on http://api.example.com/user/1/:

{
  "username": "barry",
  "first_name": "Barry",
  "last_name": "Citizen",
  "links": [
    {
      "rel": "self",
      "uri": "http://api/example.com/user/1/"
    }
  ]
}

I like the concept of having a links attribute, that contains links representing changes of state. But one thing that bothered me, from a UI perspective, is that I need to be able to know beforehand if the user has write/delete access on a resource. How can this best be represented?

I thought one way might be to have:

{
  ...
  "links": [
    {
      "rel": "self",
      "uri": "<whatever>",
      "actions": ["get", "put", "delete"]
    }
  ]
}

But I haven’t seen anything that does something similar to this. And, I have been leaning more towards using POST rather than PUT in lots of cases, as the business logic of my project is that often a user will not have access to the full data associated with a resource. PUT implies that the data that is being sent should replace the resource entirely.

For instance, a manager of a shop who has staff that she shares with another shop may not have access to the other shop’s data, and therefore may not know anything about that shop. In fact, we have at least one case where some managers must not know that another particular shop exists. Thus, she would send back an incomplete list of shops that her employee works at, thus removing him from the ones she cannot see.

As an aside, which I discovered while writing this article, there is also some discussion that it is valid to use PUT to update the bits we know about: RESTful architecture: what should we PUT?.

So, is this a valid way of representing what actions are permissable on a URI? Or is there a standard?

I then remembered that, REST in Practice talks about link types. On page 117:

self

The uri value can be used to GET the latest resource representation of the order.

update

Consumers can change the order using a POST to transfer a representation to the linked resource.

cancel

This is the uri to be used to DELETE the order resource should the consumer wish to cancel the order.

There is also a link elsewhere in the book to Link Relations. There, we see instead of an update link, an edit link, but no remove or delete links. And this seems to fly against the whole concept of using the verb, rather than the URL, as in my case, those three URLs would all be the same. Besides, this seems to be a bit anti-DRY.

However, maybe this is the best solution: making the resource:

{
  "username": "barry",
  "first_name": "Barry",
  "last_name": "Citizen",
  "links": [
    {"rel": "self", "uri": "http://api/example.com/user/1/"},
    {"rel": "edit", "uri": "http://api/example.com/user/1/"}
  ]
}

The implication here is that the edit URI can also be used to delete the object, but that still doesn’t solve my issue. If we know the user may not delete a resource, then we can prevent access to the control that would be used to delete it.

Indeed, RFC 5023 specifically talks about the Member URI, which is (a) returned in the Location header when an object is created, and then goes on to talk about sending PUT and DELETE requests to the Member URI. But there is no mention about access control.

So then as I think more about it, HATEOAS is all about navigating around using links to control system state. Thus, it makes sense to have seperate links for edit and delete. And, according to Jim Webber, one of the authors of Rest in Practice:

Side note: it’s best not to conflate links with URIs. A link is a hypermedia control, a URI is an identifier and usually an address, and we’re interested in links with well-known semantics and not very interested in the format of the URIs.

(From the REST in Practice discussion group).

The other concern I have is that it would be nice to know what query parameters we can send to a resource collection to filter it. I have put a bit of thought into using URLs for the rel attribute, and those in turn being a resource that can be read using a GET request. This would then provide information like:

What parameters can be passed to this collection resource to filter it.
What data needs to be sent to this collection resource in order to create a new object resource. This would be field names, and the data types.

Indeed, I started to plan something like this that could be automatically generated (at least the second part of it) from within django-repose. The idea is that the process of registering an ApiView class also automatically registered the reltype url route.

Having said all of that, the code I have been working on lately looks explicitly for rel=self, and uses that to store the URL that should be operated on.

I’m going to end this now: mainly because I have just run across JSON Linking with HAL, and I’m going to re-work how I generate and handle links.

Highlight 'highlight' blocks in Markdown/Textmate

2012-01-11T20:51:08+10:30

The other day, I mentioned that I had Marked.app nicely handling my {% highlight %} blocks, and syntax highlighting them. In passing at the end, I mentioned that TextMate was still formatting them as if they were Markdown.

Now, one way around this is to indent them, but then within the code block they are indented further, and that offends my sensibilities.

Now, within TextMate, syntax highlighting is based on scopes, so to do what I want (which is the same as how HTML may have CSS or JS embedded in it), we just need a language grammar pattern that matches, and applies the relevant scope.

TextMate 2 has even nicer features, where you can set the scope (but not, as it turns out, include rules) dynamically based on a match in the pattern.

Anyway, on to the rules.

Rather than edit the Markdown rules, I wanted to just inject the language grammars in from a bundle of my own, but had no luck with this. Instead, I decided to extend the Jekyll bundle.

This is what I wanted to put in the patterns (simplified a little):

1 {
2   begin = ' "%\}\n';
3   end = ' "%\}\n';
4   name = 'source.$1.embedded.html.markdown';
5   patterns = ( { include = 'source.$1'; } );
6 }

However, as I mentioned above, the expression on line 5 does not actually include source.js patterns in this case.

Instead, I needed to have a seperate pattern for each language I wanted to include patterns from. Since mostly I work in python, html and javascript, for now those ones will do.

Oh, and the last thing is that html needs to include text.html.basic.

You can see my fork at jekyll-tmbundle. The current code is:

Binding to head elements in KnockoutJS

2012-01-10T13:52:05+10:30

By default, KnockoutJS binds to <body>, by the look of things. If you have something like:

<html>
    <head>
        <title data-bind="text: title"></title>
    </head>
    <body>
        ...
        <script>
            var vm = {
                title: ko.observable("This is the page title")
            };
            ko.applyBindings(vm);
        </script>
    </body>
</html>

The title will not be bound. Instead, you’ll need to use (and I’m using jQuery):

<html>
    <head>
        <title data-bind="text: title"></title>
    </head>
    <body>
        ...
        <script>
            var vm = {
                title: ko.observable("This is the page title")
            };
            ko.applyBindings(vm, $('html')[0]);
        </script>
    </body>
</html>

You could also apply the binding twice, once to the head, and once to the body.