Schinckel.net

Last night I wrote some stuff about REST, user access and HATEOAS. After I wrote it, I started planning out what our application would look like if it were structured this way. This morning, as I prepared Weetbix for my little boy, I had a bit of an epiphany.

You could use HATEOAS with a non-javascript HTML web-site. It becomes less ‘dynamic’ in the sense that it is page → click → next page. But each hyperlink represents a change of state: either of what the user is viewing (on a GET request), or what the system is storing (on a PUT/POST/DELETE).

I had been thinking about it from a ‘rich client’ perpective, whether that was in a browser (and loading pages using $.ajax() or similar), or what we have now (a wxWidgets application). But everything makes sense from a pure HTML perspective too. In fact, you can automatically generate links and everything, if you want.

This made me think a bit more about how to define what the system does (or more specifically, what the system can and should do when). One of the problems we have now is that it is such a big system that it’s hard to know what is happening when. I had thought that the whole ‘Use Case’ concept taught in Software Engineering courses was mostly bollocks, but I can see some value in parts of it.

Flow diagrams suddenly make sense, because you need to define what possible things can happen from any given state: what other system states are available to that user at that time.

For instance, a user has logged in, and is viewing their own user details. The things they can do are:

• update their own details
• change their password
• exit viewing their own details

This maps nicely onto link relations, and the resource they are viewing becomes:

  {
    "_links": {
      "self": {"href": "..."},
      "edit": {"href": "...", "title": "Save changes"},
      "change-password": {"href": "...", "title": "Change Password"},
      "up": {"href": "...", "title": "Exit"}
    },
    "formats": {
      "date": "%Y-%m-%d",
      "time": "%H:%M:%S",
      "name": "%(first_name)s %(last_name)s"
    }
  }

If they are not permitted to perform an action: say they may not edit their own options, then the edit link is simply not there. The client could then use this to know that the data is not editable, and only represent a static text view of it, instead of editable fields. I am a big fan of stopping the user from performing an action, rather than failing.

Now, a (naïve) pure html web page could look like (I’ve only included the interesting parts):

<form method="put" action="...">
  <input name="formats:date" value="%Y-%m-%d">
  <input name="formats:time" value="%H:%M:%S">
  <input name="formats:date" value="%(first_name)s %(last_name)s">
  <input name="edit" type="submit">Save Changes</input>
</form>
<a href="...">Change Password</a>
<a href="...">Exit</a>

A more sophisticated client would probably want to have limited choices for those three formats. If it were still pure HTML (ie, generated by the server that will be handling it), then it would know about the available choices for those fields, and you could have something like:

<form method="put" action="...">
  <select name="formats:date">
    <option value="%Y-%m-%d" selected>ISO 8601 (2012-01-26)</option>
    <option value="%b %d, %Y">Long (January 26, 2012)</option>
    <option value="%m/%d/%Y">US (26/01/2012)</option>
    <option value="%d/%m/%Y">Australian (01/26/2012)</option>
    ...
  </select>
  ...
  <input name="edit" type="submit">Save Changes</input>
</form>
<a href="...">Change Password</a>
<a href="...">Exit</a>

But how do we provide these choices to a non-pure HTML client? Surely we don’t want to embed them in the resource: that seems wasteful, especially if they are unlikely to change often. Why, we can have a resource that contains the choices:

[
  {
    "value": "%Y-%m-%d",
    "title": "ISO 8601"
  },
  {
    "value": "%b %d, %Y",
    "title": "Long"
  },
  {
    "value": "%m/%d/%Y",
    "title": "US"
  }
]

Note here that I am relying on the rich client to provide the example, based on today’s date, probably. That means we can cache this more.

But where do we link to this?

I’d probably extend it so that all of the formats appear in one resource:

{
  "date": [...],
  "time": [...],
  "name": [...]
}

And then have a link:

{
  "_links": {
    "self": {"href": "..."},
    "edit": {"href": "...", "title": "Save changes"},
    "change-password": {"href": "...", "title": "Change Password"},
    "up": {"href": "...", "title": "Exit"},
    "choices:formats": {"href": "..."}
  },
  "formats": {
    "date": "%Y-%m-%d",
    "time": "%H:%M:%S",
    "name": "%(first_name)s %(last_name)s"
  }
}

Indeed, it might be even more complicated than that: with the name format, for instance, and even with the date and time, we could provide some canned types, and “Custom”, allowing the user to use the valid formatting token to make their own. But that’s another story.

link reltypes

What is still part of the same story, however, is how to know what verb types should be used for which links, and what data should be sent in the case of PUT/POST requests. I’ve been sitting down working some stuff out about this, and come up with some conventions I think I’ll look at using.

rel=self

"self": {"href": "...", "title": "User Details"}

rel=self means that this is the address of the resource that is currently being used. We could look at using the title attribute to mean that ‘this is a good title for the current place of interaction within the system’. That also means we can have a different title for the same resource, depending upon other stuff in the system, for instance.

rel=edit

"edit": {"href": "...", "title": "Save changes", "data": {
    "editable-field-name": {
      "format": "string",
      "required": true,
      "multiple": false
    }, ...
  }
}

rel=edit means that this is the URI that should be PUT to, with what the client desires the resource should be. Note that I have played around a bit and added in an optional attribute: data, which contains an object with every editable field on the source object. If this field is missing, then the implication is that all fields are editable.

This also can be used to provide the client with information as to the data type, and if it is required and/or must contain an array of objects. The formats would be string, number, boolean. I still haven’t worked out how you might include an object here: the format could be object, but then you might need to define further what should go there.

Requesting this link with a PUT does not ‘move’ the application location: essentially you are still viewing the same object. A client may choose to follow the rel=up link after a successful update, however.

rel=delete

"delete": {"href": "...", "title": "Delete"}

This link can be used to delete an object. I’m not sure what exactly should happen after a deletion: we could have some method of undo that only works until the user visits another link:

{
  "_links": {
    "undelete": {"href": "...", "title": "Restore"}
    "up": {"href": "...", "title": "..."}
  }
}

rel=up, rel=root

"up": {"href": "...", "title": "Back to <...>"}

This link type would be used to move up a level in the application ‘hierarchy’. I think it would need to be present in every resource, except the root resource.

Perhaps we could also have:

"root": {"href": "...", "title": "Home"}

Then we could always jump quickly back to the root resource, without having to navigate through many layers.

rel=[other]

Every other link might be a domain-specific link. For instance, the change-password link in the example above. But we need a way to handle what http verb should be used.

Here is an example of how I think you could indicate that this should be a POST link:

"change-password": {
  "href": "...", 
  "title": "Change password",
  "data": {
    "old-password": {"format": "string", "required": true},
    "new-password": {"format": "string", "required": true}
  }
}

By having the data attribute, we prevent the need for having a GET request to a form resource, and then a POST from that resource to its rel=edit. However, there may be some value in having that step (it matches up with web page navigation, for instance).

The way I would handle it is that every link with a data attribute is essentially defining its form data, which could be used to construct multiple HTML forms in the one page.

It would be domain specific what happens after this request has been executed. In this case, you would probably want to stay on the user details page, but just flash a message that the password had been changed (or not).

Which brings up how to reply with messages in this context. I am tempted to have a “_messages” attribute at the same level as the “_links” attribute, but I’m not totally sold on that just yet.

In the cases where you want the link to be followed with a GET, simply omit the data attribute. However, if you want to be able to have query parameters, you could use:

"posts": {
  "href": "...",
  "title": "Posts",
  "param": {
    "date": {"format": "date"},
    "tag": {"format": "string", "multiple": true}
  }
}

In this case, neither of those parameters are required, but tag can be supplied multiple times. The IANA Link Relations document seems to imply that rel=search should be used for all search stuff, but I have tried to avoid having multiple links of the same reltype. Having that means that instead of being able to use _links.reltype.uri, you would need to use _links.reltype.uri || _links.reltype[0].uri, or something like that.

versioning resources

There are also some nice bits and bobs in the IANA Link Relations document about how versions of resources can refer to one another. Having a working copy is a nice idea, and would allow you to aggregate a series of changes together, while still allowing you to have business logic (and essentially saves) happening. You can then commit a working copy to create a new version.

I’m still a bit anti-locking of resources, as the danger is that a user might lock a resource from one machine, and the commit/discard request may never arrive, resulting in the version remaining locked, even when the same user attempts to change it again. I have done some stuff with merging conflicts on a 412 response.

The two options you have if you discard locking are:

1. There may be multiple working-copy versions of a resource. Essentially each user (or each process they are using) would have a seperate working-copy. You would then need to merge conflicts when a subsequent working-copy is committed to the resource.
2. There may only be one working-copy of a resource. Multiple users may interact with it, and conflicts must be resolved before the working-copy will be updated with that user’s changes.

I think I actually prefer #2. It does mean that two users working on the same object would never be able to independently create their own version: it’s probably a bit more svn than git, but for the business domain I am thinking in, it makes a bit more sense. And it seems to be simpler to merge conflicts as soon as possible.

Again, this just makes me want to write code.

I’ve been working through the whole concept of REST over the past couple of years. I really like the fact that in theory the API should be clean, well designed, and resource-based.

One of the first takeaways I got from this whole school of thought is that it becomes the verbs in HTTP that become important. That is, given a resource, say at http://api.example.com/foo/1/, we can use GET on that resource to view it, DELETE to remove it, and PUT (or POST) to update it.

Since I prefer JSON to XML, I’ll use that for the resource representation examples.

We might get the following back when we do a GET on http://api.example.com/user/1/:

{
  "username": "barry",
  "first_name": "Barry",
  "last_name": "Citizen",
  "links": [
    {
      "rel": "self",
      "uri": "http://api/example.com/user/1/"
    }
  ]
}

I like the concept of having a links attribute, that contains links representing changes of state. But one thing that bothered me, from a UI perspective, is that I need to be able to know beforehand if the user has write/delete access on a resource. How can this best be represented?

I thought one way might be to have:

{
  ...
  "links": [
    {
      "rel": "self",
      "uri": "<whatever>",
      "actions": ["get", "put", "delete"]
    }
  ]
}

But I haven’t seen anything that does something similar to this. And, I have been leaning more towards using POST rather than PUT in lots of cases, as the business logic of my project is that often a user will not have access to the full data associated with a resource. PUT implies that the data that is being sent should replace the resource entirely.

For instance, a manager of a shop who has staff that she shares with another shop may not have access to the other shop’s data, and therefore may not know anything about that shop. In fact, we have at least one case where some managers must not know that another particular shop exists. Thus, she would send back an incomplete list of shops that her employee works at, thus removing him from the ones she cannot see.

As an aside, which I discovered while writing this article, there is also some discussion that it is valid to use PUT to update the bits we know about: RESTful architecture: what should we PUT?.

So, is this a valid way of representing what actions are permissable on a URI? Or is there a standard?

I then remembered that, REST in Practice talks about link types. On page 117:

self

The uri value can be used to GET the latest resource representation of the order. ### update

Consumers can change the order using a POST to transfer a representation to the linked resource.

cancel

This is the uri to be used to DELETE the order resource should the consumer wish to cancel the order.

There is also a link elsewhere in the book to Link Relations. There, we see instead of an update link, an edit link, but no remove or delete links. And this seems to fly against the whole concept of using the verb, rather than the URL, as in my case, those three URLs would all be the same. Besides, this seems to be a bit anti-DRY.

However, maybe this is the best solution: making the resource:

{
  "username": "barry",
  "first_name": "Barry",
  "last_name": "Citizen",
  "links": [
    {"rel": "self", "uri": "http://api/example.com/user/1/"},
    {"rel": "edit", "uri": "http://api/example.com/user/1/"}
  ]
}

The implication here is that the edit URI can also be used to delete the object, but that still doesn’t solve my issue. If we know the user may not delete a resource, then we can prevent access to the control that would be used to delete it.

Indeed, RFC 5023 specifically talks about the Member URI, which is (a) returned in the Location header when an object is created, and then goes on to talk about sending PUT and DELETE requests to the Member URI. But there is no mention about access control.

So then as I think more about it, HATEOAS is all about navigating around using links to control system state. Thus, it makes sense to have seperate links for edit and delete. And, according to Jim Webber, one of the authors of Rest in Practice:

Side note: it’s best not to conflate links with URIs. A link is a hypermedia control, a URI is an identifier and usually an address, and we’re interested in links with well-known semantics and not very interested in the format of the URIs.

(From the REST in Practice discussion group).

The other concern I have is that it would be nice to know what query parameters we can send to a resource collection to filter it. I have put a bit of thought into using URLs for the rel attribute, and those in turn being a resource that can be read using a GET request. This would then provide information like:

• What parameters can be passed to this collection resource to filter it.
• What data needs to be sent to this collection resource in order to create a new object resource. This would be field names, and the data types.

Indeed, I started to plan something like this that could be automatically generated (at least the second part of it) from within django-repose. The idea is that the process of registering an ApiView class also automatically registered the reltype url route.

Having said all of that, the code I have been working on lately looks explicitly for rel=self, and uses that to store the URL that should be operated on.

I’m going to end this now: mainly because I have just run across JSON Linking with HAL, and I’m going to re-work how I generate and handle links.

The other day, I mentioned that I had Marked.app nicely handling my {% highlight %} blocks, and syntax highlighting them. In passing at the end, I mentioned that TextMate was still formatting them as if they were Markdown.

Now, one way around this is to indent them, but then within the code block they are indented further, and that offends my sensibilities.

Now, within TextMate, syntax highlighting is based on scopes, so to do what I want (which is the same as how HTML may have CSS or JS embedded in it), we just need a language grammar pattern that matches, and applies the relevant scope.

TextMate 2 has even nicer features, where you can set the scope (but not, as it turns out, include rules) dynamically based on a match in the pattern.

Anyway, on to the rules.

Rather than edit the Markdown rules, I wanted to just inject the language grammars in from a bundle of my own, but had no luck with this. Instead, I decided to extend the Jekyll bundle.

This is what I wanted to put in the patterns (simplified a little):

{
  begin = ' "%\}\n';
  end = ' "%\}\n';
  name = 'source.$1.embedded.html.markdown';
  patterns = ( { include = 'source.$1'; } );
}

However, as I mentioned above, the expression on line 5 does not actually include source.js patterns in this case.

Instead, I needed to have a seperate pattern for each language I wanted to include patterns from. Since mostly I work in python, html and javascript, for now those ones will do.

Oh, and the last thing is that html needs to include text.html.basic.

You can see my fork at jekyll-tmbundle. The current code is:

By default, KnockoutJS binds to <body>, by the look of things. If you have something like:

<html>
	<head>
		<title data-bind="text: title"></title>
	</head>
	<body>
		...
		<script>
			var vm = {
				title: ko.observable("This is the page title")
			};
			ko.applyBindings(vm);
		</script>
	</body>
</html>

The title will not be bound. Instead, you’ll need to use (and I’m using jQuery):

<html>
	<head>
		<title data-bind="text: title"></title>
	</head>
	<body>
		...
		<script>
			var vm = {
				title: ko.observable("This is the page title")
			};
			ko.applyBindings(vm, $('html')[0]);
		</script>
	</body>
</html>

You could also apply the binding twice, once to the head, and once to the body.

I am writing a set of UI widgets for use in web apps, using the excellent KnockoutJS library. One of the more challenging ones has been the colour picker. Rather than do what everyone else has done, I tried to ape the Apple Colour Picker. But this gives us values in HSV, which aren’t that useful for web.

So I came across a page that has a JavaScript HSV to RGB converter: http://jsres.blogspot.com/2008/01/convert-hsv-to-rgb-equivalent.html. And there are so many things wrong with that code that it hurts.

• the declared variables r,g,b are not used at all.
• RGB is defined as an Array, but used as an Object.
• var_r and friends are not declared, and pollute the global namespace.

Plus, more I came across as I worked through the code.

So, I thought I’d clean it up, and make it a bit easier to follow.

Some of the bits that are ‘tricky’ are the use of toString(16), which converts a number to a base 16 representation, and the ("0" + value).slice(-2), which zero-pads a string.

The algorithm itself is fairly easy to follow: there are seven possible cases for the data conversion. If the saturation is 0, then RGB is #000000.

Otherwise, the value depends on the value of Math.floor(h/60). There is a simple lookup table (data), which stores three values based on hue, saturation and value, and then it’s just a matter of picking the correct two to use with the value, and returning that.

Most things I seem to develop have a top-level menu that runs across the top or side of the page, which has the various pages within them. In many cases, some of these will only be accessible to people who have permission to access the view to which they point. That is, I want to selectively display menu items based on if the logged in user can access the page linked to.

Previously, I have been doing this by having a check in the template that is essentially a duplicate of the check in the view. Specifically, the view check is usually a decorator (or multiple decorators), using things like login_required or user_passes_test, which takes a callable that runs the test, when passed in the user object.

I already had a django-menus app, that generates the menu items, based on the view name (it creates the links, and you may optionally pass in a text value). But each of these was wrapped in things like:

{% if request.user.is_staff  %}
  {% menu_item 'foo' 'Foo'  %}
{% endif  %}

The view foo is wrapped in a decorator:

@user_passes_test(lambda user:user.is_staff)
def foo(request):
  # ... view code removed ...

So, I spent quite a while working out how to inspect the decorators wrapping a function, and if they were of the form that might be a test of permission/some other attribute, run the test. I’ve settled on the convention that a decorator function that takes a first argument of user or u (for lambda u: u.is_staff, for instance) is executed.

The project can be found at django-menus.

Marked.app is pretty sweet. What I like most about it is that it takes about 2 minutes for my site to regenerate, so doing things like previewing a post is a bit of a pain in the arse: so I can use Marked.app to have previews every time I save.

But most of my posts are technical, and have code fragments. I’m using Liquid Templating within Jekyll (indeed, a custom highlighter that caches the files), and these were not rendered well by Marked.app. Fair enough, too, as it doesn’t know anything about them.

So, I needed a way to have the {% highlight %} tags handled by Albino, which in turn uses Pygments.

There are posts on the Marked.app site that talk about Jekyll, but they don’t actually handle the Liquid Templating syntax. For example: github-flavored-markdown.rb

But this one does:

Now all I need is for TextMate to recognise that those block delimiters mean that it is source code and to highlight it as such.

Something I seem to be re-implementing over and over again is a way of displaying user messages in response to user interactions. For instance, when form data is invalid, or a click on an element could not perform the desired action.

Using KnockoutJS, I had been using a messages object within my ViewModel, but that felt dirty: I always had to code in specific message handling. Having things hook into one another’s ViewModels became very messy.

The other day at work, I rediscovered django’s signals, and went a little nuts replacing other ways of doing things (often using callbacks) with signals. This becomes cleaner once you have multiple things that need to listen.

Signals are a bit like jQuery Events, and it occurred to me that I can do a nice messaging framework using them.

The general concept is this: you have an element in your page that you want to designate as the message display port. (Technically, you could have several: you can even set them up to filter, but that’s another story). You just set this up like (we’ll talk about the options later):

<div data-bind='messages: {}'></div>

Then, anywhere within your page, you have an element simply trigger an event. For instance, you might have this as an event handler:

// Foo
var eventHandler = function(evt) {
  var $target = $(evt.target);
  $target.trigger('message', {type: 'error', message: 'Oops!'});
}

It’s currently still under development. You can see where it is at now at messages.js. It is part of koui, which is a GUI framework that is built on top of KnockoutJS, but it can stand alone. It does depend on KO, ko.mapping, jQuery and underscore.js.

Frank Wiles gave a great talk Secrets of PostgreSQL Performance

Don’t do dumb things

• Dedicate a single server to your database
• Only fetch what you need

Do smart things

• cache everything
• limit number of queries

Tuning

• shared_buffers : 25% of available RAM
• effective_cache_size : OS disk cache size
• work_mem : in-memory sort size

Less important

• wal_buffers : set to 16MB
• checkpoint_segments : at least 10
• maintenance_work_mem : 50MB for every GB of RAM

Can also transactionally turn on grouping of transactions.

Hardware

• As much RAM as you can afford - fit whole db if you can.
• Faster disks.
  • Disk speed is important
  • RAID5 is bad
  • RAID-1+0 is good
  • WAL on own disk → 4x write performance
• CPU speed - unlikely to be the limiting factor.

Other

• use pg_bouncer to pool connections
• use tablespaces to move tables/indexes onto other disks
  • ie, indexes on fastest disk
  • stuff that might run in background and hit only specific tables that are not used by other bits

KnockoutJS is a great way to create relationships between data objects, and interface elements. You can, for instance, bind a date value to an html input[type=date] element, and have it converted into a proper date object. You could then display data based on this, or do anything else you wanted.

KnockoutJS 1.2 (the currently stable version) defaults to using jQuery templates (jQuery-tmpl), which happen to use conflicting syntax to django templates.

For instance, if you were to have the following in your django template file:

{{if foo > bar }}
  <div>Stuff Here</div>
{{/if }}

Then django would attempt to process that, as it uses bits that look like django’s template engine’s value placeholder.

A workaround to this is to look at doing something like wrapping any jQuery templates in something that prevents django from interpreting it.

But I don’t like that solution. For starters, almost every text editor will try to syntax highlight data between <script> tags as javascript, even when explicitly marked as <script type="text/html"> or any other non-javascript mime type.

So, it would be nicest (and cleanest) to be able to have each jQuery template item in a separate file in my project.

Enter {% jquery_template %}. With a custom django templatetag, you can not only have it include the template in your django template, but it will automatically add the script tags, and even add an id.

For instance, you can do:

{% jquery_template 'path/to/template.html' 'templateName'  %}

This will include the data from path/to/template.html, which it finds in any template location, but wrapped in <script type="text/html" id="templateName">.

I have a django app that contains this template tag, as well as some other useful stuff for jquery, and other javascript stuff (including knockoutjs). You can see this template tag at: jquery_template.py.

Hope it’s useful.