JavaScript Array Widget

I’ve been making more and more use of the django.contrib.postgres classes, and will often store data in an ArrayField where appropriate.

There are two form fields that are supplied with Django for handling these types: one of which has the array values in a single text input (comma separated), and the other has a different text input element for each value.

However, the latter does not really work that well with a dynamic length array (it could work with up to N items, but in my case, I really don’t often have an N).

It could be possible to build similar functionality that you see with the Django Admin’s formset handling like here, however this turns out to be lots of mucking around.

It might be simpler to just have the simple array field rendered, and have JS bolted on that builds the dynamic list of text inputs based on this.

In this instance, I am actually storing the state in the widgets themselves: this means it’s relatively easy to add in the ability to re-order. I’ve done this with the Sortable library.

Django Dynamic Formsets

Django forms are one of the most important parts of the stack: they enable us to write declarative code that will validate user input, and ensure we protect ourselves from malicious input.

Formsets are an extension of this: they deal with a set of homogeous forms, and will ensure that all of the forms are valid independently (and possibly do some inter-form validation, but that’s a topic for a later day).

The Django Admin contains an implementation of a dynamic formset: that is, it handles adding and removing forms from a formset, and maintains the management for accordingly. This post details an alternative implementation.


A Formset contains a Form (and has zero or more instances of that Form). It also contains a “Management Form”, which has metadata about the formset: the number of instances of the form that were provided initially, the number that were submitted by the user, and the maximum number of forms that should be accepted.

A Formset has a “prefix”, which is prepended to each element within the management form:

<input type="hidden" name="prefix-INITIAL_FORM_COUNT" value="...">
<input type="hidden" name="prefix-TOTAL_FORM_COUNT" value="...">
<input type="hidden" name="prefix-MIN_NUM_FORM_COUNT" value="...">
<input type="hidden" name="prefix-MAX_NUM_FORM_COUNT" value="...">

Each Form within the Formset uses the prefix, plus it’s index within the list of forms. For instance, if we have a Formset that contains three forms, each containing a single “name” field, we would have something similar to:

<input type="text" name="prefix-0-name" value="Alice">
<input type="text" name="prefix-1-name" value="Bob">
<input type="text" name="prefix-2-name" value="Carol">

Note that the form’s prefix is <formset_prefix>-<form_index>.

To make a Formset dynamic, we just need to be able to add (and possibly remove, but there’s a little more complexity there) extra forms. The managment form needs to be updated to reflect this, and we need to ensure that the new form’s fields are named appropriately.

A Formset also contains an empty_form. This is an unbound form, where the form’s “index” is set to __prefix__. Thus, the empty form for the above formset might look somewhat like:

<input type="text" name="prefix-__prefix__-name" value="">

We can leverage this to allow us to have simpler code: instead of having to duplicate elements and remove the value, we can just duplicate the empty form, and replace the string __prefix__ with whatever the index of the newly created form should be.

Here’s an implementation that has no dependencies, but does make some assumptions:

Weekday Multi-Select Widget

I knocked together a little widget that allows for selecting multiple days of the week.

KnockoutJS HTML binding

TL;DR: Don’t use KnockoutJS html binding lots of times in your page.

I’m in the middle of rewriting a large part of our application in HTML: for a lot of the interactivity stuff, anything more than just a simple behaviour, I’m turning to KnockoutJS.

Mostly, it’s been awesome. Being able to use two-way binding is the obvious big winner, but dependency tracking is also fantastic.

However, I have had some concerns with performance in the past, and this was always on my mind as I moved into quite a complicated part of the system.

Our approach is that we are not creating a single page application: different parts of the system are at different URLs, and visiting that page loads up the relevant javascript. This is a deliberate tradeoff, mostly because for the forseeable future, our software will not work without a connection to our server: most of the logic related to shift selection is handled by that. We aren’t about to change that.

While rewriting the rostering interface, I initially had Django render the HTML, and I added behaviours. This was possible, and quite fast, however as the behaviours became more complex, I was doing things like sending back scripts that caused other parts of the page to refresh themselves. It was all rather fragile.

So, I went back to KnockoutJS. After a while, I noticed significant slowdowns when dealing with pages that really shouldn’t have been that slow. I’d optimised the database access for the fetching of shifts (and indeed, it is much faster than before), but it felt like Knockout was very sluggish.

I do have quite a few ko.computed() objects, perhaps they were slowing it down? Notably, the function that filters which shifts should be shown where on the page.

So I put some console.time()/timeEnd() calls in place.

Nope: the initial parse of the data runs in less than half a millisecond: instantiating the objects took a while, but the filtering of shifts was taking much less than 100ms.

However, the initial call to ko.applyBindings() was taking several seconds.

The most annoying thing was that when the developer tools were open, it was taking far, far longer!

Eventually, through using the developer tools profiling, I discovered that the slowdown was because of repeated code like:

foo.innerHTML = bar;

Initially, I had thought this slowdown was in KnockoutJS itself, and played around with other ways of binding (such as using the knockout-repeat plugin). Still slow.

Eventually, however, I worked out that it was the act of interacting with the DOM in this manner that was slow. More specifically, the assignation to innerHTML was occurring in the html: binding.

Looking through my source code, I discovered code that looked like:

<span data-bind="html: icon"></span>

And, icon contained the HTML I wanted to put in there:

<i class="icon-ok"></i>

Which was a bad idea to begin with: it conflated UI with data to begin with. So, I replaced the code that looked like:

this.icon = '<i class="icon-ok"></i>';

With:

this.icon = {
  'icon-time': true
};

And then, in the HTML:

<i data-bind="css: icon"></i>

Bingo. All of a sudden, a page that took several seconds to re-render does so in around a second.

It’s important to note that this pattern was repeated several times for each shift: and we have possibly dozens of shifts on a page. When you really need to use the html binding that’s fine, just don’t stick it inside a loop (or worse still, inside a nested loop).

A new Garmin Communicator plugin

As part of my plan to create a workout editor, I had to look into the method of communicating between the Garmin plugin and the browser.

It feels like a Java application. It’s documented like it, too. But, it’s written in Prototype, and includes a whole stack of other tools, like XML handling, Ajax communication, and messaging. Things that should belong in seperate parts, IMHO.

So, after a fair bit of plugging around, I was able to make enough sense of it to figure out exactly how it works:

  1. You unlock the plugin with a key-pair
  2. You get a list of devices
  3. If this is a send, then you set the value of a certain property.
  4. You start an Async communication.
  5. You poll the ‘finish’ version of that communication.
  6. When the communication is finished, if this was a receive, you load the data from a property.

I understand why they have made the plugin handle it’s communication in an async manner, but seriously: why not allow for a callback function when the communication is finished? To me, that feels like it would make so much more sense.

Anyway, my other main criticism is that it is inherently unsafe for multiple operations. Instead of, as would be possible with a callback that gets executed when the communication is finished, returning the data, it puts it into a property within the plugin. Which does mean that any bit of code can read it, but also means it’s possible to accidentally overwrite it, as the same property is used for writes.

So, the API for replacing it looks more like:

var plugin = new Garmin.Communicator();
plugin.selectDevice(0);
plugin.readActivities(function(data) {
  // data contains the XML activity data.
});

It’s actually a little more complicated than this: we can pass in delegates, that will have callback methods called when certain events occur. These events are also pushed (using jQuery) onto the HTML element that is the plugin object. But, due to a jQuery bug, you need to listen further up the chain: so you can listen for these events on body.

This script will also add the plugin to the page if it cannot find it, and will run as a singleton: calling the constructor a second time will return the original object, but also add a new delegate to the list of delegates.

I’m tempted to remove the delegate handling, and simply have it as callback-based, but this is sort-of a transition from the way the Garmin team have done it. I’m concerned there may be issues with non-UI initiated read/write events (ie, those that happen on page load) ‘beating’ the plugin being ready, but that is a job for another day.

I’ve also written some Knockout bindings for this: but those are not quite ready for public consumption. I may actually write parsing code for the Training Center Database XML file, and the types it contains, and include that with this project. But, then I may be approaching the bloat seen in the actual Garmin plugin. At this stage, if you have a server that accepts TCX files, then this should be enough.

The project is on BitBucket, as usual: garmin-plugin.

KnockoutJS persistence using Simperium

I really like KnockoutJS. I’ve said that lots of times, but I mean it. It does one thing, two-way bindings between a data model and the GUI elements, really well.

Perhaps my biggest hesitation in using it in a big project is that there is no built-in persistence layer. This would appear to be a situation where something like Backbone has an advantage.

And then, last week, I came across Simperium.

“So,” I thought, “what if you were able to transparently persist KnockoutJS models using Simperium?”

// Assume we have a SIMPERIUM_APP_ID, and a logged in user's access_token.
var simperium = new Simperium(SIMPERIUM_APP_ID, {token: access_token});
// mappingOptions is a ko.mapping mappingOptions object: really only useful
// if your bucket contains homogenous objects.
var store = new BucketMapping(simperium.bucket(BUCKET_NAME), mappingOptions);

var tony = store.all()[0];

var alan = store.create({
  name: "Alan Tenari",
  date_of_birth: "1965-02-06",
  email: "alan.tenari@example.com"
});

Now, tony is an existing object we loaded up from the server, and alan is one we just created.

Both of these objects are mapped using ko.mapping, but, this is the exciting bit, every time we make a change to any of their attributes, they are automatically persisted back to simperium.

There is a little more to it than that: we may want to only persist valid objects, for instance.

This totally gets me excited. And, I’ve already written a big chunk of the code that actually does this!

But for that, you’ll just have to wait…

KnockoutJS dirty extender.

Ryan Niemeyer is the man with respect to most things KnockoutJS, and I had been using a version of his smart dirty flag in some projects. I recall making it so it didn’t have to bind to a secondary property, but I may be mistaken.

Anyway, with Knockout 2.0, we get extenders. Now, it’s possible to do things like:

var thing = ko.observable(null).extend({dirty: true});

It will then look for ko.extenders.dirty, and call that function with two arguments: the observable, and the argument (in this case, true).

Thus, it is possible to re-implement his dirty flag as an extender:

ko.extenders.dirty = function(target, startDirty) {
  var cleanValue = ko.observable(ko.mapping.toJSON(target));
  var dirtyOverride = ko.observable(ko.utils.unwrapObservable(startDirty));
  
  target.isDirty = ko.computed(function(){
    return dirtyOverride() || ko.mapping.toJSON(target) !== cleanValue();
  });
  
  target.markClean = function(){
    cleanValue(ko.mapping.toJSON(target));
    dirtyOverride(false);
  };
  target.markDirty = function(){
    dirtyOverride(true);
  };
  
  return target;
};

The advantage I think mine has over Ryan’s is that you can mark an observable as dirty (thing.markDirty()), and it will stay dirty until you explicitly mark it as clean (thing.markClean()).

Otherwise, it’s just: thing.isDirty() and you are all good.

Alternatively, you could remove the two helper functions, and implement .isDirty() as a writeable observable, that tests the incoming value and sets the internal cleanValue if it needs to.

HSV to RGB in JavaScript

I am writing a set of UI widgets for use in web apps, using the excellent KnockoutJS library. One of the more challenging ones has been the colour picker. Rather than do what everyone else has done, I tried to ape the Apple Colour Picker. But this gives us values in HSV, which aren’t that useful for web.

So I came across a page that has a JavaScript HSV to RGB converter: http://jsres.blogspot.com/2008/01/convert-hsv-to-rgb-equivalent.html. And there are so many things wrong with that code that it hurts.

  • the declared variables r,g,b are not used at all.
  • RGB is defined as an Array, but used as an Object.
  • var_r and friends are not declared, and pollute the global namespace.

Plus, more I came across as I worked through the code.

So, I thought I’d clean it up, and make it a bit easier to follow.

Some of the bits that are ‘tricky’ are the use of toString(16), which converts a number to a base 16 representation, and the ("0" + value).slice(-2), which zero-pads a string.

The algorithm itself is fairly easy to follow: there are seven possible cases for the data conversion. If the saturation is 0, then RGB is #000000.

Otherwise, the value depends on the value of Math.floor(h/60). There is a simple lookup table (data), which stores three values based on hue, saturation and value, and then it’s just a matter of picking the correct two to use with the value, and returning that.

User messaging with jQuery events

Something I seem to be re-implementing over and over again is a way of displaying user messages in response to user interactions. For instance, when form data is invalid, or a click on an element could not perform the desired action.

Using KnockoutJS, I had been using a messages object within my ViewModel, but that felt dirty: I always had to code in specific message handling. Having things hook into one another’s ViewModels became very messy.

The other day at work, I rediscovered django’s signals, and went a little nuts replacing other ways of doing things (often using callbacks) with signals. This becomes cleaner once you have multiple things that need to listen.

Signals are a bit like jQuery Events, and it occurred to me that I can do a nice messaging framework using them.

The general concept is this: you have an element in your page that you want to designate as the message display port. (Technically, you could have several: you can even set them up to filter, but that’s another story). You just set this up like (we’ll talk about the options later):

<div data-bind='messages: {}'></div>

Then, anywhere within your page, you have an element simply trigger an event. For instance, you might have this as an event handler:

// Foo
var eventHandler = function(evt) {
  var $target = $(evt.target);
  $target.trigger('message', {type: 'error', message: 'Oops!'});
}

It’s currently still under development. You can see where it is at now at messages.js. It is part of koui, which is a GUI framework that is built on top of KnockoutJS, but it can stand alone. It does depend on KO, ko.mapping, jQuery and underscore.js.

Listening to, in Comments

Just had a random thought. I’ve set up an automated method, in ecto, to add at the bottom of a post the currently playing track in iTunes. What about the ability to do the same when commenting on a blog. Is it possible for the browser to get this info, or is that getting a little too spooky. I’m guessing that browsers have restrictions in place, just to stop this sort of stuff happening. After all, if you can get this info from a JavaScript, it’s not too far to stretch grabbing the /etc/passwd file, or similar. And putting that onto a server. Maybe it wasn’t such a good idea after all.

Cold As IceForeignerThe Platinum Collection ★★½