Schinckel.net

Over the past years, I’ve worked really hard to get code coverage of my work project up to a reasonable level. I’m not attempting to get 100% coverage (although, I have made it there in some smaller projects), nor do I mistakenly believe that 100% coverage is a panacea. I’m aware that even 100% coverage doesn’t mean there are no bugs.

But there are a lot of reasons to have high test coverage. Having less than 100% coverage mean that the code not being covered is not automatically tested.

In a django project, there are several different types of code. Python code, HTML, CSS and Javascript. In my case, there’s also a lot of SQL. Some of it is doing things like defining views and functions - but I’m not going to deal with that now…

What I am going to look at is Javascript.

It would be awesome to write unit tests for all of our JS code, but we’ve also been writing Robot Framework tests for a lot of our functionality: what would be great is to capture the coverage that is already happening due to this. It would be even better if it was possible to get this coverage information for JS that is embedded in HTML templates, but that seems like it might be a harder job.

There are really only two things we need to do in order to capture the coverage data:

• Modify the scripts so they generate coverage data when they are executed
• Capture the coverage data before any navigation away from any page

The first one could be done in a number of different ways. You might be able to use a staticfiles backend to modify the files when running collectstatic, or perhaps a middleware that modified the files as they are served. That could possibly modify the inline script tags as well, although ideally you’d want to be doing that in the template rendering phase, as I think that could be the last point you get access to the template file names and locations.

There is a tool that you can use for the actual modification of the script files: nyc. This used to be called Istanbul, I believe, and it can “instrument” files. Then, when the script is executed by the browser, it stores the coverage inforamtion in window.__coverage__.

This then allows for capture of the coverage information, but how do we ensure this information is not lost when the page is navigated away from?

To do that in Django requires a small amount of JS, and a django view to capture it:

window.addEventListener('beforeunload', () => {
  if (window.__coverage__) {
    navigator.sendBeacon("{ % url 'collect-coverage' % }", JSON.stringify(window.__coverage__));
  }
})

Okay, that was easier than expected…what about the matching view?

import json
import uuid
from pathlib import Path

from django.http import HttpResponse

@csrf_exempt
def collect_coverage(request):
    coverage_dir = Path('js-coverage')
    if not coverage_dir.exists():
        coverage_dir.mkdir()

    try:
        json.dump(
            json.loads(request.data),
            (coverage_dir / 'coverage.{uuid.uuid4()}.json').open('w')
        )
    except ValueError:
        pass

    return HttpResponse('')

Oh, that wasn’t too bad either. It just checks that the incoming data is valid JSON, and then dumps it out to a uniquely named JSON file in js-coverage/.

We’ll need to install that in our urls.py:

if settings.DEBUG or settings.TESTING:
    path('coverage/', collect_coverage, name='collect-coverage')

Note well that you will need to ensure you don’t expose this view when not running your Selenium/Robot tests, else someone could very quickly fill up your disk with JSON files…

The last bit of the puzzle is to merge the various files together to generate the report:

nyc report --reporter lcov -t .js-coverage --report-dir .

This generates an lcov.info file that could be uploaded to a coverage analysis tool, and an lcov-report/ directory that contains an HTML report.

So, what about the tricky part, the bit that actually handles running the nyc instrument command on the files?

There are two types of scripts we need to be able to mutate: standalone files (ie, collectstatic), and inline scripts (within HTML templates).

Static files storage backends are relatively well-known, here’s some code that might work:

class InstrumentJSBackend(django.contrib.staticfiles.storage.StaticFilesStorage):
    def post_process(self, paths, dry_run=False, **options):
        if dry_run:
            return

        for path in paths:
            if path.endswith('.js'):
                # We need the original path name, not the one we have just copied...

            yield path, path, True

The latter can be done by using a custom template loader; it’s quite similar in construct to the supplied caching loader:

class Loader(BaseLoader):
    def __init__(self, engine, loaders):
        self.loaders = engine.get_template_loaders()
        super().__init__(engine)

    def get_dirs(self):
      for loader in self.loaders:
          if hasattr(loader, 'get_dirs'):
              yield from loader.get_dirs()

    def get_contents(self, origin):
        # Rewrite any script tags using `nyc instrument`
        contents = origin.loader.get_contents(origin)
        return self.rewrite(origin, contents)

    def get_template_sources(self, template_name):
        for loader in self.loaders:
            yield from loader.get_template_sources(template_name)

    def rewrite(self, origin, contents):

        return subprocess.check_output(['nyc', 'instrument', origin])

I’ve been making more and more use of the django.contrib.postgres classes, and will often store data in an ArrayField where appropriate.

There are two form fields that are supplied with Django for handling these types: one of which has the array values in a single text input (comma separated), and the other has a different text input element for each value.

However, the latter does not really work that well with a dynamic length array (it could work with up to N items, but in my case, I really don’t often have an N).

It could be possible to build similar functionality that you see with the Django Admin’s formset handling like here, however this turns out to be lots of mucking around.

It might be simpler to just have the simple array field rendered, and have JS bolted on that builds the dynamic list of text inputs based on this.

In this instance, I am actually storing the state in the widgets themselves: this means it’s relatively easy to add in the ability to re-order. I’ve done this with the Sortable library.

Django forms are one of the most important parts of the stack: they enable us to write declarative code that will validate user input, and ensure we protect ourselves from malicious input.

Formsets are an extension of this: they deal with a set of homogeous forms, and will ensure that all of the forms are valid independently (and possibly do some inter-form validation, but that’s a topic for a later day).

The Django Admin contains an implementation of a dynamic formset: that is, it handles adding and removing forms from a formset, and maintains the management for accordingly. This post details an alternative implementation.

A Formset contains a Form (and has zero or more instances of that Form). It also contains a “Management Form”, which has metadata about the formset: the number of instances of the form that were provided initially, the number that were submitted by the user, and the maximum number of forms that should be accepted.

A Formset has a “prefix”, which is prepended to each element within the management form:

<input type="hidden" name="prefix-INITIAL_FORM_COUNT" value="...">
<input type="hidden" name="prefix-TOTAL_FORM_COUNT" value="...">
<input type="hidden" name="prefix-MIN_NUM_FORM_COUNT" value="...">
<input type="hidden" name="prefix-MAX_NUM_FORM_COUNT" value="...">

Each Form within the Formset uses the prefix, plus it’s index within the list of forms. For instance, if we have a Formset that contains three forms, each containing a single “name” field, we would have something similar to:

<input type="text" name="prefix-0-name" value="Alice">
<input type="text" name="prefix-1-name" value="Bob">
<input type="text" name="prefix-2-name" value="Carol">

Note that the form’s prefix is <formset_prefix>-<form_index>.

To make a Formset dynamic, we just need to be able to add (and possibly remove, but there’s a little more complexity there) extra forms. The managment form needs to be updated to reflect this, and we need to ensure that the new form’s fields are named appropriately.

A Formset also contains an empty_form. This is an unbound form, where the form’s “index” is set to __prefix__. Thus, the empty form for the above formset might look somewhat like:

<input type="text" name="prefix-__prefix__-name" value="">

We can leverage this to allow us to have simpler code: instead of having to duplicate elements and remove the value, we can just duplicate the empty form, and replace the string __prefix__ with whatever the index of the newly created form should be.

Here’s an implementation that has no dependencies, but does make some assumptions:

I knocked together a little widget that allows for selecting multiple days of the week.

TL;DR: Don’t use KnockoutJS html binding lots of times in your page.

I’m in the middle of rewriting a large part of our application in HTML: for a lot of the interactivity stuff, anything more than just a simple behaviour, I’m turning to KnockoutJS.

Mostly, it’s been awesome. Being able to use two-way binding is the obvious big winner, but dependency tracking is also fantastic.

However, I have had some concerns with performance in the past, and this was always on my mind as I moved into quite a complicated part of the system.

Our approach is that we are not creating a single page application: different parts of the system are at different URLs, and visiting that page loads up the relevant javascript. This is a deliberate tradeoff, mostly because for the forseeable future, our software will not work without a connection to our server: most of the logic related to shift selection is handled by that. We aren’t about to change that.

While rewriting the rostering interface, I initially had Django render the HTML, and I added behaviours. This was possible, and quite fast, however as the behaviours became more complex, I was doing things like sending back scripts that caused other parts of the page to refresh themselves. It was all rather fragile.

So, I went back to KnockoutJS. After a while, I noticed significant slowdowns when dealing with pages that really shouldn’t have been that slow. I’d optimised the database access for the fetching of shifts (and indeed, it is much faster than before), but it felt like Knockout was very sluggish.

I do have quite a few ko.computed() objects, perhaps they were slowing it down? Notably, the function that filters which shifts should be shown where on the page.

So I put some console.time()/timeEnd() calls in place.

Nope: the initial parse of the data runs in less than half a millisecond: instantiating the objects took a while, but the filtering of shifts was taking much less than 100ms.

However, the initial call to ko.applyBindings() was taking several seconds.

The most annoying thing was that when the developer tools were open, it was taking far, far longer!

Eventually, through using the developer tools profiling, I discovered that the slowdown was because of repeated code like:

foo.innerHTML = bar;

Initially, I had thought this slowdown was in KnockoutJS itself, and played around with other ways of binding (such as using the knockout-repeat plugin). Still slow.

Eventually, however, I worked out that it was the act of interacting with the DOM in this manner that was slow. More specifically, the assignation to innerHTML was occurring in the html: binding.

Looking through my source code, I discovered code that looked like:

<span data-bind="html: icon"></span>

And, icon contained the HTML I wanted to put in there:

<i class="icon-ok"></i>

Which was a bad idea to begin with: it conflated UI with data to begin with. So, I replaced the code that looked like:

this.icon = '<i class="icon-ok"></i>';

With:

this.icon = {
  'icon-time': true
};

And then, in the HTML:

<i data-bind="css: icon"></i>

Bingo. All of a sudden, a page that took several seconds to re-render does so in around a second.

It’s important to note that this pattern was repeated several times for each shift: and we have possibly dozens of shifts on a page. When you really need to use the html binding that’s fine, just don’t stick it inside a loop (or worse still, inside a nested loop).

As part of my plan to create a workout editor, I had to look into the method of communicating between the Garmin plugin and the browser.

It feels like a Java application. It’s documented like it, too. But, it’s written in Prototype, and includes a whole stack of other tools, like XML handling, Ajax communication, and messaging. Things that should belong in seperate parts, IMHO.

So, after a fair bit of plugging around, I was able to make enough sense of it to figure out exactly how it works:

1. You unlock the plugin with a key-pair
2. You get a list of devices
3. If this is a send, then you set the value of a certain property.
4. You start an Async communication.
5. You poll the ‘finish’ version of that communication.
6. When the communication is finished, if this was a receive, you load the data from a property.

I understand why they have made the plugin handle it’s communication in an async manner, but seriously: why not allow for a callback function when the communication is finished? To me, that feels like it would make so much more sense.

Anyway, my other main criticism is that it is inherently unsafe for multiple operations. Instead of, as would be possible with a callback that gets executed when the communication is finished, returning the data, it puts it into a property within the plugin. Which does mean that any bit of code can read it, but also means it’s possible to accidentally overwrite it, as the same property is used for writes.

So, the API for replacing it looks more like:

var plugin = new Garmin.Communicator();
plugin.selectDevice(0);
plugin.readActivities(function(data) {
  // data contains the XML activity data.
});

It’s actually a little more complicated than this: we can pass in delegates, that will have callback methods called when certain events occur. These events are also pushed (using jQuery) onto the HTML element that is the plugin object. But, due to a jQuery bug, you need to listen further up the chain: so you can listen for these events on body.

This script will also add the plugin to the page if it cannot find it, and will run as a singleton: calling the constructor a second time will return the original object, but also add a new delegate to the list of delegates.

I’m tempted to remove the delegate handling, and simply have it as callback-based, but this is sort-of a transition from the way the Garmin team have done it. I’m concerned there may be issues with non-UI initiated read/write events (ie, those that happen on page load) ‘beating’ the plugin being ready, but that is a job for another day.

I’ve also written some Knockout bindings for this: but those are not quite ready for public consumption. I may actually write parsing code for the Training Center Database XML file, and the types it contains, and include that with this project. But, then I may be approaching the bloat seen in the actual Garmin plugin. At this stage, if you have a server that accepts TCX files, then this should be enough.

The project is on BitBucket, as usual: garmin-plugin.

I really like KnockoutJS. I’ve said that lots of times, but I mean it. It does one thing, two-way bindings between a data model and the GUI elements, really well.

Perhaps my biggest hesitation in using it in a big project is that there is no built-in persistence layer. This would appear to be a situation where something like Backbone has an advantage.

And then, last week, I came across Simperium.

“So,” I thought, “what if you were able to transparently persist KnockoutJS models using Simperium?”

// Assume we have a SIMPERIUM_APP_ID, and a logged in user's access_token.
var simperium = new Simperium(SIMPERIUM_APP_ID, {token: access_token});
// mappingOptions is a ko.mapping mappingOptions object: really only useful
// if your bucket contains homogenous objects.
var store = new BucketMapping(simperium.bucket(BUCKET_NAME), mappingOptions);

var tony = store.all()[0];

var alan = store.create({
  name: "Alan Tenari",
  date_of_birth: "1965-02-06",
  email: "alan.tenari@example.com"
});

Now, tony is an existing object we loaded up from the server, and alan is one we just created.

Both of these objects are mapped using ko.mapping, but, this is the exciting bit, every time we make a change to any of their attributes, they are automatically persisted back to simperium.

There is a little more to it than that: we may want to only persist valid objects, for instance.

This totally gets me excited. And, I’ve already written a big chunk of the code that actually does this!

But for that, you’ll just have to wait…

Or, quite possible the best AppleScript I have ever written.

So, I have a new iMac, and I was moving over a heap of ’legacy’ media from another machine, and some of it was DVD rips of Scrubs, Mad Men and Big Love. Now, these were mostly well tagged, but often I didn’t have all of the nice descriptions that were in iTunes.

Typing in all of this data is fairly annoying.

AppleScript, jQuery to the rescue:

tell application "iTunes"
	repeat with theTrack in selection
		tell theTrack
			set trackNo to episode number
		end tell
		tell application "Safari"
			set theDesc to do JavaScript "$('span.index span').filter(function() { return parseInt($(this).text(),10) == " & trackNo & "; }).closest('tr').find('.description .text').text();" in document 1
			set theName to do JavaScript "$('span.index span').filter(function() { return parseInt($(this).text(),10) == " & trackNo & "; }).closest('tr').find('.name .text').text();" in document 1
		end tell
		tell theTrack
			set description to theDesc
			set name to theName
			set episode ID to (my replace(show, " ", ".")) & ".S" & my zero_pad(season number, 2) & "E" & my zero_pad(episode number, 2)
		end tell
	end repeat
end tell

on zero_pad(value, string_length)
	set tmp_string to "000000000" & (value as string)
	set padded_value to characters ((length of tmp_string) - string_length + 1) thru -1 of tmp_string as string
	return padded_value
end zero_pad

on replace(content, sub, repl)
	set delims to AppleScript's text item delimiters
	set AppleScript's text item delimiters to sub
	set content to every word of content
	set AppleScript's text item delimiters to repl
	set content to content as text
	set AppleScript's text item delimiters to delims
	return content
end replace

You just need to have the relevant TV Series Season page in the frontmost window in Safari, and select all of the episodes for that season in iTunes. Then, run the script, and watch the magic.

Ryan Niemeyer is the man with respect to most things KnockoutJS, and I had been using a version of his smart dirty flag in some projects. I recall making it so it didn’t have to bind to a secondary property, but I may be mistaken.

Anyway, with Knockout 2.0, we get extenders. Now, it’s possible to do things like:

var thing = ko.observable(null).extend({dirty: true});

It will then look for ko.extenders.dirty, and call that function with two arguments: the observable, and the argument (in this case, true).

Thus, it is possible to re-implement his dirty flag as an extender:

ko.extenders.dirty = function(target, startDirty) {
  var cleanValue = ko.observable(ko.mapping.toJSON(target));
  var dirtyOverride = ko.observable(ko.utils.unwrapObservable(startDirty));
  
  target.isDirty = ko.computed(function(){
    return dirtyOverride() || ko.mapping.toJSON(target) !== cleanValue();
  });
  
  target.markClean = function(){
    cleanValue(ko.mapping.toJSON(target));
    dirtyOverride(false);
  };
  target.markDirty = function(){
    dirtyOverride(true);
  };
  
  return target;
};

The advantage I think mine has over Ryan’s is that you can mark an observable as dirty (thing.markDirty()), and it will stay dirty until you explicitly mark it as clean (thing.markClean()).

Otherwise, it’s just: thing.isDirty() and you are all good.

Alternatively, you could remove the two helper functions, and implement .isDirty() as a writeable observable, that tests the incoming value and sets the internal cleanValue if it needs to.

I am writing a set of UI widgets for use in web apps, using the excellent KnockoutJS library. One of the more challenging ones has been the colour picker. Rather than do what everyone else has done, I tried to ape the Apple Colour Picker. But this gives us values in HSV, which aren’t that useful for web.

So I came across a page that has a JavaScript HSV to RGB converter: http://jsres.blogspot.com/2008/01/convert-hsv-to-rgb-equivalent.html. And there are so many things wrong with that code that it hurts.

• the declared variables r,g,b are not used at all.
• RGB is defined as an Array, but used as an Object.
• var_r and friends are not declared, and pollute the global namespace.

Plus, more I came across as I worked through the code.

So, I thought I’d clean it up, and make it a bit easier to follow.

Some of the bits that are ’tricky’ are the use of toString(16), which converts a number to a base 16 representation, and the ("0" + value).slice(-2), which zero-pads a string.

The algorithm itself is fairly easy to follow: there are seven possible cases for the data conversion. If the saturation is 0, then RGB is #000000.

Otherwise, the value depends on the value of Math.floor(h/60). There is a simple lookup table (data), which stores three values based on hue, saturation and value, and then it’s just a matter of picking the correct two to use with the value, and returning that.