Sorting dates in DataTables

If you have tabular data, then semantically, you’ll want to put it into an HTML table. It makes sense, and is certainly easier than trying to post-style nested divs as a table.

The other really nice thing is that it’s fairly easy to use DataTables to then make that table dynamic. Especially useful if your table is large: I use it on a report of all customers in my work, and have just started using it in some user-facing pages. In essence, it is as simple as doing:

$('#table-id').dataTable();

With this, you get sortable columns, pagination, and searching.

But sorting of dates sucks, unless they are in ISO8601 format. ISO8601 is fantastic, by the way. Not only do you get dates/datetimes that are inherently no longer ambiguous, but they sort alphabetically, as you would expect. Because every field is larger than all of the fields following it, and all fields are zero-padded, eveny date or datetime will be correctly sorted.

However, the general public does not understand these two reasons for a ‘one true date format’, so we are generally forced to display it in a more readable format. Which doesn’t sort alphabetically.

There is a trick you can use to get sorting, and nice dates using DataTables, though. For example, the following (rendered) html will sort correctly, both ascending and desencding, but also only display a nice format:

<td>
  <span style="display: none;">2012-06-07</span>
  Thursday, June 7th, 2012
</td>

In django, you can use the following snippet:

<td>
  <span style="display: none;">{{ value|date:"Y-m-d" }}</span>
  {{ value|date:"l, F jS, Y" }}
</td>

Recently, DataTables also had a blog post about how to use it with Twitter Bootstrap 2. I think it looks rather nice. And with this tip, it is so much more useful.

You can also use this way of thinking on other things that should be sorted differently to how they are printed.

Spurious CORS Errors from Sentry

I realised the other day that Sentry, the awesome system we have been using for a while to track our error logs from our Django project, can also be used to track exceptions from other systems. Like Javascript. In fact, there is a client available: raven.js.

So, we have a server set up for work, but I have a side-project I have been working on, Workout Builder. So, I thought I’d set up a server in Heroku to act as my sentry server. And I found a nice simple way to get up and running: Daniel Watkins has a nice post over at Odd_Blog, Deploying Sentry on Heroku.

It’s pretty straightforward, and extremely simple. I got it up and running in no time, and then attempted to set up an email service. Rather than use my actual account for sending, I thought I’d set up a sending-only account at my domain, hosted as a Gmail Apps domain. So, I set it up, and set about testing.

All of a sudden, I’m getting errors, that didn’t appear for 30 seconds, that my test domain is not permitted to send a request due to CORS. But, I had been sending them previously.

After lots of dicking around, I discovered it was because I did not have the gmail settings quite right. Instead of telling me what the problem was, something was masking the issue (that the server was timing out because the server/port combination was not correct), and jQuery thought it was a CORS issue.

So, fixing up the email sending settings, and it’s all gravy:

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

EMAIL_HOST = 'smtp.gmail.com'
EMAIL_HOST_PASSWORD = '<oh no you don\'t>'
EMAIL_HOST_USER = 'noreply@schinckel.net'
EMAIL_PORT = 587
EMAIL_USE_TLS = True

Initially, I had used mail.google.com, port 25, and EMAIL_USE_TLS = False. Eventually, I got it all right.

A new Garmin Communicator plugin

As part of my plan to create a workout editor, I had to look into the method of communicating between the Garmin plugin and the browser.

It feels like a Java application. It’s documented like it, too. But, it’s written in Prototype, and includes a whole stack of other tools, like XML handling, Ajax communication, and messaging. Things that should belong in seperate parts, IMHO.

So, after a fair bit of plugging around, I was able to make enough sense of it to figure out exactly how it works:

  1. You unlock the plugin with a key-pair
  2. You get a list of devices
  3. If this is a send, then you set the value of a certain property.
  4. You start an Async communication.
  5. You poll the ‘finish’ version of that communication.
  6. When the communication is finished, if this was a receive, you load the data from a property.

I understand why they have made the plugin handle it’s communication in an async manner, but seriously: why not allow for a callback function when the communication is finished? To me, that feels like it would make so much more sense.

Anyway, my other main criticism is that it is inherently unsafe for multiple operations. Instead of, as would be possible with a callback that gets executed when the communication is finished, returning the data, it puts it into a property within the plugin. Which does mean that any bit of code can read it, but also means it’s possible to accidentally overwrite it, as the same property is used for writes.

So, the API for replacing it looks more like:

var plugin = new Garmin.Communicator();
plugin.selectDevice(0);
plugin.readActivities(function(data) {
  // data contains the XML activity data.
});

It’s actually a little more complicated than this: we can pass in delegates, that will have callback methods called when certain events occur. These events are also pushed (using jQuery) onto the HTML element that is the plugin object. But, due to a jQuery bug, you need to listen further up the chain: so you can listen for these events on body.

This script will also add the plugin to the page if it cannot find it, and will run as a singleton: calling the constructor a second time will return the original object, but also add a new delegate to the list of delegates.

I’m tempted to remove the delegate handling, and simply have it as callback-based, but this is sort-of a transition from the way the Garmin team have done it. I’m concerned there may be issues with non-UI initiated read/write events (ie, those that happen on page load) ‘beating’ the plugin being ready, but that is a job for another day.

I’ve also written some Knockout bindings for this: but those are not quite ready for public consumption. I may actually write parsing code for the Training Center Database XML file, and the types it contains, and include that with this project. But, then I may be approaching the bloat seen in the actual Garmin plugin. At this stage, if you have a server that accepts TCX files, then this should be enough.

The project is on BitBucket, as usual: garmin-plugin.

TCX Files and Garmin Goals

I’m partway through writing a workout planning tool: it’s web-based, similar to Garmin Connect, but hopefully with a better interface. I want to be able to create workouts, but I’m really happy with Strava for my activity tracking.

Part of the appeal is being able to export the data to my Garmin Forerunner HRM: this really is one of those ‘scratch my own itch’ tools. So, I’ve had to learn a bit about the Garmin TCX format. There is documentation: it is just an XML file that matches the desired schema.

I’ve made a lot of progress with the workout creation, and even exporting this to TCX. Today, I decided to work on the Goal planning.

Some Garmin HRMs have a neat feature where you can set goals, which the watch will track as you work out. Thus, you could decide you want to run 50km in a given week, and it will show you how far along that goal you are, and how much time you have remaining. However, there is no way on the Forerunner 405cx to set goals on the device, nor with Garmin Training Center, and you have to use Garmin Connect.

The thing is, this part of the TCX file is undocumented. It is stored in the <Extensions> section, and here is my plan to document it a little better.

The basic structure of the file is:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<TrainingCenterDatabase 
  xmlns="http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://www.garmin.com/xmlschemas/ActivityGoals/v1 
  http://www.garmin.com/xmlschemas/ActivityGoalExtensionv1.xsd 
  http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2 
  http://www.garmin.com/xmlschemas/TrainingCenterDatabasev2.xsd">

  <Folders/>

  <Author xsi:type="Application_t">
    <!-- Application info goes here -->
  </Author>

<Extensions>
  <ActivityGoals xmlns="http://www.garmin.com/xmlschemas/ActivityGoals/v1">
    <!-- List of goals goes here -->
  </ActivityGoals>
</Extensions>

We are only interested in what happens in the list of goals.

Mostly, a goal is fairly simple:

<ActivityGoal Current="0.0000000" Measure="DistanceMeters" Sport="All" Target="1000.0000000">
  <Name>Run 1km</Name>
  <Period Recurrence="Once">
    <StartDateTime>2012-07-15T00:00:00Z</StartDateTime>
    <EndDateTime>2012-07-21T23:59:59Z</EndDateTime>
  </Period>
</ActivityGoal>

From this we can see the following fields:

  • Current The amount of Measure that has been completed.
  • Measure The type of goal. Allowable values are: DistanceMeters, TimeSeconds, Calories and NumberOfSessions.
  • Sport You may limit the goal to activities of a given sport. Allowable values are: All, Running, Biking and Other. Note that Garmin Connect will allow you to choose other sports, however, the value will effectively be cast to one of these. Note also that these are the exact same values that are valid for a Workout sport type (with the addition of All).
  • Target What the actual target is.
  • Name The name of this goal. This will not be displayed on a Forerunner 405cx: not sure about other devices.
  • Period Recurrence At this stage, I’m not sure what other values than Once are permitted, but I will be investigating this: this could turn out to be a really nice way to have a repeating weekly goal.
  • StartDateTime, EndDateTime happy to see these in ISO8601 format. Not surprised by that, as the Activity spec stuff (as well as Workout scheduling) is also all in ISO8601.

I do have a couple of comments so far: the HRM watches are essentially timezone aware, and they pull their time from the GPS satellites. I wonder if goals will then respect this: I’m at +0930: if I set a goal to end at 2012-07-21T23:59:59Z, will it finish at that time (which is a UTC timestamp), or will it finish at midnight local time? Can you set goals that finish at other times than midnight?

Initial experiments appear to show no. Setting a time other than 23:59:59 means that the goal is not shown on the device. I don’t see this as a big disadvantage. Testing the timezone-ness of the period is harder: I need to wait until midnight to do so!

Secondly, what values are valid for the recurrence period? This requires some experimentation.

It appears to accept a value of Weekly, but as to if this actually does anything, I’m yet to discover. Considering it has an explicit StartDateTime and EndDateTime, unless the watch extrapolates and updates it, I’m not expecting it to do anything. Certainly, setting an EndDateTime in the past, and choosing Weekly does not appear to have any effect. Again, I’m going to have to wait until midnight clicks over to test this properly. Hopefully, it will update the start and finish times, and reset the current amount.

Also of interest: Garmin Connect sends through a Dummy <X>goal for every goal Measure you do not provide a goal for. However, this is not necessary: removing all but the goals you want to use from the generated TCX file does not prevent sending it to the device, but having an invalid Author block does prevent it from sending.

The Forerunner 405cx will only display one goal of each type (Measure). I believe it shows only the one that is closest to expiring.

When the Garmin agent sends the data to the watch, it removes it from the filesystem. This prevents it being re-sent. When data is recieved from the watch, it appears re-create the activity file from the current goals set up in Garmin Connect. This kind-of makes sense, but is annoying, as any goals that have been set up in Garmin Connect will override the goals created elsewhere.

In practice, it means that in order to send goal data to the device, you must first download the relevant activities, and calculate just how much of each goal has been completed. I was hoping to be able to avoid this: if the watch sent us the goal Current figure, then we could just load this, and apply any changes to targets, without affecting the current value. However, with my device, at least, ActivityGoals are InputToUnit only. At least, if you have no goals in Garmin Connect, it doesn’t send back bogus (dummy) goal data!

Developing RESTful Web APIs with Python, ...

This week’s Python Weekly has a link to a presentation by Nicola Iarocci, called Developing RESTful Web APIs with Python, Flask and MongoDB.

I have a few minor concerns with some aspects of the content.

No form validation

This concerns me. I’ve recently started using django’s forms for my validation layer for API calls, and also for the generation of the serialised output. It’s not completely flawless, but it seems to be working quite well. It certainly is more robust than my hand-rolled scheme of validating data, and using code that is better tested than my own is always a bonus.

Instead, as we see later, there is a data validation layer. It has basically the same goal as django’s forms, but is somewhat more nested, rather than using classes. Also, using classes makes it easier to have inheritance, a great way to have shared rules. You could do this using the same function in your custom validation, but this feels disconnected.

MongoDB
scalable, high-performance, …

The integrity of my data is important to me. It’s very rare that the db is the limiting factor in my system’s performance, and having stuff written to disk as soon as it is ‘real’ is kind-of critical.

http://api.example.com/v1/contacts/

Okay, this is where I jump on my high horse: “versioning should happen in the media-type”. Or even better, resources should be forwards and backwards compatible, and clients should be written to handle (or ignore) changes to schemata.

@mimerender( ... )

A decorator that has 5 arguments? That will be applied to every view function? Surely there’s a way to do this without having to decorate every function. Django CBV FTW here.

“Thu, 1 Mar 2012 10:00:49 UTC”

Egad. I can’t think of a reason to have machine readable dates in any format other than ISO 8601. Purely for the reason of being able to sort dates whilst they are still strings.

PATCH
Why not PUT?

Why not POST?

This is something that has been debated for ages. I think I kind-of agree with the author: PATCH is more explicitly a partial update. It does make me think about using some type of diff, but I guess using concurrency control covers the same ground.

"<link rel='parent' ... />"

Okay, HTML/XML inside a JSON object?

Why not have:

{
  "rel": "parent",
  "title": "...",
  "href": "..."
}

At least that way you’ll be able to parse the data out natively.

“updated”: “…”,
“etag”: “…”

I’m not sure if it is necessary/warranted/desired to have the etag as part of the representation. Especially if the etag is generated from the content: that would kind-of preclude it.

Personally, I generate etags from a higher resolution timestamp (possibly hashed with the object id, class or whatever). Whilst etags are opaque, having them as human readable helps with troubleshooting.

To me, this seems to be metadata, and should not be part of the object. I think you could argue a case that within Collection+JSON you could add this in, for convenience. It certainly would make it easier not to have to store the etag in a seperate variable on the client, for one.

The discussion about Concurrency Control is quite good. Which reminds me: I enjoyed most of this presentation. I have some minor nitpicks, but some of those I understand the author’s choices. Some I don’t (date format). It’s certainly better than the REST API Design Rulebook, which is a load of junk.