Schinckel.net

jQuery dynamic forms

Some javascript, currently using jQuery, that will convert a form into an ajax form.

First up, a shim to enable FormData for non-compliant browsers. Based on formdata.js, but with some changes. Man, callback based code is a bitch when all you need to do is get some value, and have to work around that.

(function(w, $) {
  // Don't override if it is native.
  if (w.FormData)
    return;
      
  function FormData(form) {
    var fd = this;
    this.fake = true;
    this.boundary = "--------FormData" + Math.random();
    this._fields = [];
    this.contentType = 'multipart/form-data; boundary=' + this.boundary;
          
    if (form) {
      var $form = $(form);
      $.each($form.serializeArray(), function(i, obj){
        fd.append(obj.name, obj.value);
      });
      $form.find('[type=file]').each(function(i, file) {
        fd.append(file.name, file.files[0]);
      });
    }
  }
    
  // A listener to automatically add the binary version of the data. This may suck.
  $('[type=file]').change(function updateData(change) {
    var reader = new FileReader();
    reader.onload = function(load) {
      $(change.target.files[0]).data('binary-file-data', load.target.result);
    }
    reader.readAsBinaryString(change.target.files[0]);
  });
    
  // The interface FormData provides...
  FormData.prototype.append = function(key, value) {
    this._fields.push([key, value]);
  }
    
  // But, we will actually look more like a string to XMLHttpRequest.
  FormData.prototype.toString = function() {
    var boundary = this.boundary;
    var body = '';
    $.each(this._fields, function(i, field) {
      body += '--' + boundary + '\r\n';
      if (field[1].name) {
        var file = field[1];
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\"; filename=\""+ file.name +"\"\r\n";
        body += "Content-Type: "+ file.type +"\r\n\r\n";
        body += $(file).data('binary-file-data') + "\r\n";
      } else {
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\";\r\n\r\n";
        body += field[1] + "\r\n";
      }
    });
          
    body += "--" + boundary +"--";
    return body;
  }
  w.FormData = FormData;
})(window, jQuery);

Now, for the jQuery code. This will override the submit event on all forms. If there are any <input type=file> elements, then it will use a FormData object, else it will just .serialize() the object. The response will override the .html() content of the form, but not the form itself.

$('form').submit(function(evt) {
  evt.preventDefault();
  var form = evt.target, $form = $(form), data;
  var options = {
    cache: false,
    type: form.method,
    url: form.action,
    done: function(data) {
      $form.html(data);
    },
    fail: function(xhr, status, error) {
      console.log(status, error, xhr);
    }
  };
    
  if ($form.find('[type=file]').length) {
    data = new FormData(form);
    // Native FormData objects set this automatically (how???), but we need to manually do it.
    options.contentType = data.contentType || false;
    options.processData = false;
  } else {
    data = $form.serialize();
  }
  options.data = data;
  $.ajax(options);
});

Sorting dates in DataTables

If you have tabular data, then semantically, you’ll want to put it into an HTML table. It makes sense, and is certainly easier than trying to post-style nested divs as a table.

The other really nice thing is that it’s fairly easy to use DataTables to then make that table dynamic. Especially useful if your table is large: I use it on a report of all customers in my work, and have just started using it in some user-facing pages. In essence, it is as simple as doing:

$('#table-id').dataTable();

With this, you get sortable columns, pagination, and searching.

But sorting of dates sucks, unless they are in ISO8601 format. ISO8601 is fantastic, by the way. Not only do you get dates/datetimes that are inherently no longer ambiguous, but they sort alphabetically, as you would expect. Because every field is larger than all of the fields following it, and all fields are zero-padded, eveny date or datetime will be correctly sorted.

However, the general public does not understand these two reasons for a ‘one true date format’, so we are generally forced to display it in a more readable format. Which doesn’t sort alphabetically.

There is a trick you can use to get sorting, and nice dates using DataTables, though. For example, the following (rendered) html will sort correctly, both ascending and desencding, but also only display a nice format:

<td>
  <span style="display: none;">2012-06-07</span>
  Thursday, June 7th, 2012
</td>

In django, you can use the following snippet:

<td>
  <span style="display: none;">{{ value|date:"Y-m-d" }}</span>
  {{ value|date:"l, F jS, Y" }}
</td>

Recently, DataTables also had a blog post about how to use it with Twitter Bootstrap 2. I think it looks rather nice. And with this tip, it is so much more useful.

You can also use this way of thinking on other things that should be sorted differently to how they are printed.

A new Garmin Communicator plugin

As part of my plan to create a workout editor, I had to look into the method of communicating between the Garmin plugin and the browser.

It feels like a Java application. It’s documented like it, too. But, it’s written in Prototype, and includes a whole stack of other tools, like XML handling, Ajax communication, and messaging. Things that should belong in seperate parts, IMHO.

So, after a fair bit of plugging around, I was able to make enough sense of it to figure out exactly how it works:

  1. You unlock the plugin with a key-pair
  2. You get a list of devices
  3. If this is a send, then you set the value of a certain property.
  4. You start an Async communication.
  5. You poll the ‘finish’ version of that communication.
  6. When the communication is finished, if this was a receive, you load the data from a property.

I understand why they have made the plugin handle it’s communication in an async manner, but seriously: why not allow for a callback function when the communication is finished? To me, that feels like it would make so much more sense.

Anyway, my other main criticism is that it is inherently unsafe for multiple operations. Instead of, as would be possible with a callback that gets executed when the communication is finished, returning the data, it puts it into a property within the plugin. Which does mean that any bit of code can read it, but also means it’s possible to accidentally overwrite it, as the same property is used for writes.

So, the API for replacing it looks more like:

var plugin = new Garmin.Communicator();
plugin.selectDevice(0);
plugin.readActivities(function(data) {
  // data contains the XML activity data.
});

It’s actually a little more complicated than this: we can pass in delegates, that will have callback methods called when certain events occur. These events are also pushed (using jQuery) onto the HTML element that is the plugin object. But, due to a jQuery bug, you need to listen further up the chain: so you can listen for these events on body.

This script will also add the plugin to the page if it cannot find it, and will run as a singleton: calling the constructor a second time will return the original object, but also add a new delegate to the list of delegates.

I’m tempted to remove the delegate handling, and simply have it as callback-based, but this is sort-of a transition from the way the Garmin team have done it. I’m concerned there may be issues with non-UI initiated read/write events (ie, those that happen on page load) ‘beating’ the plugin being ready, but that is a job for another day.

I’ve also written some Knockout bindings for this: but those are not quite ready for public consumption. I may actually write parsing code for the Training Center Database XML file, and the types it contains, and include that with this project. But, then I may be approaching the bloat seen in the actual Garmin plugin. At this stage, if you have a server that accepts TCX files, then this should be enough.

The project is on BitBucket, as usual: garmin-plugin.

The Organism Application

I had an email from a self-confessed django beginner, asking for some assistance. Here is my solution, as I worked through it.

The Application

The application is designed to allow tracking information related to identifying various organisms. An organism may have many identifying features, such as on a tree, the height, and the leaf morphology, or on a bird, the colour of the feathers, size of the egg and so on. To make it simpler for the users, it would be useful to classify organisms as belonging to a type, which can then be used to limit the available choices of identifying features: if an organism is a bird, then we only show those features that make sense for a bird.

To do all of this, we can have a class structure that looks somewhat like:

# models.py
from django.db import models

class OrganismType(models.Model):
    description = models.CharField(max_length=200)

class IdentificationField(models.Model):
    type = models.ForeignKey(OrganismType, related_name='id_fields')
    name = models.CharField(max_length=200)
    
    class Meta:
        unique_together = ('type', 'name')

class Organism(models.Model):
    common_name = models.CharField(max_length=200)
    latin_name = models.CharField(max_length=200, unique=True)
    type = models.ForeignKey(OrganismType, related_name='organisms')

class IdentificationDetail(models.Model):
    organism = models.ForeignKey(Organism, related_name="id_details")
    field = models.ForeignKey(IdentificationField)
    description = models.CharField(max_length=250)
    
    class Meta:
        unique_together = ('organism', 'field')

You’ll see I’ve also included a couple of unique_together constraints: I’ve assumed that each field for a given organism should only appear once.

Bending the admin to our will

Next, we can put all of this into the admin. This is really quite simple, but, as we will see, has it’s limits.

# admin.py
from django.contrib import admin

from models import OrganismType, Organism, IdentificationField, IdentificationDetail

class IdentificationFieldInline(admin.TabularInline):
    model = IdentificationField
    extra = 0

class OrganismTypeAdmin(admin.ModelAdmin):
    inlines = [IdentificationFieldInline]

class IdentificationDetailInline(admin.TabularInline):
    model = IdentificationDetail
    extra = 0

class OrganismAdmin(admin.ModelAdmin):
    inlines = [IdentificationDetailInline]    
    list_display = ('common_name', 'latin_name', 'type')
    list_filter = ('type',)

admin.site.register(OrganismType, OrganismTypeAdmin)
admin.site.register(Organism, OrganismAdmin)

I’ve removed the extra empty forms on the formsets, it looks much cleaner. I’ve also used a couple of the nice features of the admin to make display of stuff better.

At this point, thanks to the magic of django, you now have an administrative interface. But, it doesn’t quite do what we want: that is, we haven’t limited which identification fields will be available in the organism’s inlines.

To do that, we need to fiddle with the formset.

# forms.py
from django import forms

from models import IdentificationDetail, Organism

class IdentificationDetailFormSet(forms.models.BaseInlineFormSet):
    def __init__(self, *args, **kwargs):
        super(IdentificationDetailFormSet, self).__init__(*args, **kwargs)
        for form in self.forms:
            self.update_choices(form)
    
    # We need to override the constructor (and the associated property) for the
    # empty form, so dynamic forms work.
    def _get_empty_form(self, **kwargs):
        form = super(IdentificationDetailFormSet, self)._get_empty_form(**kwargs)
        self.update_choices(form)
        return form
    empty_form = property(_get_empty_form)
    
    # This updates one form's 'field' field queryset, if there is an organism with type
    # associated with the formset. Otherwise, make the choice list empty.
    def update_choices(self, form):
        if 'type' in self.data:
            id_fields = OrganismType.objects.get(pk=self.data['type']).id_fields.all()
        elif self.instance.pk and self.instance.type:
            id_fields = self.instance.type.id_fields.all()
        else:
            id_fields = IdentificationDetail.objects.none()
        
        form.fields['field'].queryset = id_fields

This process is something I’ve talked about before (and finding that post was what pointed the questioner in my direction), but I’ll discuss it again anyway. This is perhaps a more concrete example anyway.

We want to change the queryset available to a given field (in this case, confusingly called field), based on the value of a related object. In this case, we want to set the queryset of an identification detail’s field to all of the available identification fields on the related organism’s type. Whew!

As it turns out, it’s easier to see this in the code. Note also that if there is no selected organism type (as would be the case when an empty form is presented), no fields can be selected.

This alone would work: except that changing the organism’s type should change the available list of field types. There are two approaches that can be used: have all of the data available in the page somewhere, and use JavaScript to filter the available list of field types, or fetch the data dynamically from the server (again, using JavaScript) at the time the type is changed. If I were using something like KnockoutJS, then the former would be easier, and improve the responsiveness: the change would be immediate. Since I’m not using anything that doesn’t come with django, I’ll fetch the data on each change.

So, we are going to need some JavaScript. When we do the end-user page, it’s easy to see how to put that in, but we need to understand how to override django’s admin templates in order to inject it in this case.

The django documentation has some nice detail about how to do this: Overriding admin templates. In this case, we need to create a file within our app at templates/admin/organisms/organism/change_form.html. We want to just add data to the regular template, so we just inherit from it.

{% extends 'admin/change_form.html' %}

{% block after_related_objects %}
{{ block.super }}
<script>
django.jQuery(function($){
  $('#id_type').change(function(evt){
    $.ajax({
      url: "/admin/organisms/organismtype/" + this.value + '/fields/',
      type: 'get',
      success: function(data) {
        $('tr.form-row td.field-field select').html(data);
      }
    });
  });
});
</script>
{% endblock %}

The script here adds a change event handler to the organism type <select> element, that hits the server, and gets the list of fields for that type. It then sets the content of the inline identification detail field fields to the data the server returned. This clears whatever had been stored there previously, but that is probably the behaviour we want in this case. Note that I am hard-coding the URL for now: we’ll see a way to handle that in a better way later.

Only one thing remains: to actually write the view that returns the desired content of the <select> element. For now, we will put this into the admin class of the organism type. Again, later we’ll move this to a proper seperate view, but doing it this way shows how easy it is to extend the admin interface.

Back in our admin.py file, we want to change the OrganismTypeAdmin class:

# admin.py

from django.contrib import admin
from django.conf.urls import patterns, url
from django.http import HttpResponse

# [snip]

class OrganismTypeAdmin(admin.ModelAdmin):
    inlines = [IdentificationFieldInline]
    
    def get_urls(self, **kwargs):
        urls = super(OrganismTypeAdmin, self).get_urls(**kwargs)
        urls = patterns('', 
            url(r'^(.*)/fields/$', self.get_fields, name='organisms_organismtype_fields'),
        ) + urls
        return urls
    urls = property(get_urls)
    
    def get_fields(self, request, *args, **kwargs):
        data = "<option value>---------</option>"
        if args[0]:
            data += "".join([
                "<option value='%(id)s'>%(name)s</option>" % x 
                for x in OrganismType.objects.get(pk=args[0]).id_fields.values()
            ])
        return HttpResponse(data)

We can use the fact that the admin model object provides its own urls, and we can override the method that generates them. We need to put our fields view before the existing ones (and allow empty strings where we want the primary key), else it will be matched by another route.

Finally, we write the view itself. If there was no primary key provided, we return a “null” option, otherwise we include that and the actual list of choices.

Doing it for real

Of course, in a real environment, we probably don’t want to give access to the admin interface to anyone but trusted users. And even then, limit that to as few as possible. In this case, I would suggest that the admin users would be creating the OrganismType objects, but creating Organism objects would be done by regular users. Which means we really only have a couple of pages that need to be written for the outside world:

  • View a list of organisms.
    • Filter the list of organisms by OrganismType
    • Search for an organism by common name or latin name
    • Search for an organism by some other means (feather colour, etc)
  • Create a new organism
  • Edit an existing organism
  • Fetch a list of field types for a given organism type (the get_fields view above.)

This may come in a future post: I had forgotten about this and need some time to get back into it.

jQuery long-poll for CouchDB changes.

I spent a bit of time this weekend playing with CouchDB. I think I have almost figured out how to do some stuff that had been bugging me. Coming to a document-based store after so much time and effort on relational DBMS does mean you really need to approach things from a different direction.

Our project is basically a multi-tenanted hosted application. Each customer has their own data, and within a relational model you basically have a field in each table that references the customer the data belongs to. Either that, or you have a seperate database and installation per-customer, but that doesn’t scale well on a system like ours that get intermittent use throughout the week.

I’m going to talk more about CouchDB and segmenting the data later, but the best solution is to have a single database per-customer. This makes more sense when you know how CouchDB works: a CouchDB database is a container for documents. Grouping these by customer means you can easily replicate one customer’s data, or move it to a different node.

One really nice feature of CouchDB is the changes feed. From this, you can subscribe to all of the changes that occur in a database, and this could be filtered (so a user would only be notified of changes to documents that they have read-access on, for instance).

This could potentially solve lots of problems that we have with different users from the same company working on the same roster at the same time, and those changes automatically appearing in everyone’s browser.

There are three types of change feeds that are interesting:

  1. The list of changes since I last checked (I send in a sequence number).
  2. The same, but handled using long-polling (I keep the request open until a change occurs).
  3. A continuous polling approach, where changes are sent to my open connection as they occur.

I really liked the sound of the last one, but $.stream, the only library I could find that did that for jQuery had some issues: like the fact is sent the request by POST, and that I couldn’t actually get it to see any data that was coming back in.

To solve the problem using #1, you could write some code that keeps track of the sequence number, and runs a request every X seconds. But I liked the idea of long-polling.

The idea I had was to run a request, and in the success handler, recursively call the function. To actually handle the incoming data, I thought that it might be a good solution to use jQuery events. I’m having them triggered on $(document), as I haven’t seen a standard way to do this. The other option might be on $(window), or a passed-in object.

Fetching iTunes Store Data

Or, quite possible the best AppleScript I have ever written.

So, I have a new iMac, and I was moving over a heap of ’legacy’ media from another machine, and some of it was DVD rips of Scrubs, Mad Men and Big Love. Now, these were mostly well tagged, but often I didn’t have all of the nice descriptions that were in iTunes.

Typing in all of this data is fairly annoying.

AppleScript, jQuery to the rescue:

tell application "iTunes"
	repeat with theTrack in selection
		tell theTrack
			set trackNo to episode number
		end tell
		tell application "Safari"
			set theDesc to do JavaScript "$('span.index span').filter(function() { return parseInt($(this).text(),10) == " & trackNo & "; }).closest('tr').find('.description .text').text();" in document 1
			set theName to do JavaScript "$('span.index span').filter(function() { return parseInt($(this).text(),10) == " & trackNo & "; }).closest('tr').find('.name .text').text();" in document 1
		end tell
		tell theTrack
			set description to theDesc
			set name to theName
			set episode ID to (my replace(show, " ", ".")) & ".S" & my zero_pad(season number, 2) & "E" & my zero_pad(episode number, 2)
		end tell
	end repeat
end tell

on zero_pad(value, string_length)
	set tmp_string to "000000000" & (value as string)
	set padded_value to characters ((length of tmp_string) - string_length + 1) thru -1 of tmp_string as string
	return padded_value
end zero_pad

on replace(content, sub, repl)
	set delims to AppleScript's text item delimiters
	set AppleScript's text item delimiters to sub
	set content to every word of content
	set AppleScript's text item delimiters to repl
	set content to content as text
	set AppleScript's text item delimiters to delims
	return content
end replace

You just need to have the relevant TV Series Season page in the frontmost window in Safari, and select all of the episodes for that season in iTunes. Then, run the script, and watch the magic.

User messaging with jQuery events

Something I seem to be re-implementing over and over again is a way of displaying user messages in response to user interactions. For instance, when form data is invalid, or a click on an element could not perform the desired action.

Using KnockoutJS, I had been using a messages object within my ViewModel, but that felt dirty: I always had to code in specific message handling. Having things hook into one another’s ViewModels became very messy.

The other day at work, I rediscovered django’s signals, and went a little nuts replacing other ways of doing things (often using callbacks) with signals. This becomes cleaner once you have multiple things that need to listen.

Signals are a bit like jQuery Events, and it occurred to me that I can do a nice messaging framework using them.

The general concept is this: you have an element in your page that you want to designate as the message display port. (Technically, you could have several: you can even set them up to filter, but that’s another story). You just set this up like (we’ll talk about the options later):

<div data-bind='messages: {}'></div>

Then, anywhere within your page, you have an element simply trigger an event. For instance, you might have this as an event handler:

// Foo
var eventHandler = function(evt) {
  var $target = $(evt.target);
  $target.trigger('message', {type: 'error', message: 'Oops!'});
}

It’s currently still under development. You can see where it is at now at messages.js. It is part of koui, which is a GUI framework that is built on top of KnockoutJS, but it can stand alone. It does depend on KO, ko.mapping, jQuery and underscore.js.

django and jQuery templates

KnockoutJS is a great way to create relationships between data objects, and interface elements. You can, for instance, bind a date value to an html input[type=date] element, and have it converted into a proper date object. You could then display data based on this, or do anything else you wanted.

KnockoutJS 1.2 (the currently stable version) defaults to using jQuery templates (jQuery-tmpl), which happen to use conflicting syntax to django templates.

For instance, if you were to have the following in your django template file:

{{if foo > bar }}
  <div>Stuff Here</div>
{{/if }}

Then django would attempt to process that, as it uses bits that look like django’s template engine’s value placeholder.

A workaround to this is to look at doing something like wrapping any jQuery templates in something that prevents django from interpreting it.

But I don’t like that solution. For starters, almost every text editor will try to syntax highlight data between <script> tags as javascript, even when explicitly marked as <script type="text/html"> or any other non-javascript mime type.

So, it would be nicest (and cleanest) to be able to have each jQuery template item in a separate file in my project.

Enter {% jquery_template %}. With a custom django templatetag, you can not only have it include the template in your django template, but it will automatically add the script tags, and even add an id.

For instance, you can do:

{% jquery_template 'path/to/template.html' 'templateName'  %}

This will include the data from path/to/template.html, which it finds in any template location, but wrapped in <script type="text/html" id="templateName">.

I have a django app that contains this template tag, as well as some other useful stuff for jquery, and other javascript stuff (including knockoutjs). You can see this template tag at: jquery_template.py.

Hope it’s useful.

Knockout Collection

I am loving KnockoutJS. It makes it super easy to bind data values to UI elements in a declarative manner. You no longer have to worry about callbacks updating your data model and/or your view widgets.

The addition to KnockoutJS that I have been working on is a ‘collection’, that can be used to contain a set of objects, which can be fetched from a server, and each of which has it’s own resource URI that will be used to update or delete it.

For instance, we may have a collection URI:

GET "http://example.com/people/"

When we access this using a GET request, we might see something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[
  {
    "first_name": "Adam",
    "last_name": "Smith",
    "links": [
      {"rel":"self", "uri": "http://example.com/people/552/"}
    ]
  },
  {
    "first_name": "John",
    "last_name": "Citizen",
    "links": [
      {"rel":"self", "uri": "http://example.com/people/32/"}
    ]
  }  
]

Each linked resource contains the full (or as much as the logged-in user is able to see) representation. Example:

GET "http://example.com/people/552/"

1
2
3
4
5
6
7
8
9

{
  "first_name": "Adam",
  "last_name": "Smith",
  "date_of_birth": "1910-02-11",
  "email": "adam.smith@example.com",
  "links": [
    {"rel":"self", "uri": "http://example.com/people/552/"}
  ]
}

Now, this is just the beginning. Obviously, we want to turn all of these fields into observables. I also wanted to know when any data had changed (so the “Save” button can be disabled when the object is not dirty). Clearly, being able to write the data back to the server, as well as create new objects, and delete them. Further, I needed to be able to do conditional reads and writes (only allow the object to be saved if no-one else has touched it since we last fetched it).

The place where the ko.mapping plugin broke down for me was that updating the resource from the full representation didn’t add the new fields that came back from the server. It may be that indeed this is possible (I think it is), but at the time, I could not see how to do this. It may be that I will rewrite this to use the ko.mapping stuff, but I’m not so sure right now.

Anyway, after a couple of revisions, I have a working framework.

To use it, you can just do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// Add a dependentObservable called 'name'.
var processPerson = function(item) {
  item.name = ko.dependentObservable(function(){
    return item.first_name() + ' ' + item.last_name();
  });
};

var people = ko.collection({
  url: "http://example.com/people/",
  processItem: processPerson
});

There is one main caveat at this stage:

  • It is expected that each object will have a ’name’ property. If your server does not return one, you’ll need to setup a dependentObservable as shown in processPerson above.

First, the ko.collection object:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273

ko.collection = function(options) {
  // Let jQuery know we always want JSON
  $.ajaxSetup({
    contentType: 'application/json',
    dataType: 'json',
    cache: false // This is browser cache! Needs to be set for Firefox.
  });
  
  options = options || {};
  var url = options.url;                  // Allow passing in a url.
  var processItem = options.processItem;  // Allow passing in a function to process each item after it is fetched.
  var etag;
  
  
  // Initial setup. We need to set these early so we can access them, even
  // if we have no data for them.
  var self = {
    items: ko.observableArray([]),
    selectedItem: ko.observable(null),
    selectedIndexes: ko.observableArray([]),
    filters: ko.observable({})
  };
  
  /*
  Message handling.
  
  We have a messages observableArray, but we use this dependent observable
  to access it. This allows us to have messages that expire.
  
  self.messages() => provide access to the array of messages.
  self.messages({
    type: "error|notice|warning|whatever",    => This will usually be used to apply a class
    message: "Text of message",               => This text will be displayed
    timeout: 1500                             => If this is non-zero, message expires (and 
                                                 is automatically removed after this many 
                                                 milliseconds)
  });
  
  Every message object gets given a callback function (.remove()), that,
  when executed, well immediately remove that message, and get rid of the
  timer that normally removes that message after timeout.
  
  The messages object is also given a flush() function, that will remove
  all of the messages within it.
  
  Not sure if I should move this to a seperate plugin?
  */
  var messages = ko.observableArray([]);
  self.messages = ko.dependentObservable({
    read: function() {
      return messages();
    },
    write: function(message) {
      var timeout;
      message.remove = function() {
        messages.remove(message);
        clearTimeout(timeout);
      };
      messages.remove(function(item) {
        return item.message === message.message;
      });
      messages.push(message);
      if (message.timeout) {
        timeout = setTimeout(function(){
          messages.remove(message);
        }, message.timeout);
      }
    }
  });
  self.messages.flush = function() {
    $.each(messages, function(message){
      message.remove();
    });
  };
    
  /*
  filteredItems : a subset of self.items() that has been passed through
                  all of the self.filters(), and selects only those that
                  match. A filter must be an object of the form:
                  {
                    value: ko.observable(""),
                    attr: "name",
                    test: function(test_value, obj_value) {}
                  }
                  
                  The filtering code handles getting the correct values to
                  pass to the test function, the attr is the name of the 
                  attribute on each member of self.items() that will be
                  tested.
                  Having 'value' passed in means we can have a default
                  value when app starts.
  */
  self.filteredItems = ko.dependentObservable(function() {
    var filteredItems = self.items();
    $.each(self.filters(), function(name, filt){
      filteredItems = ko.utils.arrayFilter(filteredItems, function(item){
        if (!filt.attr || !item[filt.attr]) {
          return true;
        }
        return filt.test(filt.value(), item[filt.attr]());
      });
    });
    return filteredItems;
  });
  
  /*
    This is really only used by a select[multiple] object, and is used in
    conjunction with selectedIndexes.
    
    TODO: make this a writeable dependentObservable.
  */
  self.selectedItems = ko.dependentObservable(function() {
    return self.items().filter(function(el){
      return $.inArray(self.items().indexOf(el), self.selectedIndexes()) >= 0;
    });
  });
  
  /*
    Filter self.items() finding only those that have at least one attribute
    that is marked as dirty.
  */
  self.dirtyItems = ko.dependentObservable(function() {
    return self.items().filter(function(el){
      return el.isDirty();
    });
  });
  
  /*
    Filter self.items(), finding only those that have at least one attribute
    marked as conflicted.
  */
  self.conflictedItems = ko.dependentObservable(function() {
    return self.items().filter(function(el){
      return el.hasConflicts();
    });
  });
  
  self.setSource = function(newUrl) {
    url = newUrl;
  };
  
  /*
    Fetch all items from the url we have for the index.
    
    It is allowable that the index does not return the full body of each
    item, but instead only contains perhaps a name, and links for that
    item. Then, we can use self.selectedItem().fetch() to get the full
    data for the item.
  */
  self.fetchItems = function() {
    if (!url) {
      return;
    }
    var headers = {};
    if (etag) {
      headers['If-None-Match'] = etag;
    }
    $.ajax({
      url: url,
      type: "get",
      headers: headers,
      statusCode: {
        200: function(data, textStatus, jqXHR) {
          // Successful. If we already had objects, then
          // we need to update that list.
          $.each(self.items(), function(i, item){
            // Is there an item in the new data items list that matches
            // the item we are now looking at?
            var matchingItem = data.filter(function(el){
              links = el.links.filter(function(link){
                return link.rel==="self";
              });
              return links[0] && links[0].uri === item._url();
            })[0];
            if (matchingItem) {
              // Update the item that matched.
              item.updateData(matchingItem);
              if (processItem) {
                processItem(item);
              }
              // Remove from data.
              data.splice(data.indexOf(matchingItem), 1);
              // Not sure if this should be here.
              // item.isDirty(false);
            } else {
              // Not found in incoming data: remove from our local store.
              // Will this break $.each(self.items(), ...) ?
              self.items.remove(item);
            }
          });
          
          // Any items that we have left in data (which will be all if we
          // haven't loaded this up before) now need to be added to items().
          // On a clean fetch, this will be the first code that is run.
          $.each(data, function(i, el){
            var item = ko.collectionItem(el, self);
            if (processItem) {
              processItem(item, el);
            }
            self.items.push(item);
          });
          
          // Finally, update the etag.
          etag = jqXHR.getResponseHeader('Etag');
        }
      }
    });
  };
  
  /*
    A shortcut method that allows us to bind an action to fetch the
    data from the server for the currently selected item.
  */
  self.fetchSelectedItemDetail = function(evt) {
    if (self.selectedItem && self.selectedItem()) {
      self.selectedItem().fetch();
    }
  };
  
  /*
    Create an item. I haven't implemented this yet, because I haven't 
    figured out a way to see what fields are needed to be created when
    there are no currently loaded items. I'm thinking about using a
    Wizard in my application, so this might be overridden by the app.
  */
  self.createItem = function(evt) {
    console.log("ADDING ITEM (NOT FINISHED YET)");
    // The trick here is knowing what fields need to be created.
    // self.items.push(ko.collectionItem({}));
  };
  
  /*
    Permanently remove the selectedItem, and delete it on the server.
  */
  self.removeSelectedItem = function(evt) {
    if (self.selectedItem && self.selectedItem()) {
      var sure = confirm("This will permanently remove " + self.selectedItem().name());
      if (sure){
        self.selectedItem().destroy();        
      }
    }
  };
  
  /*
    Iterate through self.items(), finding those that match all of the data
    we pass in.
    
    For instance, you can do things like: 
    
      viewModel.findMatchingItems({date_of_birth: "1995-01-01"})
    
    This is used internally to find matches for objects when updating. Not
    sure why it is exposed as a public member function though.
  */
  self.findMatchingItems = function(options) {
    return self.items().filter(function(el){
      var match = true;
      $.each(options, function(opt, val) {
        if (el[opt]() !== val) {
          // Returning false causes $.each to stop, too.
          return match = false;
        }
      });
      return match;
    });
  };
    
  if (url) {
    self.fetchItems();
  }
  
  return ko.observable(self);
};

Second, the ko.collectionItem object. This may be eventually hidden in the collection object, as it isn’t really intended to be used seperately.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384

ko.collectionItem = function(initialData, parentCollection) {
  var self = {
    isFetched: ko.observable(false)
  };
  var links = [];
  var etag = null;
  var url = null;
  var attributes = ko.observableArray([]);
  var collection = parentCollection;
  var dirtyFlag = ko.observable(false);
  
  /* Private methods */
  
  /*
    Given the incoming 'data' for this object, look through the fields for
    things that differ between the server representation and the client
    representation. Store both values for any differences in an attribute
    of the observable called conflicts().
    
    For each conflict, create a member function on the observable that
    allows you to resolve the conflict. When the last conflict is resolved,
    our etag is updated to the value the server gave us.
    
    This method returns true if all conflicts could be resolved (ie, the
    data in all fields was the same, just the etag had changed).
  */
  var parseConflicts = function(data, newEtag) {
    $.each(data, function(attr, value){
      if (attr !== "links") {
        if ($.compare(value, self[attr]() === undefined ? "" : self[attr]())) {
          // Server and client values match.
          // We need to do some funky stuff with undefined values, and treat
          // them as "". I don't really like this, but it works for now.
          self[attr].conflicts([]);
          self[attr].resolveConflict = function(){};
        } else {
          self[attr].conflicts([value, self[attr]() === undefined ? "" : self[attr]()]);
          self[attr].resolveConflict = function(chosenValue) {
            // Mark the entire object as dirty, so we can allow it to be
            // saved, even if we set it to the original value we had (which
            // differed from the server's value).
            self.isDirty(true);
            self[attr](chosenValue);
            self[attr].conflicts([]);
            if (!self.hasConflicts()) {
              // If this was the last conflict, we can use the new etag from
              // the server.
              etag = newEtag;
            }
          };
        }        
      }
    });
    var conflicts = self.hasConflicts();
    if (!conflicts) {
      etag = newEtag;
    }
    return !conflicts;
  };
  
  /*
  Given an object containing errors, we want to apply each of these
  errors onto the relevant field. We want to remove any errors that are
  already on any field.
  
  If we have any errors leftover, we need to notify globally, using the
  parentCollection's messages object.
  */
  var markErrors = function(errors) {
    $.each(attributes(), function(i,attr){
      if (!self[attr].errors) {
        self[attr].errors = ko.observableArray([]);
      }
      if (errors[attr]) {
        self[attr].errors(errors[attr]);
        delete errors[attr];
      } else {
        self[attr].errors([]);
      }
    });
    
    $.each(errors, function(field){
      parentCollection.messages({type:"error", message: field + ": " + errors[field].join("<br>"), timeout: 3000});
    });
  };
  
  /*
    Get the attributes ready for sending to the server.
    
    We can't just iterate through properties, as some will not apply. We
    use the convention that we will only send back properties that the
    server sent to us.
  */
  var prepareAttributes = function() {
    var data = {};
    $.each(attributes(), function(i,attr){
      data[attr] = self[attr]();
    });
    return data;
  };
  /* Public methods */
  
  /*
    Update the data fields associated with this object from the provided
    data.
    
    This may create new attributes, which need to be noted so we can send
    those values back to the server.
    
    We can mark all updated attributes as not dirty, not conflicted, and
    not having errors.
  */
  self.updateData = function(data) {
    if (data.links) {
      // We want to store the links, but not attach them to the object.
      links = data.links;
      delete data.links;
      $.each(links, function(i, obj){
        if (obj.rel === "self") {
          url = obj.uri;
        }
      });
    }
    
    $.each(data, function(attr, value){
      if (attributes().indexOf(attr) < 0) {
        self[attr] = ko.observable(value);
        self[attr].errors = ko.observableArray([]);
        self[attr].conflicts = ko.observableArray([]);
        ko.dirtyFlag(self[attr], false);
        // Need to add this last to cause the dirtyFields dependentObservable
        // to work correctly when editing the last field.
        attributes.push(attr);
      } else {
        self[attr](value);
        self[attr].errors([]);
        self[attr].conflicts([]);
        self[attr].isDirty.reset();
      }
    });
    // Put the links back in case a post-processor needs them.
    data.links = links;
  };
  
  self.serialize = function(evt) {
    return JSON.stringify(prepareAttributes());
  };
  
  /*
    Discard any local changes, and pull the data from the server.
  */
  self.revert = function(evt) {
    etag = null;
    self.fetch();
    parentCollection.messages({type:'warning', message:'The object "' + self.name() + '" was reverted to the version stored on the server.', timeout: 5000});
  };
  
  /*
    Attempt to save the data to the server.
    
    Only permitted to do this if we have successfully fetched the data
    at some point.
    
    Notes: We use POST instead of PUT, in case we do not have access to
           all of the fields of the object. PUT implies the complete resource
           is being updated.
           Errors may come back in {'field-errors': []}, or {'detail':[]}.
           Currently, this makes assumptions about server type, which are
           bad. I need to refactor the error handling code. (400,409)
           Precondition Failed (412) needs to be handled differently, as
           we need to fetch the data from the server if none was provided
           as to the current state of the resource.
           
  */
  self.save = function(evt) {
    if (self.isFetched()) {
      $.ajax({
        url: url,
        type: 'post', // We can't PUT in case we don't know about all fields.
        headers: {'If-Match': etag},
        data: self.serialize(),
        statusCode: {
          200: function(data, textStatus, jqXHR) {
            // Object saved.
            // Incase some fields were reformatted by the server, redo our data.
            self.updateData(data);
            etag = jqXHR.getResponseHeader('Etag');
            parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was saved.', timeout: 2500});
            self.isDirty(false);
          },
          201: function(data, textStatus, jqXHR) {
            // Object saved for the first time (created)
            // Incase some fields were reformatted by the server, redo our data.
            self.updateData(data);
            etag = jqXHR.getResponseHeader('Etag');
            url = jqXHR.getResponseHeader('Location');
            parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was created.', timeout: 2500});
            self.isDirty(false);
          },
          400: function(jqXHR, textStatus, errorThrown) {
            var data = JSON.parse(jqXHR.responseText);
            if (data['field-errors']) {
              markErrors(data['field-errors']);
            }
            parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be saved. Please check the highlighted field(s).', timeout: 10000});
          },
          409: function(jqXHR, textStatus, errorThrown) {
            // Errors saving the data. Likely to be validation errors.
            // We should have a detail object with info to display.
            var data = JSON.parse(jqXHR.responseText);
            if (data.detail) {
              markErrors(data.detail);
            }
            parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be saved. Please check the highlighted field(s).', timeout: 10000});
          },
          412: function(jqXHR, textStatus, errorThrown) {
            // Data was changed on server since we last fetched it.
            // There may be conflicts to deal with.
            // See if the server gave us a current version back...
            var data, serverEtag;
            if (jqXHR.responseText) {
              data = JSON.parse(jqXHR.responseText);
            } else {
              $.ajax({
                url: url,
                async: false,
                success: function(newData, textStatus, jqXHR) {
                  data = newData;
                  serverEtag = jqXHR.getResponseHeader('Etag');
                }
              });
            }
            if (parseConflicts(data, serverEtag)) {
              // We were able to resolve all of the conflicts, now we can
              // try to re-save; but only if it was the first time we saved,
              // to prevent inifinite recursion.
              if (evt) {
                self.save();
              }
            } else {
              parentCollection.messages({type:'error', message:'The object "' + self.name() + '" has been modified on the server. Please check the changed field(s) and select the appropriate value(s).', timeout: 10000});
            }
          }
        }
      });
    }
  };
  
  /*
    Permanently delete the object from the server.
  */
  self.destroy = function(evt) {
    if (self.isFetched() && etag) {
      console.log("DELETING ITEM");
      $.ajax({
        url: url,
        type: 'delete',
        headers: {'If-Match': etag},
        success: function(data, textStatus, jqXHR) {
          if (collection) {
            collection.items.remove(self);
          }
          parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was deleted.', timeout: 2500});
        },
        error: function(jqXHR, textStatus, errorThrown) {
          // Display error message about not being able to delete?
          parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be deleted.', timeout: 10000});
        }
      });
    }
  };
  
  /*
    (Re)Fetch the resource from the server.
    
    Handle conflicts if the arise (when the object has already been fetchd)
  */
  self.fetch = function(evt) {
    var headers = {};
    if (etag) {
      headers['If-None-Match'] = etag;
    }
    $.ajax({
      type: 'get',
      url: url,
      headers: headers,
      statusCode: {
        200: function(data, textStatus, jqXHR) {
          // If we have an etag already, this means the object has been
          // updated on the server, and we need to look for conflicts.
          if (etag) {
            var serverEtag = jqXHR.getResponseHeader('Etag');
            // If we were unable to handle all conflicts, we need to exit.
            if (!parseConflicts(data, serverEtag)) {
              parentCollection.messages({type:'error', message:'The object "' + self.name() + '" has been modified on the server. Please check the changed field(s) and select the appropriate value(s).', timeout: 10000});
              return;
            };
          }
          
          // Otherwise, we can update the data and the etag.
          self.updateData(data);
          etag = jqXHR.getResponseHeader('Etag');
          self.isFetched(true);
        },
        304: function() {
        }
      },
      error: function(jqXHR, textStatus, errorThrown) {
        parentCollection.messages({type:"error", message:"There was an error fetching the data from the server"});
      }
    });
  };
  
  
  
  /* Dependent Observables */
  self.dirtyFields = ko.dependentObservable(function(){
    return ko.utils.arrayFilter(attributes(), function(attr){
      return self[attr] && self[attr].isDirty && self[attr].isDirty();
    });
  });
  
  self.conflictedFields = ko.dependentObservable(function() {
    return ko.utils.arrayFilter(attributes(), function(attr){
      return self[attr] && self[attr].conflicts && self[attr].conflicts().length > 0;
    });
  });
  
  var filterAttributes = function(property) {
    return function() {
      return ko.utils.arrayFilter(attributes(), function(attr){
        return self[attr] && self[attr][property] && self[attr][property]().length > 0;
      }).length > 0;
    };      
  };
  
  self.hasErrors = ko.dependentObservable(filterAttributes('errors'));
  self.hasConflicts = ko.dependentObservable(filterAttributes('conflicts'));
  
  /*
    An object is dirty when:
      - any of its fields/attributes are dirty. (we aks them), OR
      - we have explicitly marked it as dirty.
      
    We need to do the latter for when we have merged a conflict, by choosing
    our value, which differed from the server. The local model would
    normally think it wasn't dirty, but it differs from the server, and
    does need to be saved.
  */
  self.isDirty = ko.dependentObservable({
    read: function() {
      return self.dirtyFields().length > 0 || dirtyFlag();
    },
    write: function(value) {
      dirtyFlag(value);
      if (!value) {
        $.each(attributes(), function(attr){
          if (self[attr] && self[attr].isDirty) {
            console.log(attr);
            self[attr].isDirty.reset();          
          }
        });
      }
    }
  });
  
  /*
    Can this object be saved back to the server?
    Only when it is dirty, and has been fetched.
  */
  self.canSave = ko.dependentObservable(function() {
    return self.isDirty() && self.isFetched();
  });
  
  self._etag = function(){return etag;};
  self._attributes = function(){ return attributes();};
  self._url = function() {return url;};
  
  if (initialData) {
    self.updateData(initialData);
  }
  
  return self;
};