Knockout Collection

I am loving KnockoutJS. It makes it super easy to bind data values to UI elements in a declarative manner. You no longer have to worry about callbacks updating your data model and/or your view widgets.

The addition to KnockoutJS that I have been working on is a ‘collection’, that can be used to contain a set of objects, which can be fetched from a server, and each of which has it’s own resource URI that will be used to update or delete it.

For instance, we may have a collection URI:

GET "http://example.com/people/"

When we access this using a GET request, we might see something like:

 1[
 2  {
 3    "first_name": "Adam",
 4    "last_name": "Smith",
 5    "links": [
 6      {"rel":"self", "uri": "http://example.com/people/552/"}
 7    ]
 8  },
 9  {
10    "first_name": "John",
11    "last_name": "Citizen",
12    "links": [
13      {"rel":"self", "uri": "http://example.com/people/32/"}
14    ]
15  }  
16]

Each linked resource contains the full (or as much as the logged-in user is able to see) representation. Example:

GET "http://example.com/people/552/"
1{
2  "first_name": "Adam",
3  "last_name": "Smith",
4  "date_of_birth": "1910-02-11",
5  "email": "adam.smith@example.com",
6  "links": [
7    {"rel":"self", "uri": "http://example.com/people/552/"}
8  ]
9}

Now, this is just the beginning. Obviously, we want to turn all of these fields into observables. I also wanted to know when any data had changed (so the “Save” button can be disabled when the object is not dirty). Clearly, being able to write the data back to the server, as well as create new objects, and delete them. Further, I needed to be able to do conditional reads and writes (only allow the object to be saved if no-one else has touched it since we last fetched it).

The place where the ko.mapping plugin broke down for me was that updating the resource from the full representation didn’t add the new fields that came back from the server. It may be that indeed this is possible (I think it is), but at the time, I could not see how to do this. It may be that I will rewrite this to use the ko.mapping stuff, but I’m not so sure right now.

Anyway, after a couple of revisions, I have a working framework.

To use it, you can just do:

 1// Add a dependentObservable called 'name'.
 2var processPerson = function(item) {
 3  item.name = ko.dependentObservable(function(){
 4    return item.first_name() + ' ' + item.last_name();
 5  });
 6};
 7
 8var people = ko.collection({
 9  url: "http://example.com/people/",
10  processItem: processPerson
11});

There is one main caveat at this stage:

  • It is expected that each object will have a ‘name’ property. If your server does not return one, you’ll need to setup a dependentObservable as shown in processPerson above.

First, the ko.collection object:

  1ko.collection = function(options) {
  2  // Let jQuery know we always want JSON
  3  $.ajaxSetup({
  4    contentType: 'application/json',
  5    dataType: 'json',
  6    cache: false // This is browser cache! Needs to be set for Firefox.
  7  });
  8  
  9  options = options || {};
 10  var url = options.url;                  // Allow passing in a url.
 11  var processItem = options.processItem;  // Allow passing in a function to process each item after it is fetched.
 12  var etag;
 13  
 14  
 15  // Initial setup. We need to set these early so we can access them, even
 16  // if we have no data for them.
 17  var self = {
 18    items: ko.observableArray([]),
 19    selectedItem: ko.observable(null),
 20    selectedIndexes: ko.observableArray([]),
 21    filters: ko.observable({})
 22  };
 23  
 24  /*
 25  Message handling.
 26  
 27  We have a messages observableArray, but we use this dependent observable
 28  to access it. This allows us to have messages that expire.
 29  
 30  self.messages() => provide access to the array of messages.
 31  self.messages({
 32    type: "error|notice|warning|whatever",    => This will usually be used to apply a class
 33    message: "Text of message",               => This text will be displayed
 34    timeout: 1500                             => If this is non-zero, message expires (and 
 35                                                 is automatically removed after this many 
 36                                                 milliseconds)
 37  });
 38  
 39  Every message object gets given a callback function (.remove()), that,
 40  when executed, well immediately remove that message, and get rid of the
 41  timer that normally removes that message after timeout.
 42  
 43  The messages object is also given a flush() function, that will remove
 44  all of the messages within it.
 45  
 46  Not sure if I should move this to a seperate plugin?
 47  */
 48  var messages = ko.observableArray([]);
 49  self.messages = ko.dependentObservable({
 50    read: function() {
 51      return messages();
 52    },
 53    write: function(message) {
 54      var timeout;
 55      message.remove = function() {
 56        messages.remove(message);
 57        clearTimeout(timeout);
 58      };
 59      messages.remove(function(item) {
 60        return item.message === message.message;
 61      });
 62      messages.push(message);
 63      if (message.timeout) {
 64        timeout = setTimeout(function(){
 65          messages.remove(message);
 66        }, message.timeout);
 67      }
 68    }
 69  });
 70  self.messages.flush = function() {
 71    $.each(messages, function(message){
 72      message.remove();
 73    });
 74  };
 75    
 76  /*
 77  filteredItems : a subset of self.items() that has been passed through
 78                  all of the self.filters(), and selects only those that
 79                  match. A filter must be an object of the form:
 80                  {
 81                    value: ko.observable(""),
 82                    attr: "name",
 83                    test: function(test_value, obj_value) {}
 84                  }
 85                  
 86                  The filtering code handles getting the correct values to
 87                  pass to the test function, the attr is the name of the 
 88                  attribute on each member of self.items() that will be
 89                  tested.
 90                  Having 'value' passed in means we can have a default
 91                  value when app starts.
 92  */
 93  self.filteredItems = ko.dependentObservable(function() {
 94    var filteredItems = self.items();
 95    $.each(self.filters(), function(name, filt){
 96      filteredItems = ko.utils.arrayFilter(filteredItems, function(item){
 97        if (!filt.attr || !item[filt.attr]) {
 98          return true;
 99        }
100        return filt.test(filt.value(), item[filt.attr]());
101      });
102    });
103    return filteredItems;
104  });
105  
106  /*
107    This is really only used by a select[multiple] object, and is used in
108    conjunction with selectedIndexes.
109    
110    TODO: make this a writeable dependentObservable.
111  */
112  self.selectedItems = ko.dependentObservable(function() {
113    return self.items().filter(function(el){
114      return $.inArray(self.items().indexOf(el), self.selectedIndexes()) >= 0;
115    });
116  });
117  
118  /*
119    Filter self.items() finding only those that have at least one attribute
120    that is marked as dirty.
121  */
122  self.dirtyItems = ko.dependentObservable(function() {
123    return self.items().filter(function(el){
124      return el.isDirty();
125    });
126  });
127  
128  /*
129    Filter self.items(), finding only those that have at least one attribute
130    marked as conflicted.
131  */
132  self.conflictedItems = ko.dependentObservable(function() {
133    return self.items().filter(function(el){
134      return el.hasConflicts();
135    });
136  });
137  
138  self.setSource = function(newUrl) {
139    url = newUrl;
140  };
141  
142  /*
143    Fetch all items from the url we have for the index.
144    
145    It is allowable that the index does not return the full body of each
146    item, but instead only contains perhaps a name, and links for that
147    item. Then, we can use self.selectedItem().fetch() to get the full
148    data for the item.
149  */
150  self.fetchItems = function() {
151    if (!url) {
152      return;
153    }
154    var headers = {};
155    if (etag) {
156      headers['If-None-Match'] = etag;
157    }
158    $.ajax({
159      url: url,
160      type: "get",
161      headers: headers,
162      statusCode: {
163        200: function(data, textStatus, jqXHR) {
164          // Successful. If we already had objects, then
165          // we need to update that list.
166          $.each(self.items(), function(i, item){
167            // Is there an item in the new data items list that matches
168            // the item we are now looking at?
169            var matchingItem = data.filter(function(el){
170              links = el.links.filter(function(link){
171                return link.rel==="self";
172              });
173              return links[0] && links[0].uri === item._url();
174            })[0];
175            if (matchingItem) {
176              // Update the item that matched.
177              item.updateData(matchingItem);
178              if (processItem) {
179                processItem(item);
180              }
181              // Remove from data.
182              data.splice(data.indexOf(matchingItem), 1);
183              // Not sure if this should be here.
184              // item.isDirty(false);
185            } else {
186              // Not found in incoming data: remove from our local store.
187              // Will this break $.each(self.items(), ...) ?
188              self.items.remove(item);
189            }
190          });
191          
192          // Any items that we have left in data (which will be all if we
193          // haven't loaded this up before) now need to be added to items().
194          // On a clean fetch, this will be the first code that is run.
195          $.each(data, function(i, el){
196            var item = ko.collectionItem(el, self);
197            if (processItem) {
198              processItem(item, el);
199            }
200            self.items.push(item);
201          });
202          
203          // Finally, update the etag.
204          etag = jqXHR.getResponseHeader('Etag');
205        }
206      }
207    });
208  };
209  
210  /*
211    A shortcut method that allows us to bind an action to fetch the
212    data from the server for the currently selected item.
213  */
214  self.fetchSelectedItemDetail = function(evt) {
215    if (self.selectedItem && self.selectedItem()) {
216      self.selectedItem().fetch();
217    }
218  };
219  
220  /*
221    Create an item. I haven't implemented this yet, because I haven't 
222    figured out a way to see what fields are needed to be created when
223    there are no currently loaded items. I'm thinking about using a
224    Wizard in my application, so this might be overridden by the app.
225  */
226  self.createItem = function(evt) {
227    console.log("ADDING ITEM (NOT FINISHED YET)");
228    // The trick here is knowing what fields need to be created.
229    // self.items.push(ko.collectionItem({}));
230  };
231  
232  /*
233    Permanently remove the selectedItem, and delete it on the server.
234  */
235  self.removeSelectedItem = function(evt) {
236    if (self.selectedItem && self.selectedItem()) {
237      var sure = confirm("This will permanently remove " + self.selectedItem().name());
238      if (sure){
239        self.selectedItem().destroy();        
240      }
241    }
242  };
243  
244  /*
245    Iterate through self.items(), finding those that match all of the data
246    we pass in.
247    
248    For instance, you can do things like: 
249    
250      viewModel.findMatchingItems({date_of_birth: "1995-01-01"})
251    
252    This is used internally to find matches for objects when updating. Not
253    sure why it is exposed as a public member function though.
254  */
255  self.findMatchingItems = function(options) {
256    return self.items().filter(function(el){
257      var match = true;
258      $.each(options, function(opt, val) {
259        if (el[opt]() !== val) {
260          // Returning false causes $.each to stop, too.
261          return match = false;
262        }
263      });
264      return match;
265    });
266  };
267    
268  if (url) {
269    self.fetchItems();
270  }
271  
272  return ko.observable(self);
273};

Second, the ko.collectionItem object. This may be eventually hidden in the collection object, as it isn’t really intended to be used seperately.

  1ko.collectionItem = function(initialData, parentCollection) {
  2  var self = {
  3    isFetched: ko.observable(false)
  4  };
  5  var links = [];
  6  var etag = null;
  7  var url = null;
  8  var attributes = ko.observableArray([]);
  9  var collection = parentCollection;
 10  var dirtyFlag = ko.observable(false);
 11  
 12  /* Private methods */
 13  
 14  /*
 15    Given the incoming 'data' for this object, look through the fields for
 16    things that differ between the server representation and the client
 17    representation. Store both values for any differences in an attribute
 18    of the observable called conflicts().
 19    
 20    For each conflict, create a member function on the observable that
 21    allows you to resolve the conflict. When the last conflict is resolved,
 22    our etag is updated to the value the server gave us.
 23    
 24    This method returns true if all conflicts could be resolved (ie, the
 25    data in all fields was the same, just the etag had changed).
 26  */
 27  var parseConflicts = function(data, newEtag) {
 28    $.each(data, function(attr, value){
 29      if (attr !== "links") {
 30        if ($.compare(value, self[attr]() === undefined ? "" : self[attr]())) {
 31          // Server and client values match.
 32          // We need to do some funky stuff with undefined values, and treat
 33          // them as "". I don't really like this, but it works for now.
 34          self[attr].conflicts([]);
 35          self[attr].resolveConflict = function(){};
 36        } else {
 37          self[attr].conflicts([value, self[attr]() === undefined ? "" : self[attr]()]);
 38          self[attr].resolveConflict = function(chosenValue) {
 39            // Mark the entire object as dirty, so we can allow it to be
 40            // saved, even if we set it to the original value we had (which
 41            // differed from the server's value).
 42            self.isDirty(true);
 43            self[attr](chosenValue);
 44            self[attr].conflicts([]);
 45            if (!self.hasConflicts()) {
 46              // If this was the last conflict, we can use the new etag from
 47              // the server.
 48              etag = newEtag;
 49            }
 50          };
 51        }        
 52      }
 53    });
 54    var conflicts = self.hasConflicts();
 55    if (!conflicts) {
 56      etag = newEtag;
 57    }
 58    return !conflicts;
 59  };
 60  
 61  /*
 62  Given an object containing errors, we want to apply each of these
 63  errors onto the relevant field. We want to remove any errors that are
 64  already on any field.
 65  
 66  If we have any errors leftover, we need to notify globally, using the
 67  parentCollection's messages object.
 68  */
 69  var markErrors = function(errors) {
 70    $.each(attributes(), function(i,attr){
 71      if (!self[attr].errors) {
 72        self[attr].errors = ko.observableArray([]);
 73      }
 74      if (errors[attr]) {
 75        self[attr].errors(errors[attr]);
 76        delete errors[attr];
 77      } else {
 78        self[attr].errors([]);
 79      }
 80    });
 81    
 82    $.each(errors, function(field){
 83      parentCollection.messages({type:"error", message: field + ": " + errors[field].join("<br>"), timeout: 3000});
 84    });
 85  };
 86  
 87  /*
 88    Get the attributes ready for sending to the server.
 89    
 90    We can't just iterate through properties, as some will not apply. We
 91    use the convention that we will only send back properties that the
 92    server sent to us.
 93  */
 94  var prepareAttributes = function() {
 95    var data = {};
 96    $.each(attributes(), function(i,attr){
 97      data[attr] = self[attr]();
 98    });
 99    return data;
100  };
101  /* Public methods */
102  
103  /*
104    Update the data fields associated with this object from the provided
105    data.
106    
107    This may create new attributes, which need to be noted so we can send
108    those values back to the server.
109    
110    We can mark all updated attributes as not dirty, not conflicted, and
111    not having errors.
112  */
113  self.updateData = function(data) {
114    if (data.links) {
115      // We want to store the links, but not attach them to the object.
116      links = data.links;
117      delete data.links;
118      $.each(links, function(i, obj){
119        if (obj.rel === "self") {
120          url = obj.uri;
121        }
122      });
123    }
124    
125    $.each(data, function(attr, value){
126      if (attributes().indexOf(attr) < 0) {
127        self[attr] = ko.observable(value);
128        self[attr].errors = ko.observableArray([]);
129        self[attr].conflicts = ko.observableArray([]);
130        ko.dirtyFlag(self[attr], false);
131        // Need to add this last to cause the dirtyFields dependentObservable
132        // to work correctly when editing the last field.
133        attributes.push(attr);
134      } else {
135        self[attr](value);
136        self[attr].errors([]);
137        self[attr].conflicts([]);
138        self[attr].isDirty.reset();
139      }
140    });
141    // Put the links back in case a post-processor needs them.
142    data.links = links;
143  };
144  
145  self.serialize = function(evt) {
146    return JSON.stringify(prepareAttributes());
147  };
148  
149  /*
150    Discard any local changes, and pull the data from the server.
151  */
152  self.revert = function(evt) {
153    etag = null;
154    self.fetch();
155    parentCollection.messages({type:'warning', message:'The object "' + self.name() + '" was reverted to the version stored on the server.', timeout: 5000});
156  };
157  
158  /*
159    Attempt to save the data to the server.
160    
161    Only permitted to do this if we have successfully fetched the data
162    at some point.
163    
164    Notes: We use POST instead of PUT, in case we do not have access to
165           all of the fields of the object. PUT implies the complete resource
166           is being updated.
167           Errors may come back in {'field-errors': []}, or {'detail':[]}.
168           Currently, this makes assumptions about server type, which are
169           bad. I need to refactor the error handling code. (400,409)
170           Precondition Failed (412) needs to be handled differently, as
171           we need to fetch the data from the server if none was provided
172           as to the current state of the resource.
173           
174  */
175  self.save = function(evt) {
176    if (self.isFetched()) {
177      $.ajax({
178        url: url,
179        type: 'post', // We can't PUT in case we don't know about all fields.
180        headers: {'If-Match': etag},
181        data: self.serialize(),
182        statusCode: {
183          200: function(data, textStatus, jqXHR) {
184            // Object saved.
185            // Incase some fields were reformatted by the server, redo our data.
186            self.updateData(data);
187            etag = jqXHR.getResponseHeader('Etag');
188            parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was saved.', timeout: 2500});
189            self.isDirty(false);
190          },
191          201: function(data, textStatus, jqXHR) {
192            // Object saved for the first time (created)
193            // Incase some fields were reformatted by the server, redo our data.
194            self.updateData(data);
195            etag = jqXHR.getResponseHeader('Etag');
196            url = jqXHR.getResponseHeader('Location');
197            parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was created.', timeout: 2500});
198            self.isDirty(false);
199          },
200          400: function(jqXHR, textStatus, errorThrown) {
201            var data = JSON.parse(jqXHR.responseText);
202            if (data['field-errors']) {
203              markErrors(data['field-errors']);
204            }
205            parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be saved. Please check the highlighted field(s).', timeout: 10000});
206          },
207          409: function(jqXHR, textStatus, errorThrown) {
208            // Errors saving the data. Likely to be validation errors.
209            // We should have a detail object with info to display.
210            var data = JSON.parse(jqXHR.responseText);
211            if (data.detail) {
212              markErrors(data.detail);
213            }
214            parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be saved. Please check the highlighted field(s).', timeout: 10000});
215          },
216          412: function(jqXHR, textStatus, errorThrown) {
217            // Data was changed on server since we last fetched it.
218            // There may be conflicts to deal with.
219            // See if the server gave us a current version back...
220            var data, serverEtag;
221            if (jqXHR.responseText) {
222              data = JSON.parse(jqXHR.responseText);
223            } else {
224              $.ajax({
225                url: url,
226                async: false,
227                success: function(newData, textStatus, jqXHR) {
228                  data = newData;
229                  serverEtag = jqXHR.getResponseHeader('Etag');
230                }
231              });
232            }
233            if (parseConflicts(data, serverEtag)) {
234              // We were able to resolve all of the conflicts, now we can
235              // try to re-save; but only if it was the first time we saved,
236              // to prevent inifinite recursion.
237              if (evt) {
238                self.save();
239              }
240            } else {
241              parentCollection.messages({type:'error', message:'The object "' + self.name() + '" has been modified on the server. Please check the changed field(s) and select the appropriate value(s).', timeout: 10000});
242            }
243          }
244        }
245      });
246    }
247  };
248  
249  /*
250    Permanently delete the object from the server.
251  */
252  self.destroy = function(evt) {
253    if (self.isFetched() && etag) {
254      console.log("DELETING ITEM");
255      $.ajax({
256        url: url,
257        type: 'delete',
258        headers: {'If-Match': etag},
259        success: function(data, textStatus, jqXHR) {
260          if (collection) {
261            collection.items.remove(self);
262          }
263          parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was deleted.', timeout: 2500});
264        },
265        error: function(jqXHR, textStatus, errorThrown) {
266          // Display error message about not being able to delete?
267          parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be deleted.', timeout: 10000});
268        }
269      });
270    }
271  };
272  
273  /*
274    (Re)Fetch the resource from the server.
275    
276    Handle conflicts if the arise (when the object has already been fetchd)
277  */
278  self.fetch = function(evt) {
279    var headers = {};
280    if (etag) {
281      headers['If-None-Match'] = etag;
282    }
283    $.ajax({
284      type: 'get',
285      url: url,
286      headers: headers,
287      statusCode: {
288        200: function(data, textStatus, jqXHR) {
289          // If we have an etag already, this means the object has been
290          // updated on the server, and we need to look for conflicts.
291          if (etag) {
292            var serverEtag = jqXHR.getResponseHeader('Etag');
293            // If we were unable to handle all conflicts, we need to exit.
294            if (!parseConflicts(data, serverEtag)) {
295              parentCollection.messages({type:'error', message:'The object "' + self.name() + '" has been modified on the server. Please check the changed field(s) and select the appropriate value(s).', timeout: 10000});
296              return;
297            };
298          }
299          
300          // Otherwise, we can update the data and the etag.
301          self.updateData(data);
302          etag = jqXHR.getResponseHeader('Etag');
303          self.isFetched(true);
304        },
305        304: function() {
306        }
307      },
308      error: function(jqXHR, textStatus, errorThrown) {
309        parentCollection.messages({type:"error", message:"There was an error fetching the data from the server"});
310      }
311    });
312  };
313  
314  
315  
316  /* Dependent Observables */
317  self.dirtyFields = ko.dependentObservable(function(){
318    return ko.utils.arrayFilter(attributes(), function(attr){
319      return self[attr] && self[attr].isDirty && self[attr].isDirty();
320    });
321  });
322  
323  self.conflictedFields = ko.dependentObservable(function() {
324    return ko.utils.arrayFilter(attributes(), function(attr){
325      return self[attr] && self[attr].conflicts && self[attr].conflicts().length > 0;
326    });
327  });
328  
329  var filterAttributes = function(property) {
330    return function() {
331      return ko.utils.arrayFilter(attributes(), function(attr){
332        return self[attr] && self[attr][property] && self[attr][property]().length > 0;
333      }).length > 0;
334    };      
335  };
336  
337  self.hasErrors = ko.dependentObservable(filterAttributes('errors'));
338  self.hasConflicts = ko.dependentObservable(filterAttributes('conflicts'));
339  
340  /*
341    An object is dirty when:
342      - any of its fields/attributes are dirty. (we aks them), OR
343      - we have explicitly marked it as dirty.
344      
345    We need to do the latter for when we have merged a conflict, by choosing
346    our value, which differed from the server. The local model would
347    normally think it wasn't dirty, but it differs from the server, and
348    does need to be saved.
349  */
350  self.isDirty = ko.dependentObservable({
351    read: function() {
352      return self.dirtyFields().length > 0 || dirtyFlag();
353    },
354    write: function(value) {
355      dirtyFlag(value);
356      if (!value) {
357        $.each(attributes(), function(attr){
358          if (self[attr] && self[attr].isDirty) {
359            console.log(attr);
360            self[attr].isDirty.reset();          
361          }
362        });
363      }
364    }
365  });
366  
367  /*
368    Can this object be saved back to the server?
369    Only when it is dirty, and has been fetched.
370  */
371  self.canSave = ko.dependentObservable(function() {
372    return self.isDirty() && self.isFetched();
373  });
374  
375  self._etag = function(){return etag;};
376  self._attributes = function(){ return attributes();};
377  self._url = function() {return url;};
378  
379  if (initialData) {
380    self.updateData(initialData);
381  }
382  
383  return self;
384};

Cascading Selects: Leaf Node Only

I’ve written quite a lot of stuff about trees in Postgres, but not quite so much about how they might be used. One situation where they could be used is in categories: where each category may have sub-categories, and those may or may not have further sub-categories. When editing an object that has a single category, it would be useful to only show the values at each level, and then making a selection results in the display of another select element, that only has those values that apply at that level, and with that chain of parents selected.

Changing a selection should remove all subsequent items, and redisplay the “next” select with the new available options.

As you select different options from the “Category” select, you will see that only the leaf node is shown in the box on the right: but also (to show that it’s not just rendering the last value), we have a second “category” key/value pair from the text input.

This is a bit of a proof of concept: it doesn’t actually do an AJAX request: although there is a commented out fetch() request that shows where this would occur, but it doesn’t include the current selection.

The responses from the server would need to look something like:

[
  {"value": 8, "label": "Eight"},
  {"value": 9, "label": "Nine"},
  {"value": 10, "label": "Ten"},
  {"value": 11, "label": "Eleven"}
]

jQuery long-poll for CouchDB changes.

I spent a bit of time this weekend playing with CouchDB. I think I have almost figured out how to do some stuff that had been bugging me. Coming to a document-based store after so much time and effort on relational DBMS does mean you really need to approach things from a different direction.

Our project is basically a multi-tenanted hosted application. Each customer has their own data, and within a relational model you basically have a field in each table that references the customer the data belongs to. Either that, or you have a seperate database and installation per-customer, but that doesn’t scale well on a system like ours that get intermittent use throughout the week.

I’m going to talk more about CouchDB and segmenting the data later, but the best solution is to have a single database per-customer. This makes more sense when you know how CouchDB works: a CouchDB database is a container for documents. Grouping these by customer means you can easily replicate one customer’s data, or move it to a different node.

One really nice feature of CouchDB is the changes feed. From this, you can subscribe to all of the changes that occur in a database, and this could be filtered (so a user would only be notified of changes to documents that they have read-access on, for instance).

This could potentially solve lots of problems that we have with different users from the same company working on the same roster at the same time, and those changes automatically appearing in everyone’s browser.

There are three types of change feeds that are interesting:

  1. The list of changes since I last checked (I send in a sequence number).
  2. The same, but handled using long-polling (I keep the request open until a change occurs).
  3. A continuous polling approach, where changes are sent to my open connection as they occur.

I really liked the sound of the last one, but $.stream, the only library I could find that did that for jQuery had some issues: like the fact is sent the request by POST, and that I couldn’t actually get it to see any data that was coming back in.

To solve the problem using #1, you could write some code that keeps track of the sequence number, and runs a request every X seconds. But I liked the idea of long-polling.

The idea I had was to run a request, and in the success handler, recursively call the function. To actually handle the incoming data, I thought that it might be a good solution to use jQuery events. I’m having them triggered on $(document), as I haven’t seen a standard way to do this. The other option might be on $(window), or a passed-in object.

Leveraging HTML and Django Forms: Pagination of Filtered Results

Django’s forms are fantastic for parsing user input, but I’ve come up with a nice way to use them, in conjunction with HTML forms, for pagination, using the inbuilt Django pagination features.

It all stems from the fact that I’ve begun using forms quite heavily for GET purposes, rather than just for POST. Basically, anytime you have a URL that may have some parts of the query string that may need to be built, it’s simpler to use a form element, than to manually build up the url in your template.

Thus, where you may have something like:

<a href="{% url 'foo' %}?page={{ page }}">

It may be better do do something more like:

<form action="{% url 'foo' %}">
  <input type="hidden" name="page" value="{{ page }}">
</form>

Indeed, you can even use named buttons for submission, which will refer to the page. That is the key to the process outlined below.


Django comes with lots of “batteries”, including form handling and pagination. The Class Based Views (CBV) that deal with collections of objects will include pagination, although it is possible to use this pagination in your own views. For simplicity, we’ll stick with a simple ListView.

Let’s begin with that simple view: in our views.py:

from django.views.generic import ListView, DetailView

from .models import Person

person_list = ListView.as_view(
    queryset=Person.objects.all(),
    template_name='person/list.html',
    paginate_by=10,
)

person_detail = DetailView.as_view(
    queryset=Person.objects.all(),
    template_name='person/detail.html',
)

Essentially, that’s all you need to do. You could use implied template names, but I almost never do this. The takeaway from this block is that we are stating the queryset that our ListView will use as the base, the template it should render, and the number of items per page.

I’ve stubbed out the person_detail view, just so we can refer to it in our urlconf, and then in turn in the template. Because of the simplicity of it, we could have just done all of this in our urls.py.

Speaking of our urls.py, we have something like:

from django.conf.urls import url

import views

urlpatterns = [
    url(r'^people/$', views.person_list, name='person_list'),
    url(r'^people/(?P<pk>\d+)/', views.person_detail, name='person_detail'),
]

Then, in our template, we can render it as (ignoring the majority of the page):

<ul class="people">
  {% for object in object_list %}
    <li>
      <a href="{% url 'person_detail' pk=object.pk %}">
        {{ object }}
      </a>
    </li>
  {% endfor %}
</ul>

But this doesn’t give us our pagination. It will only show the first ten results, with no way to access the others. All we need to do to access the others is to append ?page=X, but, as we will see, there is another way.

Typically, your pagination block might look something like:

<ul class="pagination">
  <li>
    {% if page_obj.has_previous %}
      <a href="?page={{ page_obj.previous_page_number }}">
        prev
      </a>
    {% else %}
      <span>prev</span>
    {% endif %}
  </li>

  {% for page_number in paginator.page_range %}
    {% if page_number = page_obj.number %}
      <li class="active">
        <span>{{ page_number }}</span>
      </li>
    {% else %}
      <li>
        <a href="?page={{ page_number }}">
          {{ page_number }}
        </a>
      </li>
    {% endif %}
  {% endfor %}


  <li>
    {% if page_obj.has_next %}
      <a href="?page={{ page_obj.next_page_number }}">
        next
      </a>
    {% else %}
      <span>next</span>
    {% endif %}
  </li>
</ul>

Depending upon your CSS framework, if you use one, there may already be some pre-prepared styles to help you out with this.

This is all well and good, until you want paginated search results. Then, you can no longer rely on being able to rely on using ?page=N, as this would remove any search terms you were already using. Also, if you were using ajax to fetch and display stuff, you may need to use the whole URL, rather than just the query string.

Instead, we can use a Django form for searching, and just add in the pagination bits.

We will build a page that displays an optionally filtered list of people.

Our Person model will be deliberately simple:

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=256)

Likewise, our form will be simple. All we need to do is have the form able to filter our queryset.

from django import forms

class PersonSearchForm(forms.Form):
    query = forms.CharField(label=_('Filter'), required=False)

    def filter_queryset(self, request, queryset):
        if self.cleaned_data['name']:
            return queryset.filter(name__icontains=self.cleaned_data['query'])
        return queryset

Finally, we will need to subclass a ListView. We’ll mixin from FormMixin, so we get the form-handling capabilities:

from django.views.generic.edit import FormMixin
from django.views.generic import ListView

class FilteredListView(FormMixin, ListView):
    def get_form_kwargs(self):
        return {
          'initial': self.get_initial(),
          'prefix': self.get_prefix(),
          'data': self.request.GET or None
        }

    def get(self, request, *args, **kwargs):
        self.object_list = self.get_queryset()

        form = self.get_form(self.get_form_class())

        if form.is_valid():
            self.object_list = form.filter_queryset(request, self.object_list)

        context = self.get_context_data(form=form, object_list=self.object_list)
        return self.render_to_response(context)

There’s a little bit to comment on there: we override the get_form_kwargs so we pull our form’s data from request.GET, instead of the default.

We also override get, so we filter results if the form validates (which it will if there was data provided). We delegate responsibility for the actual filtering to the form class.

Everything else is just standard.

We will want to actually use this view:

people_list = FilteredListView.as_view(
    form_class=PersonSearchForm,
    template_name='person/list.html',
    queryset=Person.objects.all(),
    paginate_by=10
)

Now we need to render this.

<form id="person-list-filter" action="{% url 'person_list' %}">
  <input name="{{ form.query.html_name }}" value="{{ form.query.value }}" type="search">
  <button type="submit" name="page" value="1">{% trans 'Search' %}</button>
</form>

<div class="results">
  {% include 'person/list-results.html' %}
</div>

You may notice that the search button will result in page=1 being used. This is deliberate.

Our person/list-results.html is just the same as what our person/list.html looked like before, with the addition of the pagination template inclusion.

{% include 'pagination.html' with form_target='person-list-filter' %}

<ul class="people">
  {% for object in object_list %}
    <li>
      <a href="{% url 'person_detail' pk=object.pk %}">
        {{ object }}
      </a>
    </li>
  {% endfor %}
</ul>

Our pagination.html is very similar to how our other template above looked too, but using <button> elements instead of <a>, and we will disable those that should not be clickable. Also, the buttons contain an attribute indicating which form they should be bound to.

<ul class="pagination">
  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_previous %}
        name="page"
        value="{{ page_obj.previous_page_number }}"
        type="submit"
      {% else %}
        disabled="disabled"
      {% endif %}>
      prev
    </button>
  </li>

  {% for page_number in paginator.page_range %}
    <li class="{% if page_number == page_obj.number %}active{% endif %}">
      <button
        name="page"
        value="{{ page_number }}"
        type="submit"
        form="{{ form_target }}"
        {% if page_number == page_obj.number %}
          disabled="disabled"
        {% endif %}>
        {{ page_number }}
      </button>
    </li>
  {% endfor %}

  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_next %}
        name="page"
        value="{{ page_obj.next_page_number }}"
        type="submit"
      {% else %}
        disabled="disabled"
      {% endif %}>
      next
    </button>
  </li>
</ul>

We are getting close now. This will be enough to have clicking on the next/previous or page number buttons resubmitting our search form, resulting in the page reloading with the correct results.

But we can do a bit better. We can easily load the results using AJAX, and just insert them into the page.

We just need one additional method on our View class:

class FilteredListView(FormMixin, ListView):
    # ...

    def get_template_names(self):
        if self.request.is_ajax():
            return [self.ajax_template_name]
        return [self.template_name]

    # ...

and one addition to our view declaration:

people_list = FilteredListView.as_view(
    form_class=PersonSearchForm,
    template_name='person/list.html',
    ajax_template_name='person/list-results.html',
    queryset=Person.objects.all(),
    paginate_by=10,
)

I’ll use jQuery, is it makes for easier to follow code:

// Submit handler for our form: submit it using AJAX instead.
$('#person-list-filter').on('submit', function(evt) {
  evt.preventDefault();

  var form = evt.target;

  $.ajax({
    url: form.action,
    data: $(form).serialize(),
    success: function(data) {
      $('#results').html(data)
    }
  });
});

// Because we are using buttons, which ajax submit will not send,
// we need to add a hidden field with the relevant page number
// when we send our request.
$('#person-list-filter').on('click', '[name=page]', function(evt) {
  var $button = $(evt.target).closest('button');
  var $form = $button[0].form;

  if (!$form.find('[type=hidden][name=page]')) {
    $form.append('<input type="hidden" name="page">');
  }

  $form.find('[type=hidden][name=page]').val($button.val());

  $form.submit();
});

That should do nicely.


There is another thing that we need to think about. If we leave the next/prev buttons, then we need to handle multiple clicks on those buttons, which fetch the subsequent page, and possibly cancel the existing AJAX request.

I do have a solution for this, too, although it complicates things a fair bit. First, we need to add some attributes to the next/prev buttons:

<ul class="pagination">
  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_previous %}
        name="page"
        value="{{ page_obj.previous_page_number }}"
        type="submit"
        data-increment="-1"
        data-stop-at="1"
      {% else %}
        disabled="disabled"
      {% endif %}>
      prev
    </button>
  </li>

  {% for page_number in paginator.page_range %}
    <li class="{% if page_number == page_obj.number %}active{% endif %}">
      <button
        name="page"
        value="{{ page_number }}"
        type="submit"
        form="{{ form_target }}"
        {% if page_number == page_obj.number %}
          disabled="disabled"
        {% endif %}>
        {{ page_number }}
      </button>
    </li>
  {% endfor %}

  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_next %}
        name="page"
        value="{{ page_obj.next_page_number }}"
        type="submit"
        data-increment="1"
        data-stop-at="{{ paginator.num_pages }}"
      {% else %}
        disabled="disabled"
      {% endif %}>
      next
    </button>
  </li>
</ul>

And our click handler changes a bit too:

$('#person-list-filter').on('click', 'button[name=page]', function() {
  var page = parseInt(this.value, 10);
  var $form = $(this.form);
  // Only update the value of the hidden form.
  if (!$form.find('[name=page][type=hidden]')) {
    $form.insert('<input name=page type=hidden>');
  }
  $form.find('[name=page][type=hidden]').val(page);
  // Increment any prev/next buttons values by their increment amount,
  // and set the disabled flag on any that have reached their stop-at
  $form.find('[data-increment]').each(function() {
    this.value = parseInt(this.dataset.increment, 10) + page;
    // We want to disable the button if we get to the 'stop-at' value,
    // but this needs to happen after any submit events have occurred.
    if (this.dataset.stopAt) {
      setTimeout(function() {
        this.disabled = (this.value == this.dataset.stopAt);
      }.bind(this), 0);
    }
  });

  $form.submit();
});

Since this was posted, I have written a number of pages that use this pattern. Some of the improvements that could be made are listed below:

It’s possible to have these results automatically update as the user types. Obviously, this only makes sense if we have AJAX submission happening!

$('#person-list-filter').on('keyup', function() {
  this.submit();
})

If you have lots and lots of results, you probably won’t want to show every button. Often you will see the first few, and a couple either side of the current page (and sometimes the last few). This is almost possible to do with pure CSS, but not quite. I do have a solution for this, but it’s probably worthy of a complete post of its own.

Another situation that is likely to happen is this:

  • User clicks on a page other than page 1 of results. Let’s say page N.
  • User enters text in search field which results in fewer than N pages of results being available.
  • User gets error message.

We can fix this with an overridden method:

class FilteredListView(FormMixin, ListView):
    # ...

    def paginate_queryset(self, queryset, page_size):
        try:
            return super(FilteredListView, self).paginate_queryset(queryset, page_size)
        except Http404:
            self.kwargs['page'] = 'last'
            return super(FilteredListView, self).paginate_queryset(queryset, page_size)

    # ...

You’ll also need to add in a get_prefix() method if you are using an old Django, but really you should just upgrade.


Updated: I’ve added in some more error checking into the templates, to prevent exceptions when attempting to render previous and next page links (thanks inoks).

Updated: I’ve changed to use the preferred urlpattern syntax. (thanks knbk).

Updated: Delegate to the form for filtering. Add discussion of other extensions. Add button[form] attributes.

Django AJAX Forms

I think the more Django code I write, the more I like one particular feature.

Forms.

Simple as that. Forms are the reason I keep coming back to django, and discard other web frameworks in other languages, even though I really want to try them.

One pattern I have been using a fair bit, which was touched on in another post, is using AJAX to handle form submission, and displaying the response.

Before we continue, a quick recap on what Django’s forms offer us.

  • A declarative approach to defining the fields a form has, including validation functions.
  • Will render themselves to HTML input elements, as appropriate.
  • Handle validation of incoming form-encoded (or otherwise provided) data.
  • Fields can validate themselves, and can include validation error messages as part of the HTML output.
  • (Model forms) handle instantiation of and updating of model instances.

A normal form-submission cycle contains a POST or GET request to the server, which responds with a fresh HTML page, which the browser renders. The normal pattern for successful POST requests is to redirect to a GET afterwards, to prevent duplicate submission of forms.

By doing an ajax request instead of a full-page request means we can:

  • reduce the amount of data that is sent back from the server
  • improve apparent performance by only re-rendering the relevant data
  • reduce the amount of time spent rendering parts of the page that have not changed, such as menu, etc.

The way I have been doing it, in broad terms, is to have a template just for the form. If the request is an ajax request, then this will be rendered and returned. If it’s not an ajax request, then the full page will be returned.

Some example code, for one way to do this:

def view(request, pk):
  instance = MyModel.objects.get(pk=pk)
  
  if request.is_ajax():
    template = 'form.html'
  else:
    template = 'page.html'
  
  
  if request.method == 'POST':
    form = MyForm(request.POST, instance=instance)
    if form.is_valid():
      form.save()
      if not request.is_ajax():
        return redirect('redirect-name-here')
  else:
    form = MyForm(instance=instance)
  
  return render(request, template, {'form': form})

Our template files. page.html:

{% extends 'base.html' %}

{% block main %}
  {% include 'form.html' %}
{% endblock %}

{% block script %}
{# Assumes jQuery is loaded... #}
{# This should be in a seperate script file #}
<script>
$(document).on('submit', 'form.dynamic-form', function(form) {
  var $form = $(form);
  $.ajax({
    type: form.method,
    url: form.action,
    data: $form.serialize(),
    success: function(data) {
      $form.replace(data);
    }
  });
});
</script>
{% endblock %}

And form.html:

<form action="/path/to/url/" method="POST" class="dynamic-form">
  {% csrf_token %}
  {{ form }}
  <button type="input">Submit</button>
</form>

Obviously, this is a fairly cut-down example, but it gets the message across.

One thing I dislike in general about django is that failed form submissions are returned with a status code of 200: personally I think a 409 is more appropriate in most cases, but returning a 200 actually means this code is simpler.

Django AJAX Edit Mode

I can’t even remember where I saw this, but the suggestion was that viewing and editing data are different operations, and should be different modes.

For instance, when viewing some data, you would need to explicitly decide to enter the edit mode. In a web page, this would be by following a link to a page that had a form that allowed for editing. Attempting to submit the form would result in either the same page being displayed, along with any validation errors, or being redirected back to the viewing page.

This is the pattern I’ve been working on implementing with a module for our work system. However, we can reduce the amount of data sent, and the amount of page redraw, by using dynamic element replacement. So, we can have some AJAX that loads in the edit form in-place, and replaces the view form. Saving it then either returns the edit form with validation errors, or the view form again.

We can do all of this with one view. Depending upon how it is accessed, it will return either a different template, or render the template differently.

Solution One: render a different template.

# views.py
from django.views import generic

class AjaxEditView(generic.UpdateView):
  model = MyModel
  
  def get_template_names(self):
    if self.request.is_ajax():
      if 'edit' in self.request.GET:
        return 'partial/edit.html'
      return 'partial/display.html'
    return 'detail.html'
  
  def get_success_url(self):
    return reverse('mymodel_detail', kwargs={'pk': 1})

The disadvantages of this are that we need to name the url route, and use it here in this view, and that we have different templates. The templates are almost identical, however:

<!-- partial/display.html -->
{% load url from future %}

<form method="GET" action="{% url 'mymodel_detail' object.pk %}">
  <button>EDIT</button>
  <input type="hidden" name="edit" value="1">
  
  <table>
    {% for field in form %}
      <tr>
        <th>{{ field.label }}</th>
        <td>{{ field.value }}</td>
      </tr>
    {% endfor %}
  </table>
</form>
<!-- partial/edit.html -->
{% load url from future %}

<form method="POST" action="{% url 'mymodel_detail' object.pk %}">
  <button type="submit">SAVE</button>
  <a class="cancel" href="{% url 'mymodel_detail' object.pk %}">CANCEL</a>
  
  {% csrf_token %}
  
  <table>
    {% for field in form %}
    <tr>
      <th>{{ field.label }}</th>
      <td>{{ field.errors }}{{ field }}</td>
      <td>{{ field.help_text }}</t>
    </tr>
    {% endfor %}
  </table>
  
</form>

The partial/display.html template has a few things of note: it has a method="GET", and a single <input> element, which tells the view to render the response on a GET request ready for editing.

The partial/edit.html template has a link/button for cancelling. In this case, we could look at the form for the location we should load, but this is a bit more explicit.

Solution Two: context_data variable

The other solution uses just one AJAX template, but adds to the context of the view if it should be rendered in editable form or not.

# views.py
from django.views import generic

class AjaxEditView(generic.UpdateView):
  model = MyModel
  
  def get_context_data(self, **context):
    context['edit'] = self.request.GET.get('edit', False)
    return super(AjaxEditView, self).get_context_data(**context)
    
  def get_template_names(self):
    if self.request.is_ajax():
      return 'partial/form.html'
    return 'detail.html'
  
  def get_success_url(self):
    return reverse('mymodel_detail', kwargs={'pk': 1})

And our template looks like:

<!-- partial/form.html -->
{% load url from future %}

<form method="{% if edit %}POST{% else %}GET{% endif %}" 
      action="{% url 'user_detail' object.pk %}">
  
  <button type="submit">
    {% if edit %}SAVE{% else %}EDIT{% endif %}
  </button>
  
  {% if edit %}
    <a class="cancel" href="{% url 'user_detail' object.pk %}">
      CANCEL
    </a>
    {% csrf_token %}
  {% else %}
    <input name="edit" type="hidden" value="1">
  {% endif %}
  
  <table>
    {% for field in form %}
    <tr>
      <th>{{ field.label }}</th>
      <td>
        {% if edit %}
          {{ field.errors }}{{ field }}
        {% else %}
          {{ field.value }}
        {% endif %}
      </td>
      {% if edit %}
      <td>{{ field.help_text }}</td>
      {% endif %}
    </tr>
    {% endfor %}
  </table>
  
</form>

The downside of this one is that the template is much more complicated. I’ve been using the latter in a work project, but I may switch later.

The last part that ties all of this together is the Javascript. It’s fairly simple, written using jQuery:

$(function() {
  // Submit handler. Simply submit the form, and replace the
  // form in it's entirety with the response from the server.
  $(document).on('submit', 'form', function(evt) {
    var form = evt.target;
    var $form = $(form);
    evt.preventDefault();
    $.ajax({
      url: form.action,
      type: form.method,
      data: $form.serialize(),
      success: function(data){
        $form.replaceWith($.parseHTML(data));
      }
    });
  });
  
  // Cancel editing button handler. Do an ajax fetch on the
  // url of the button, and replace the parent form with
  // the response from the server.
  $(document).on('click', '.cancel', function(evt) {
    evt.preventDefault();
    $.ajax({
      url: evt.href,
      success: function(data){ 
        $(evt.target).closest('form').replaceWith($.parseHTML(data));
      }
    });
  });
});

The other thing that it is possible to do is make it so that the edit button will only display if the logged in user is permitted to edit that object.

In practice, I’m combining multiple display/edit views within one view (for related concepts: for instance Bank account details, Tax File Number and Superannuation details in the one page, but they have seperate models). I have some ideas about a nice way to handle this, but that’s for another post.

There is a project available on bitbucket that demonstrates this: dynamic-form-demo. There is a seperate branch for each solution outlined above.

jQuery dynamic forms

Some javascript, currently using jQuery, that will convert a form into an ajax form.

First up, a shim to enable FormData for non-compliant browsers. Based on formdata.js, but with some changes. Man, callback based code is a bitch when all you need to do is get some value, and have to work around that.

(function(w, $) {
  // Don't override if it is native.
  if (w.FormData)
    return;
      
  function FormData(form) {
    var fd = this;
    this.fake = true;
    this.boundary = "--------FormData" + Math.random();
    this._fields = [];
    this.contentType = 'multipart/form-data; boundary=' + this.boundary;
          
    if (form) {
      var $form = $(form);
      $.each($form.serializeArray(), function(i, obj){
        fd.append(obj.name, obj.value);
      });
      $form.find('[type=file]').each(function(i, file) {
        fd.append(file.name, file.files[0]);
      });
    }
  }
    
  // A listener to automatically add the binary version of the data. This may suck.
  $('[type=file]').change(function updateData(change) {
    var reader = new FileReader();
    reader.onload = function(load) {
      $(change.target.files[0]).data('binary-file-data', load.target.result);
    }
    reader.readAsBinaryString(change.target.files[0]);
  });
    
  // The interface FormData provides...
  FormData.prototype.append = function(key, value) {
    this._fields.push([key, value]);
  }
    
  // But, we will actually look more like a string to XMLHttpRequest.
  FormData.prototype.toString = function() {
    var boundary = this.boundary;
    var body = '';
    $.each(this._fields, function(i, field) {
      body += '--' + boundary + '\r\n';
      if (field[1].name) {
        var file = field[1];
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\"; filename=\""+ file.name +"\"\r\n";
        body += "Content-Type: "+ file.type +"\r\n\r\n";
        body += $(file).data('binary-file-data') + "\r\n";
      } else {
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\";\r\n\r\n";
        body += field[1] + "\r\n";
      }
    });
          
    body += "--" + boundary +"--";
    return body;
  }
  w.FormData = FormData;
})(window, jQuery);

Now, for the jQuery code. This will override the submit event on all forms. If there are any <input type=file> elements, then it will use a FormData object, else it will just .serialize() the object. The response will override the .html() content of the form, but not the form itself.

$('form').submit(function(evt) {
  evt.preventDefault();
  var form = evt.target, $form = $(form), data;
  var options = {
    cache: false,
    type: form.method,
    url: form.action,
    done: function(data) {
      $form.html(data);
    },
    fail: function(xhr, status, error) {
      console.log(status, error, xhr);
    }
  };
    
  if ($form.find('[type=file]').length) {
    data = new FormData(form);
    // Native FormData objects set this automatically (how???), but we need to manually do it.
    options.contentType = data.contentType || false;
    options.processData = false;
  } else {
    data = $form.serialize();
  }
  options.data = data;
  $.ajax(options);
});