Schinckel.net

KnockoutJS Watchdog

I first discovered KnockoutJS many years ago. It’s a really neat way of managing dependencies between values, and also updating HTML accordingly. Recently, I rewrote my fronius-dashboard to use a Quart app and KnockoutJS instead of a Phoenix LiveView, as the latter was much harder to build (and dependency changes seem to break that fairly frequently).

One thing I did notice is that from time to time I’ll come back to a pinned tab version of the app and it will no longer be updating. I’m not sure exactly why that is, but it seems to be connected still (the EventSource object says it is still connected, anyway).

So, I thought maybe a watchdog process would be in order.

Basically, if there are no updates for a certain period of time, then it should just reload the page.

KnockoutJS doesn’t have anything like this built in, but it does have the facility to throttle updates, and and specifically, to use the “only update when changes stop” facility. As such, we can build a watchdog fairly easily using just that:

let watchdog = ko.computed(() => viewModel.grid()).extend({
  rateLimit: {
    timeout: 10000,
    method: "notifyWhenChangesStop"
  }
});
watchdog.subscribe(() => {
  document.location.reload();
});

In this case, I have one value that will almost certainly update every second (or more frequently than this), called grid.

If there are no updates for 10 seconds, then it will reload the page.

I’m not sure that this will actually solve my problem though - I suspect that the stoppage happens because the browser tab is not frontmost, and the JS is suspended, and doesn’t un-suspend properly. Time will tell, I guess.

KnockoutJS HTML binding

TL;DR: Don’t use KnockoutJS html binding lots of times in your page.

I’m in the middle of rewriting a large part of our application in HTML: for a lot of the interactivity stuff, anything more than just a simple behaviour, I’m turning to KnockoutJS.

Mostly, it’s been awesome. Being able to use two-way binding is the obvious big winner, but dependency tracking is also fantastic.

However, I have had some concerns with performance in the past, and this was always on my mind as I moved into quite a complicated part of the system.

Our approach is that we are not creating a single page application: different parts of the system are at different URLs, and visiting that page loads up the relevant javascript. This is a deliberate tradeoff, mostly because for the forseeable future, our software will not work without a connection to our server: most of the logic related to shift selection is handled by that. We aren’t about to change that.

While rewriting the rostering interface, I initially had Django render the HTML, and I added behaviours. This was possible, and quite fast, however as the behaviours became more complex, I was doing things like sending back scripts that caused other parts of the page to refresh themselves. It was all rather fragile.

So, I went back to KnockoutJS. After a while, I noticed significant slowdowns when dealing with pages that really shouldn’t have been that slow. I’d optimised the database access for the fetching of shifts (and indeed, it is much faster than before), but it felt like Knockout was very sluggish.

I do have quite a few ko.computed() objects, perhaps they were slowing it down? Notably, the function that filters which shifts should be shown where on the page.

So I put some console.time()/timeEnd() calls in place.

Nope: the initial parse of the data runs in less than half a millisecond: instantiating the objects took a while, but the filtering of shifts was taking much less than 100ms.

However, the initial call to ko.applyBindings() was taking several seconds.

The most annoying thing was that when the developer tools were open, it was taking far, far longer!

Eventually, through using the developer tools profiling, I discovered that the slowdown was because of repeated code like:

foo.innerHTML = bar;

Initially, I had thought this slowdown was in KnockoutJS itself, and played around with other ways of binding (such as using the knockout-repeat plugin). Still slow.

Eventually, however, I worked out that it was the act of interacting with the DOM in this manner that was slow. More specifically, the assignation to innerHTML was occurring in the html: binding.

Looking through my source code, I discovered code that looked like:

<span data-bind="html: icon"></span>

And, icon contained the HTML I wanted to put in there:

<i class="icon-ok"></i>

Which was a bad idea to begin with: it conflated UI with data to begin with. So, I replaced the code that looked like:

this.icon = '<i class="icon-ok"></i>';

With:

this.icon = {
  'icon-time': true
};

And then, in the HTML:

<i data-bind="css: icon"></i>

Bingo. All of a sudden, a page that took several seconds to re-render does so in around a second.

It’s important to note that this pattern was repeated several times for each shift: and we have possibly dozens of shifts on a page. When you really need to use the html binding that’s fine, just don’t stick it inside a loop (or worse still, inside a nested loop).

KnockoutJS persistence using Simperium

I really like KnockoutJS. I’ve said that lots of times, but I mean it. It does one thing, two-way bindings between a data model and the GUI elements, really well.

Perhaps my biggest hesitation in using it in a big project is that there is no built-in persistence layer. This would appear to be a situation where something like Backbone has an advantage.

And then, last week, I came across Simperium.

“So,” I thought, “what if you were able to transparently persist KnockoutJS models using Simperium?”

// Assume we have a SIMPERIUM_APP_ID, and a logged in user's access_token.
var simperium = new Simperium(SIMPERIUM_APP_ID, {token: access_token});
// mappingOptions is a ko.mapping mappingOptions object: really only useful
// if your bucket contains homogenous objects.
var store = new BucketMapping(simperium.bucket(BUCKET_NAME), mappingOptions);

var tony = store.all()[0];

var alan = store.create({
  name: "Alan Tenari",
  date_of_birth: "1965-02-06",
  email: "alan.tenari@example.com"
});

Now, tony is an existing object we loaded up from the server, and alan is one we just created.

Both of these objects are mapped using ko.mapping, but, this is the exciting bit, every time we make a change to any of their attributes, they are automatically persisted back to simperium.

There is a little more to it than that: we may want to only persist valid objects, for instance.

This totally gets me excited. And, I’ve already written a big chunk of the code that actually does this!

But for that, you’ll just have to wait…

Re: HTML is not XAML

This is a response to MVVM on MVC: HTML is not XAML. I attempted to post a comment, first from my iPad, but after a few characters the textarea became unresponsive. Then, from my iMac, I was able to enter a comment, but not post it. It seems like something weird is happening when it shows me the CAPTCHA, and it dismisses the dialog before I can do anything. Disabling JavaScript prevents me from commenting. (But to be honest, using DISQUS does the same).

I’ll start by saying that I’m not a Silverlight developer, indeed, I do nothing with any part of the Microsoft development stack. I don’t think I even have any Microsoft software installed on either my work or home machines. I have been doing a lot of stuff with KnockoutJS lately, though.

Jeremy makes some valid points about designer ↔ developer interactions. Maybe I’m (un)lucky, but my interactions with a designer seem to be that (s)he provides me with an image, and I make the HTML match. Either that, or I do the design work myself. In that case, I design in the browser. Safari’s inspector allows you to alter CSS rules and view their impact live. This also means that my HTML is always as sparse as I can possibly make it.

Before I get to the main point, regarding using bindings inside the HTML, there is one thing I just need to point out. Jeremy has the code:

<div id="menuContainer">
  <ul id="menu">
    <li data-bind="foreach: menuItem">
      <span data-bind="text: name">Name</span>
    </li>
  </ul>
</div>

This would only create one <li> element, with multiple <span> elements. In addition, the Name text is superfluous, and would be replaced when the bindings were applied. To match the intent of the previously listed code, I think he meant (and I’m spelling some stuff differently):

<nav class="main-menu">
  <ul data-bind="foreach: menuItems">
    <li>
      <a data-bind="text: name, attr: {href: url}"></a>
    </li>
  </ul>
</nav>

Jeremy then goes on to discuss a way to have all of these bindings applied using code.

Personally, being able to bind the data declaratively is one thing that really draws me to KnockoutJS. It’s easy to see the HTML structure, and what data is bound to them. In fact, in some ways it reminds me lots of Cocoa Bindings.

One of his beefs is that designers may muck with the code. I think this could be easily remedied by a little education: don’t touch anything that has data-bind="...". This really isn’t that different to don’t touch anything’s id.

But a deeper problem is that by adding the bindings in code means that you can’t see from the HTML what flow control will be used to handle the layout. Assuming you are still able to apply the foreach binding to the ul in the example above if it had an id instead, it’s not obvious that there may be multiple items. Maybe that’s not the greatest example, as it is a list, so there probably will, but foreach can be used anywhere.

And there are more, too: if allows you to have bits that are not rendered (significantly different to just making them invisible). Plus, if you use the with binding, then you would need to keep in your head the nested structure of what you are accessing inside the with block. Do it in HTML, and you can see (assuming you have reasonable indenting).

Jeremy seems to come to the agreement (in one of his comments), that having the application of the bindings in the code makes things even more complicated, but I propose that it even makes them more brittle. No longer are you relying just on the names of the ViewModel’s attributes, but you are also relying on the ids of the HTML elements. And this is the kicker: a binding to a name that no longer exists in the ViewModel will fail when you try to view the page, meaning nothing will work (and you’ll see a nice red message in your console). What will the following code do if there is no matching HTML element?

$('#spelled-this-id-worng').attr('data-bind', 'value: theValue');

It does nothing.

But it does it silently.

Styling radio buttons like a <del>boss</del> segmented button

I quite like the concept of segmented buttons, where you have a list of related buttons, and can select one or more of them. In MacOS X, and iOS, the ones that are selected have a nice indented look.

I’m currently working on a GUI framework for KnockoutJS, and today I had reason to use this type of control. Initially, I had the following markup that I intended to style:

<nav>
  <ul class="segmented">
    <li><a>Organisation</a></li>
    <li><a>Users</a></li>
    <li><a>Units</a></li>
    <li><a>Tags</a></li>
  </ul>
</nav>

But, then it occurred to me that HTML radio buttons are a good fit for this use case. They can be set so that only one of them will be selected, which means you can actually get them to work without using any JavaScript to keep selected status in sync. And the bonus is that the labels will be clickable, so we don’t need JavaScript for associating them with the radio buttons.

<nav class="segmented">
  <input type="radio" name="seg-1" value="Organisation" id="seg-Organisation">
  <label for="seg-Organisation">Organisation</label>
  <input type="radio" name="seg-1" value="Users" id="seg-Users">
  <label for="seg-Users">Users</label>
  <input type="radio" name="seg-1" value="Units" id="seg-Units">
  <label for="seg-Units">Units</label>
  <input type="radio" name="seg-1" value="Tags" id="seg-Tags">
  <label for="seg-Tags">Tags</label>
</nav>

Now, there’s slightly more markup, but that’s okay. So, what does that look like?

Hmm, not quite what we want. We’ll actually want to hide the radio button widgets, and style the labels. Rather than try to have this in the page, here’s one I prepared earlier:

As you can see, this is with no JavaScript.

Obviously, this is fairly crappy styling: it just looks like some Windows buttons. Let’s tart it up a bit. This is the default styling for koui:

As a bonus, I’ve disabled one of the elements.

From the perspective of KnockoutJS, we can use the checked binding applied to the radio buttons to see which one is selected. If you were submitting this inside a form, you may want to not use display: none; on them, as they may not submit under certain browsers. For ko, however, it’s fine.

I’m going to be using this technique within koui: for segmented buttons, which I haven’t worked out a nice way to define bindings for, and for the tab_view binding, which associates these buttons with a view below, containing a choice of data based on the selection.

Update: There is one drawback with the technique I used here. It is detailed at How to fix the broken iPad form label issue.

It’s actually a rather simple solution: All you need to do is stop the propagation of the event. I had played around with re-firing the event onto the input element, but that might fire twice in some browsers, which might be bad (especially if you were using checkboxes, rather than radio buttons!)

KnockoutJS dirty extender.

Ryan Niemeyer is the man with respect to most things KnockoutJS, and I had been using a version of his smart dirty flag in some projects. I recall making it so it didn’t have to bind to a secondary property, but I may be mistaken.

Anyway, with Knockout 2.0, we get extenders. Now, it’s possible to do things like:

var thing = ko.observable(null).extend({dirty: true});

It will then look for ko.extenders.dirty, and call that function with two arguments: the observable, and the argument (in this case, true).

Thus, it is possible to re-implement his dirty flag as an extender:

ko.extenders.dirty = function(target, startDirty) {
  var cleanValue = ko.observable(ko.mapping.toJSON(target));
  var dirtyOverride = ko.observable(ko.utils.unwrapObservable(startDirty));
  
  target.isDirty = ko.computed(function(){
    return dirtyOverride() || ko.mapping.toJSON(target) !== cleanValue();
  });
  
  target.markClean = function(){
    cleanValue(ko.mapping.toJSON(target));
    dirtyOverride(false);
  };
  target.markDirty = function(){
    dirtyOverride(true);
  };
  
  return target;
};

The advantage I think mine has over Ryan’s is that you can mark an observable as dirty (thing.markDirty()), and it will stay dirty until you explicitly mark it as clean (thing.markClean()).

Otherwise, it’s just: thing.isDirty() and you are all good.

Alternatively, you could remove the two helper functions, and implement .isDirty() as a writeable observable, that tests the incoming value and sets the internal cleanValue if it needs to.

Binding to head elements in KnockoutJS

By default, KnockoutJS binds to <body>, by the look of things. If you have something like:

<html>
	<head>
		<title data-bind="text: title"></title>
	</head>
	<body>
		...
		<script>
			var vm = {
				title: ko.observable("This is the page title")
			};
			ko.applyBindings(vm);
		</script>
	</body>
</html>

The title will not be bound. Instead, you’ll need to use (and I’m using jQuery):

<html>
	<head>
		<title data-bind="text: title"></title>
	</head>
	<body>
		...
		<script>
			var vm = {
				title: ko.observable("This is the page title")
			};
			ko.applyBindings(vm, $('html')[0]);
		</script>
	</body>
</html>

You could also apply the binding twice, once to the head, and once to the body.

HSV to RGB in JavaScript

I am writing a set of UI widgets for use in web apps, using the excellent KnockoutJS library. One of the more challenging ones has been the colour picker. Rather than do what everyone else has done, I tried to ape the Apple Colour Picker. But this gives us values in HSV, which aren’t that useful for web.

So I came across a page that has a JavaScript HSV to RGB converter: http://jsres.blogspot.com/2008/01/convert-hsv-to-rgb-equivalent.html. And there are so many things wrong with that code that it hurts.

  • the declared variables r,g,b are not used at all.
  • RGB is defined as an Array, but used as an Object.
  • var_r and friends are not declared, and pollute the global namespace.

Plus, more I came across as I worked through the code.

So, I thought I’d clean it up, and make it a bit easier to follow.

Some of the bits that are ’tricky’ are the use of toString(16), which converts a number to a base 16 representation, and the ("0" + value).slice(-2), which zero-pads a string.

The algorithm itself is fairly easy to follow: there are seven possible cases for the data conversion. If the saturation is 0, then RGB is #000000.

Otherwise, the value depends on the value of Math.floor(h/60). There is a simple lookup table (data), which stores three values based on hue, saturation and value, and then it’s just a matter of picking the correct two to use with the value, and returning that.

django and jQuery templates

KnockoutJS is a great way to create relationships between data objects, and interface elements. You can, for instance, bind a date value to an html input[type=date] element, and have it converted into a proper date object. You could then display data based on this, or do anything else you wanted.

KnockoutJS 1.2 (the currently stable version) defaults to using jQuery templates (jQuery-tmpl), which happen to use conflicting syntax to django templates.

For instance, if you were to have the following in your django template file:

{{if foo > bar }}
  <div>Stuff Here</div>
{{/if }}

Then django would attempt to process that, as it uses bits that look like django’s template engine’s value placeholder.

A workaround to this is to look at doing something like wrapping any jQuery templates in something that prevents django from interpreting it.

But I don’t like that solution. For starters, almost every text editor will try to syntax highlight data between <script> tags as javascript, even when explicitly marked as <script type="text/html"> or any other non-javascript mime type.

So, it would be nicest (and cleanest) to be able to have each jQuery template item in a separate file in my project.

Enter {% jquery_template %}. With a custom django templatetag, you can not only have it include the template in your django template, but it will automatically add the script tags, and even add an id.

For instance, you can do:

{% jquery_template 'path/to/template.html' 'templateName'  %}

This will include the data from path/to/template.html, which it finds in any template location, but wrapped in <script type="text/html" id="templateName">.

I have a django app that contains this template tag, as well as some other useful stuff for jquery, and other javascript stuff (including knockoutjs). You can see this template tag at: jquery_template.py.

Hope it’s useful.

Knockout Collection

I am loving KnockoutJS. It makes it super easy to bind data values to UI elements in a declarative manner. You no longer have to worry about callbacks updating your data model and/or your view widgets.

The addition to KnockoutJS that I have been working on is a ‘collection’, that can be used to contain a set of objects, which can be fetched from a server, and each of which has it’s own resource URI that will be used to update or delete it.

For instance, we may have a collection URI:

GET "http://example.com/people/"

When we access this using a GET request, we might see something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[
  {
    "first_name": "Adam",
    "last_name": "Smith",
    "links": [
      {"rel":"self", "uri": "http://example.com/people/552/"}
    ]
  },
  {
    "first_name": "John",
    "last_name": "Citizen",
    "links": [
      {"rel":"self", "uri": "http://example.com/people/32/"}
    ]
  }  
]

Each linked resource contains the full (or as much as the logged-in user is able to see) representation. Example:

GET "http://example.com/people/552/"

1
2
3
4
5
6
7
8
9

{
  "first_name": "Adam",
  "last_name": "Smith",
  "date_of_birth": "1910-02-11",
  "email": "adam.smith@example.com",
  "links": [
    {"rel":"self", "uri": "http://example.com/people/552/"}
  ]
}

Now, this is just the beginning. Obviously, we want to turn all of these fields into observables. I also wanted to know when any data had changed (so the “Save” button can be disabled when the object is not dirty). Clearly, being able to write the data back to the server, as well as create new objects, and delete them. Further, I needed to be able to do conditional reads and writes (only allow the object to be saved if no-one else has touched it since we last fetched it).

The place where the ko.mapping plugin broke down for me was that updating the resource from the full representation didn’t add the new fields that came back from the server. It may be that indeed this is possible (I think it is), but at the time, I could not see how to do this. It may be that I will rewrite this to use the ko.mapping stuff, but I’m not so sure right now.

Anyway, after a couple of revisions, I have a working framework.

To use it, you can just do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// Add a dependentObservable called 'name'.
var processPerson = function(item) {
  item.name = ko.dependentObservable(function(){
    return item.first_name() + ' ' + item.last_name();
  });
};

var people = ko.collection({
  url: "http://example.com/people/",
  processItem: processPerson
});

There is one main caveat at this stage:

  • It is expected that each object will have a ’name’ property. If your server does not return one, you’ll need to setup a dependentObservable as shown in processPerson above.

First, the ko.collection object:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273

ko.collection = function(options) {
  // Let jQuery know we always want JSON
  $.ajaxSetup({
    contentType: 'application/json',
    dataType: 'json',
    cache: false // This is browser cache! Needs to be set for Firefox.
  });
  
  options = options || {};
  var url = options.url;                  // Allow passing in a url.
  var processItem = options.processItem;  // Allow passing in a function to process each item after it is fetched.
  var etag;
  
  
  // Initial setup. We need to set these early so we can access them, even
  // if we have no data for them.
  var self = {
    items: ko.observableArray([]),
    selectedItem: ko.observable(null),
    selectedIndexes: ko.observableArray([]),
    filters: ko.observable({})
  };
  
  /*
  Message handling.
  
  We have a messages observableArray, but we use this dependent observable
  to access it. This allows us to have messages that expire.
  
  self.messages() => provide access to the array of messages.
  self.messages({
    type: "error|notice|warning|whatever",    => This will usually be used to apply a class
    message: "Text of message",               => This text will be displayed
    timeout: 1500                             => If this is non-zero, message expires (and 
                                                 is automatically removed after this many 
                                                 milliseconds)
  });
  
  Every message object gets given a callback function (.remove()), that,
  when executed, well immediately remove that message, and get rid of the
  timer that normally removes that message after timeout.
  
  The messages object is also given a flush() function, that will remove
  all of the messages within it.
  
  Not sure if I should move this to a seperate plugin?
  */
  var messages = ko.observableArray([]);
  self.messages = ko.dependentObservable({
    read: function() {
      return messages();
    },
    write: function(message) {
      var timeout;
      message.remove = function() {
        messages.remove(message);
        clearTimeout(timeout);
      };
      messages.remove(function(item) {
        return item.message === message.message;
      });
      messages.push(message);
      if (message.timeout) {
        timeout = setTimeout(function(){
          messages.remove(message);
        }, message.timeout);
      }
    }
  });
  self.messages.flush = function() {
    $.each(messages, function(message){
      message.remove();
    });
  };
    
  /*
  filteredItems : a subset of self.items() that has been passed through
                  all of the self.filters(), and selects only those that
                  match. A filter must be an object of the form:
                  {
                    value: ko.observable(""),
                    attr: "name",
                    test: function(test_value, obj_value) {}
                  }
                  
                  The filtering code handles getting the correct values to
                  pass to the test function, the attr is the name of the 
                  attribute on each member of self.items() that will be
                  tested.
                  Having 'value' passed in means we can have a default
                  value when app starts.
  */
  self.filteredItems = ko.dependentObservable(function() {
    var filteredItems = self.items();
    $.each(self.filters(), function(name, filt){
      filteredItems = ko.utils.arrayFilter(filteredItems, function(item){
        if (!filt.attr || !item[filt.attr]) {
          return true;
        }
        return filt.test(filt.value(), item[filt.attr]());
      });
    });
    return filteredItems;
  });
  
  /*
    This is really only used by a select[multiple] object, and is used in
    conjunction with selectedIndexes.
    
    TODO: make this a writeable dependentObservable.
  */
  self.selectedItems = ko.dependentObservable(function() {
    return self.items().filter(function(el){
      return $.inArray(self.items().indexOf(el), self.selectedIndexes()) >= 0;
    });
  });
  
  /*
    Filter self.items() finding only those that have at least one attribute
    that is marked as dirty.
  */
  self.dirtyItems = ko.dependentObservable(function() {
    return self.items().filter(function(el){
      return el.isDirty();
    });
  });
  
  /*
    Filter self.items(), finding only those that have at least one attribute
    marked as conflicted.
  */
  self.conflictedItems = ko.dependentObservable(function() {
    return self.items().filter(function(el){
      return el.hasConflicts();
    });
  });
  
  self.setSource = function(newUrl) {
    url = newUrl;
  };
  
  /*
    Fetch all items from the url we have for the index.
    
    It is allowable that the index does not return the full body of each
    item, but instead only contains perhaps a name, and links for that
    item. Then, we can use self.selectedItem().fetch() to get the full
    data for the item.
  */
  self.fetchItems = function() {
    if (!url) {
      return;
    }
    var headers = {};
    if (etag) {
      headers['If-None-Match'] = etag;
    }
    $.ajax({
      url: url,
      type: "get",
      headers: headers,
      statusCode: {
        200: function(data, textStatus, jqXHR) {
          // Successful. If we already had objects, then
          // we need to update that list.
          $.each(self.items(), function(i, item){
            // Is there an item in the new data items list that matches
            // the item we are now looking at?
            var matchingItem = data.filter(function(el){
              links = el.links.filter(function(link){
                return link.rel==="self";
              });
              return links[0] && links[0].uri === item._url();
            })[0];
            if (matchingItem) {
              // Update the item that matched.
              item.updateData(matchingItem);
              if (processItem) {
                processItem(item);
              }
              // Remove from data.
              data.splice(data.indexOf(matchingItem), 1);
              // Not sure if this should be here.
              // item.isDirty(false);
            } else {
              // Not found in incoming data: remove from our local store.
              // Will this break $.each(self.items(), ...) ?
              self.items.remove(item);
            }
          });
          
          // Any items that we have left in data (which will be all if we
          // haven't loaded this up before) now need to be added to items().
          // On a clean fetch, this will be the first code that is run.
          $.each(data, function(i, el){
            var item = ko.collectionItem(el, self);
            if (processItem) {
              processItem(item, el);
            }
            self.items.push(item);
          });
          
          // Finally, update the etag.
          etag = jqXHR.getResponseHeader('Etag');
        }
      }
    });
  };
  
  /*
    A shortcut method that allows us to bind an action to fetch the
    data from the server for the currently selected item.
  */
  self.fetchSelectedItemDetail = function(evt) {
    if (self.selectedItem && self.selectedItem()) {
      self.selectedItem().fetch();
    }
  };
  
  /*
    Create an item. I haven't implemented this yet, because I haven't 
    figured out a way to see what fields are needed to be created when
    there are no currently loaded items. I'm thinking about using a
    Wizard in my application, so this might be overridden by the app.
  */
  self.createItem = function(evt) {
    console.log("ADDING ITEM (NOT FINISHED YET)");
    // The trick here is knowing what fields need to be created.
    // self.items.push(ko.collectionItem({}));
  };
  
  /*
    Permanently remove the selectedItem, and delete it on the server.
  */
  self.removeSelectedItem = function(evt) {
    if (self.selectedItem && self.selectedItem()) {
      var sure = confirm("This will permanently remove " + self.selectedItem().name());
      if (sure){
        self.selectedItem().destroy();        
      }
    }
  };
  
  /*
    Iterate through self.items(), finding those that match all of the data
    we pass in.
    
    For instance, you can do things like: 
    
      viewModel.findMatchingItems({date_of_birth: "1995-01-01"})
    
    This is used internally to find matches for objects when updating. Not
    sure why it is exposed as a public member function though.
  */
  self.findMatchingItems = function(options) {
    return self.items().filter(function(el){
      var match = true;
      $.each(options, function(opt, val) {
        if (el[opt]() !== val) {
          // Returning false causes $.each to stop, too.
          return match = false;
        }
      });
      return match;
    });
  };
    
  if (url) {
    self.fetchItems();
  }
  
  return ko.observable(self);
};

Second, the ko.collectionItem object. This may be eventually hidden in the collection object, as it isn’t really intended to be used seperately.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384

ko.collectionItem = function(initialData, parentCollection) {
  var self = {
    isFetched: ko.observable(false)
  };
  var links = [];
  var etag = null;
  var url = null;
  var attributes = ko.observableArray([]);
  var collection = parentCollection;
  var dirtyFlag = ko.observable(false);
  
  /* Private methods */
  
  /*
    Given the incoming 'data' for this object, look through the fields for
    things that differ between the server representation and the client
    representation. Store both values for any differences in an attribute
    of the observable called conflicts().
    
    For each conflict, create a member function on the observable that
    allows you to resolve the conflict. When the last conflict is resolved,
    our etag is updated to the value the server gave us.
    
    This method returns true if all conflicts could be resolved (ie, the
    data in all fields was the same, just the etag had changed).
  */
  var parseConflicts = function(data, newEtag) {
    $.each(data, function(attr, value){
      if (attr !== "links") {
        if ($.compare(value, self[attr]() === undefined ? "" : self[attr]())) {
          // Server and client values match.
          // We need to do some funky stuff with undefined values, and treat
          // them as "". I don't really like this, but it works for now.
          self[attr].conflicts([]);
          self[attr].resolveConflict = function(){};
        } else {
          self[attr].conflicts([value, self[attr]() === undefined ? "" : self[attr]()]);
          self[attr].resolveConflict = function(chosenValue) {
            // Mark the entire object as dirty, so we can allow it to be
            // saved, even if we set it to the original value we had (which
            // differed from the server's value).
            self.isDirty(true);
            self[attr](chosenValue);
            self[attr].conflicts([]);
            if (!self.hasConflicts()) {
              // If this was the last conflict, we can use the new etag from
              // the server.
              etag = newEtag;
            }
          };
        }        
      }
    });
    var conflicts = self.hasConflicts();
    if (!conflicts) {
      etag = newEtag;
    }
    return !conflicts;
  };
  
  /*
  Given an object containing errors, we want to apply each of these
  errors onto the relevant field. We want to remove any errors that are
  already on any field.
  
  If we have any errors leftover, we need to notify globally, using the
  parentCollection's messages object.
  */
  var markErrors = function(errors) {
    $.each(attributes(), function(i,attr){
      if (!self[attr].errors) {
        self[attr].errors = ko.observableArray([]);
      }
      if (errors[attr]) {
        self[attr].errors(errors[attr]);
        delete errors[attr];
      } else {
        self[attr].errors([]);
      }
    });
    
    $.each(errors, function(field){
      parentCollection.messages({type:"error", message: field + ": " + errors[field].join("<br>"), timeout: 3000});
    });
  };
  
  /*
    Get the attributes ready for sending to the server.
    
    We can't just iterate through properties, as some will not apply. We
    use the convention that we will only send back properties that the
    server sent to us.
  */
  var prepareAttributes = function() {
    var data = {};
    $.each(attributes(), function(i,attr){
      data[attr] = self[attr]();
    });
    return data;
  };
  /* Public methods */
  
  /*
    Update the data fields associated with this object from the provided
    data.
    
    This may create new attributes, which need to be noted so we can send
    those values back to the server.
    
    We can mark all updated attributes as not dirty, not conflicted, and
    not having errors.
  */
  self.updateData = function(data) {
    if (data.links) {
      // We want to store the links, but not attach them to the object.
      links = data.links;
      delete data.links;
      $.each(links, function(i, obj){
        if (obj.rel === "self") {
          url = obj.uri;
        }
      });
    }
    
    $.each(data, function(attr, value){
      if (attributes().indexOf(attr) < 0) {
        self[attr] = ko.observable(value);
        self[attr].errors = ko.observableArray([]);
        self[attr].conflicts = ko.observableArray([]);
        ko.dirtyFlag(self[attr], false);
        // Need to add this last to cause the dirtyFields dependentObservable
        // to work correctly when editing the last field.
        attributes.push(attr);
      } else {
        self[attr](value);
        self[attr].errors([]);
        self[attr].conflicts([]);
        self[attr].isDirty.reset();
      }
    });
    // Put the links back in case a post-processor needs them.
    data.links = links;
  };
  
  self.serialize = function(evt) {
    return JSON.stringify(prepareAttributes());
  };
  
  /*
    Discard any local changes, and pull the data from the server.
  */
  self.revert = function(evt) {
    etag = null;
    self.fetch();
    parentCollection.messages({type:'warning', message:'The object "' + self.name() + '" was reverted to the version stored on the server.', timeout: 5000});
  };
  
  /*
    Attempt to save the data to the server.
    
    Only permitted to do this if we have successfully fetched the data
    at some point.
    
    Notes: We use POST instead of PUT, in case we do not have access to
           all of the fields of the object. PUT implies the complete resource
           is being updated.
           Errors may come back in {'field-errors': []}, or {'detail':[]}.
           Currently, this makes assumptions about server type, which are
           bad. I need to refactor the error handling code. (400,409)
           Precondition Failed (412) needs to be handled differently, as
           we need to fetch the data from the server if none was provided
           as to the current state of the resource.
           
  */
  self.save = function(evt) {
    if (self.isFetched()) {
      $.ajax({
        url: url,
        type: 'post', // We can't PUT in case we don't know about all fields.
        headers: {'If-Match': etag},
        data: self.serialize(),
        statusCode: {
          200: function(data, textStatus, jqXHR) {
            // Object saved.
            // Incase some fields were reformatted by the server, redo our data.
            self.updateData(data);
            etag = jqXHR.getResponseHeader('Etag');
            parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was saved.', timeout: 2500});
            self.isDirty(false);
          },
          201: function(data, textStatus, jqXHR) {
            // Object saved for the first time (created)
            // Incase some fields were reformatted by the server, redo our data.
            self.updateData(data);
            etag = jqXHR.getResponseHeader('Etag');
            url = jqXHR.getResponseHeader('Location');
            parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was created.', timeout: 2500});
            self.isDirty(false);
          },
          400: function(jqXHR, textStatus, errorThrown) {
            var data = JSON.parse(jqXHR.responseText);
            if (data['field-errors']) {
              markErrors(data['field-errors']);
            }
            parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be saved. Please check the highlighted field(s).', timeout: 10000});
          },
          409: function(jqXHR, textStatus, errorThrown) {
            // Errors saving the data. Likely to be validation errors.
            // We should have a detail object with info to display.
            var data = JSON.parse(jqXHR.responseText);
            if (data.detail) {
              markErrors(data.detail);
            }
            parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be saved. Please check the highlighted field(s).', timeout: 10000});
          },
          412: function(jqXHR, textStatus, errorThrown) {
            // Data was changed on server since we last fetched it.
            // There may be conflicts to deal with.
            // See if the server gave us a current version back...
            var data, serverEtag;
            if (jqXHR.responseText) {
              data = JSON.parse(jqXHR.responseText);
            } else {
              $.ajax({
                url: url,
                async: false,
                success: function(newData, textStatus, jqXHR) {
                  data = newData;
                  serverEtag = jqXHR.getResponseHeader('Etag');
                }
              });
            }
            if (parseConflicts(data, serverEtag)) {
              // We were able to resolve all of the conflicts, now we can
              // try to re-save; but only if it was the first time we saved,
              // to prevent inifinite recursion.
              if (evt) {
                self.save();
              }
            } else {
              parentCollection.messages({type:'error', message:'The object "' + self.name() + '" has been modified on the server. Please check the changed field(s) and select the appropriate value(s).', timeout: 10000});
            }
          }
        }
      });
    }
  };
  
  /*
    Permanently delete the object from the server.
  */
  self.destroy = function(evt) {
    if (self.isFetched() && etag) {
      console.log("DELETING ITEM");
      $.ajax({
        url: url,
        type: 'delete',
        headers: {'If-Match': etag},
        success: function(data, textStatus, jqXHR) {
          if (collection) {
            collection.items.remove(self);
          }
          parentCollection.messages({type:'success', message:'The object "' + self.name() + '" was deleted.', timeout: 2500});
        },
        error: function(jqXHR, textStatus, errorThrown) {
          // Display error message about not being able to delete?
          parentCollection.messages({type:'error', message:'The object "' + self.name() + '" could not be deleted.', timeout: 10000});
        }
      });
    }
  };
  
  /*
    (Re)Fetch the resource from the server.
    
    Handle conflicts if the arise (when the object has already been fetchd)
  */
  self.fetch = function(evt) {
    var headers = {};
    if (etag) {
      headers['If-None-Match'] = etag;
    }
    $.ajax({
      type: 'get',
      url: url,
      headers: headers,
      statusCode: {
        200: function(data, textStatus, jqXHR) {
          // If we have an etag already, this means the object has been
          // updated on the server, and we need to look for conflicts.
          if (etag) {
            var serverEtag = jqXHR.getResponseHeader('Etag');
            // If we were unable to handle all conflicts, we need to exit.
            if (!parseConflicts(data, serverEtag)) {
              parentCollection.messages({type:'error', message:'The object "' + self.name() + '" has been modified on the server. Please check the changed field(s) and select the appropriate value(s).', timeout: 10000});
              return;
            };
          }
          
          // Otherwise, we can update the data and the etag.
          self.updateData(data);
          etag = jqXHR.getResponseHeader('Etag');
          self.isFetched(true);
        },
        304: function() {
        }
      },
      error: function(jqXHR, textStatus, errorThrown) {
        parentCollection.messages({type:"error", message:"There was an error fetching the data from the server"});
      }
    });
  };
  
  
  
  /* Dependent Observables */
  self.dirtyFields = ko.dependentObservable(function(){
    return ko.utils.arrayFilter(attributes(), function(attr){
      return self[attr] && self[attr].isDirty && self[attr].isDirty();
    });
  });
  
  self.conflictedFields = ko.dependentObservable(function() {
    return ko.utils.arrayFilter(attributes(), function(attr){
      return self[attr] && self[attr].conflicts && self[attr].conflicts().length > 0;
    });
  });
  
  var filterAttributes = function(property) {
    return function() {
      return ko.utils.arrayFilter(attributes(), function(attr){
        return self[attr] && self[attr][property] && self[attr][property]().length > 0;
      }).length > 0;
    };      
  };
  
  self.hasErrors = ko.dependentObservable(filterAttributes('errors'));
  self.hasConflicts = ko.dependentObservable(filterAttributes('conflicts'));
  
  /*
    An object is dirty when:
      - any of its fields/attributes are dirty. (we aks them), OR
      - we have explicitly marked it as dirty.
      
    We need to do the latter for when we have merged a conflict, by choosing
    our value, which differed from the server. The local model would
    normally think it wasn't dirty, but it differs from the server, and
    does need to be saved.
  */
  self.isDirty = ko.dependentObservable({
    read: function() {
      return self.dirtyFields().length > 0 || dirtyFlag();
    },
    write: function(value) {
      dirtyFlag(value);
      if (!value) {
        $.each(attributes(), function(attr){
          if (self[attr] && self[attr].isDirty) {
            console.log(attr);
            self[attr].isDirty.reset();          
          }
        });
      }
    }
  });
  
  /*
    Can this object be saved back to the server?
    Only when it is dirty, and has been fetched.
  */
  self.canSave = ko.dependentObservable(function() {
    return self.isDirty() && self.isFetched();
  });
  
  self._etag = function(){return etag;};
  self._attributes = function(){ return attributes();};
  self._url = function() {return url;};
  
  if (initialData) {
    self.updateData(initialData);
  }
  
  return self;
};