Get the class of a Django view function

I needed to be able to get the class of a view function, once it had been instantiated via MyView.as_view(). I’d done something similar in the past to get the base callable view, but this was slightly different.

from django.views.generic.base import View

def get_class(func):
    if not getattr(func, 'func_closure', None):
        return
        
    for closure in func.func_closure:
        contents = closure.cell_contents
        
        if not contents:
            continue
        
        if getattr(contents, '__bases__', None) and issubclass(contents, View):
            return contents
        
        result = get_class(contents)
        if result:
            return result

This is a recursive function that does a depth-first search on the function object, until it finds an object that is a class, and is a subclass of django.views.generic.base.View.

You can use it like:

from django.core.urlresolvers import resolve
view = resolve('/path/to/url')

view_class = get_class(view.func)

KnockoutJS HTML binding

TL;DR: Don’t use KnockoutJS html binding lots of times in your page.

I’m in the middle of rewriting a large part of our application in HTML: for a lot of the interactivity stuff, anything more than just a simple behaviour, I’m turning to KnockoutJS.

Mostly, it’s been awesome. Being able to use two-way binding is the obvious big winner, but dependency tracking is also fantastic.

However, I have had some concerns with performance in the past, and this was always on my mind as I moved into quite a complicated part of the system.

Our approach is that we are not creating a single page application: different parts of the system are at different URLs, and visiting that page loads up the relevant javascript. This is a deliberate tradeoff, mostly because for the forseeable future, our software will not work without a connection to our server: most of the logic related to shift selection is handled by that. We aren’t about to change that.

While rewriting the rostering interface, I initially had Django render the HTML, and I added behaviours. This was possible, and quite fast, however as the behaviours became more complex, I was doing things like sending back scripts that caused other parts of the page to refresh themselves. It was all rather fragile.

So, I went back to KnockoutJS. After a while, I noticed significant slowdowns when dealing with pages that really shouldn’t have been that slow. I’d optimised the database access for the fetching of shifts (and indeed, it is much faster than before), but it felt like Knockout was very sluggish.

I do have quite a few ko.computed() objects, perhaps they were slowing it down? Notably, the function that filters which shifts should be shown where on the page.

So I put some console.time()/timeEnd() calls in place.

Nope: the initial parse of the data runs in less than half a millisecond: instantiating the objects took a while, but the filtering of shifts was taking much less than 100ms.

However, the initial call to ko.applyBindings() was taking several seconds.

The most annoying thing was that when the developer tools were open, it was taking far, far longer!

Eventually, through using the developer tools profiling, I discovered that the slowdown was because of repeated code like:

foo.innerHTML = bar;

Initially, I had thought this slowdown was in KnockoutJS itself, and played around with other ways of binding (such as using the knockout-repeat plugin). Still slow.

Eventually, however, I worked out that it was the act of interacting with the DOM in this manner that was slow. More specifically, the assignation to innerHTML was occurring in the html: binding.

Looking through my source code, I discovered code that looked like:

<span data-bind="html: icon"></span>

And, icon contained the HTML I wanted to put in there:

<i class="icon-ok"></i>

Which was a bad idea to begin with: it conflated UI with data to begin with. So, I replaced the code that looked like:

this.icon = '<i class="icon-ok"></i>';

With:

this.icon = {
  'icon-time': true
};

And then, in the HTML:

<i data-bind="css: icon"></i>

Bingo. All of a sudden, a page that took several seconds to re-render does so in around a second.

It’s important to note that this pattern was repeated several times for each shift: and we have possibly dozens of shifts on a page. When you really need to use the html binding that’s fine, just don’t stick it inside a loop (or worse still, inside a nested loop).

Scheme line "values"

Years ago, when I first saw TextMate demonstrated, one of the ways it was used as a teaching tool, when teaching Ruby, was to have the current line executed, and the value it returned appended to the current line:

(2 + 3) * 4 / 5 # => 4

That is, pressing Cmd-Shift-Ctrl-E would execute the line, and update the marker.

Today, while playing around with Scheme, I came up with a neat way to do the same type of thing.

Initially, I made it so that it executed the current line, and added/updated the marker. Then, I realised I could load the file, and then execute the current line.

You can create a new bundle command, and bind it to whatever key you want, with a scope selector of source.scheme, Input of Line, Output of Replace Input.

#!/usr/bin/env bash

[[ -f "${TM_SUPPORT_PATH}/lib/bash_init.sh" ]] && . "${TM_SUPPORT_PATH}/lib/bash_init.sh"

# Evaluate the current line in our Scheme interpreter
#
# The interpreter you use should be set in the environment
# variable TM_SCHEME

# The whole file will be loaded, and the current line's value executed,
# and added to the line as a comment.

INTERPRET=${TM_SCHEME}
CMD=$(basename "$INTERPRET")

LINE=`cat /dev/stdin | sed 's/; =>.*//'`
VALUE=`echo $LINE | $INTERPRET --load $TM_FILEPATH | grep ';Value: ' | sed 's/;Value: //'`

echo -n $LINE "; =>" $VALUE

Unfortunately, trailing comments are handled as a seperate line, so getting the ruby-like behaviour of updating all of the ; => comments will have to wait for another day.

django-boardinghouse

I wrote a heap of code last April, under the name Multi-tenanted Django. It was fairly complete, but not especially well documented, and not really that well tested.

Recently, I’ve been having to write some reporting code at work that dealt with objects that are generated by django-reversion. If I was using tenancy-based partitioning, it would be really easy for me to just fetch the changes that were made to data from a given company: instead I need to do heaps of queries, and lots of filtering.

Which got me enthused on django-multi-schema, which has since been renamed to django-boardinghouse. And, it now has it’s own documentation, and an example project.

I’m still a bit cagey about releasing it to pypi, as the example project is pretty simple, and I’d like to build that (or another project) up a bit to see if I’ve made any more bad decisions: I’ve already changed it to opt-in to seperate schema to opt-out, and added in a configurable SCHEMA_MODEL.

It currently passes all tests under django 1.4 – 1.6, and has some functionality under django 1.7, but the migration handling code is not well tested just yet.

Generating Coverage Badges

Drone.io is a pretty neat (free for open-source projects) continuous integration server. The best feature from my perspective is that it works with BitBucket repositories.

It’s pretty nice having status badges indicating if a build is passing or failing, but even better is also getting a coverage report.

I’ve been using django-coverage for this, and ages ago manually created a set of drone.io style badges, and added a patch to copy the relevant file across. But then, shortly after, drone.io changed their status badge format. I never got around to redoing my badges, as it was pretty time consuming.

Enter Pillow.

from PIL import Image, ImageDraw, ImageFont

SIZE = (95, 18)

BACKGROUND = hex_colour('#4A4A4A')
SUCCESS = hex_colour('#94B944')
WARNING = hex_colour('#E4A83C')
ERROR = hex_colour('#B10610')

# You may need a different font filename if you aren't on a Mac
FONT = ImageFont.truetype(size=10, filename="/Library/Fonts/Arial.ttf")
FONT_SHADOW = hex_colour('#525252')

PADDING_TOP = 3

def build_image(percentage, colour):
    # Create a brand-new Image object, with the background
    # as the main badge colour.
    image = Image.new('RGB', SIZE, color=BACKGROUND)
    drawing = ImageDraw.Draw(image)
    
    # Write the word 'coverage' in our specified font.
    # Fake a text-shadow by drawing the text twice.
    # TODO: Make the text-shadow better.
    drawing.text((8, PADDING_TOP+1), 'coverage', font=FONT, fill=FONT_SHADOW)
    drawing.text((7, PADDING_TOP), 'coverage', font=FONT)
    
    # Do the percentage text.
    # TODO: Make the text-shadow better.
    # TODO: Make the text centred in the coloured box.
    drawing.rectangle([(55, 0), SIZE], colour, colour)
    drawing.text((63, PADDING_TOP+1), '%s%%' % percentage, font=FONT, fill=FONT_SHADOW)
    drawing.text((62, PADDING_TOP), '%s%%' % percentage, font=FONT)

Creating the required RGB tuple from a hex colour is also fairly easy:

def hex_colour(hex):
    if hex[0] == '#':
        hex = hex[1:]
    return int(hex[:2], 16), int(hex[2:4], 16), int(hex[4:6], 16)

Finally, you can just generate an image for every percentage point, and save them:

SUCCESS_CUTOFF = 85
WARNING_CUTOFF = 45

# range(101) -> [0, 1, 2, ..., 99, 100]
for i in range(101):
    file = open('%i.png' % i, 'wb')
    
    if i < WARNING_CUTOFF:
        build_image(i, ERROR).save(file)
    elif i < SUCCESS_CUTOFF:
        build_image(i, WARNING).save(file)
    else:
        build_image(i, SUCCESS).save(file)

It’s not quite perfect: that isn’t quite the font they use, but it will do for now.

Dear ezyreg

I went to register with EzyReg to set up monthly direct debit for my car registration, and to register I was required to enter a secret question or two.

These secret questions were all essentially publicly available information, or things that were inane enough that I probably would not remember them. Thus, they are useless from both sides of the fence: I would not be able to use the meaningless ones to reset a password (not really that much of a problem), but if I used one of the “one true answer” ones, someone could easily discover, for instance, what my Mother’s maiden name was.

I could enter a random value, but I decided to get on my high horse, and supply feedback.

Then, after writing for a few minutes: “You may only enter 1000 characters in feedback”.

What. The. Fuck.

So, after splitting it into two comments, here is the entirety of my comment to the braindead fuckwads who wrote the registration system:

I want to use monthly direct debit to pay my car registration, but with the current requirement to have an ezyreg account, and the requirement that said account is “secured” with secret questions means that I cannot in good faith complete the registration process.

There are well documented flaws with secret questions as a second-level of security, or that can be used to reset or change a password. This becomes an active attack vector that, in the case of someone who uses good password hygiene, partially defeats the processes I have in place to protect access to my accounts.

This is made even worse in the case of your security questions, of which there are very few, and I am unable to create my own.

Some examples of the arguments against security questions, and why they are a security risk, not an improvement:

http://www.oneid.com/blog/passwords-are-bad-but-security-questions-are-worse/

https://www.schneier.com/blog/archives/2005/02/the_curse_of_th.html

http://www.intego.com/mac-security-blog/your-secret-question-may-not-be-so-secret-easy-to-guess-password-retrieval-questions-you-should-avoid-and-why/

Please consider removing this requirement from your account registration process.

pbgrep: grep your clipboard history

I’ve used ClipMenu as my clipboard history manager for several years now: it’s unobtrusive, and does almost exactly what I need.

Except, you can’t search the clipboard history.

I keep thousands of items in my clipboard history, and today I was trying to find a specific item, that I know was in there. And I couldn’t find it after about a minute of scanning through submenus.

Now, ClipMenu can persist it’s history to disk, in ~/Library/Application Support/ClipMenu/clips.data. Which is a binary plist file.

We can view it using plutil:

$ plutil -p ~/Library/Application\ Support/ClipMenu/clips.data -o -

I made the decision to limit searching for single-line clips: this means I can grep for lines that contain:

<string>.*(QUERY).*</string>

Doing single-line matches means I can use grep (or, as I discovered later, ack), which should be faster than firing up a python interpreter.

My first iteration was:

$ plutil -convert xml1 ~/Library/Application\ Support/ClipMenu/clips.data -o - \
  | grep "<string>.*test.*</string>" -o

This works quite well, but includes the XML string tags. I did strip them out using sed, but this is an extra command. It turns out that grep’s regular expressions can’t handle positive lookahead/behind assertions, and Mac OS X’s grep does not support --perl-mode, so I reached for ack:

function pbgrep() {
  plutil -convert xml1 ~/Library/Application\ Support/ClipMenu/clips.data -o - \
    | ack "(?<=<string>).*$1.*(?=</string>)" -o
}

That now takes pride of place in my .bashrc, and I can pbgrep foo to my hearts content.

I guess I could (if there was only one match), put the value back into the clipboard. That might be kind-of nice.

Custom Element Form Submission

Custom elements are starting to get some traction in Web Development. There have been some really nice recent posts, the one which got me back on track was Web Components: Why You’re Already an Expert, but also Custom Elements: defining new elements in HTML and Performance and Custom Elements. Also, Polymer makes heavy use of Custom Elements, and I’ve had a bit of a look there too.

I really like KnockoutJS, so started playing around with that. I have a nicely defined DatePicker element, that would work well being turned into a Custom Element, and allow me to use the Shadow DOM to hide the internals of the code.

However, I soon hit a problem. Browsers, and jQuery, will only add data to a form when the element type is <input>, <select>, <textarea> and <keygen>. In jQuery, for instance, this is hard-coded in as rsubmittable, see source.

This means that you cannot just use a <x-date-picker> element, and have it submit. You still need to use some type of a hidden <input> element, and link that to the value. “Subclassing” <input> is not sufficient.

This seems to be a fairly large oversight, and I have not been able to find anything else on the internet that discusses this.

I haven’t tried “subclassing” <button> yet, to see if it is possible to create custom elements that can be used to submit forms, rather than provide data for them.

Per-command Virtualenv

Recently, I finally got around to re-installing OS X from scratch on my work machine. It was past time it needed to happen, to the extent where I would frequently be unable to wake machine from display sleep, and saving a file in a monitored directory would take the wsgi-monitor package tens of seconds to restart django.

One thing I wanted to do this time was only install stuff as necssary, but also put every pip installed command line tool in it’s own virtualenv. However, this has one drawback, in that it is a little repetitive.

For instance, to install Fabric, my deployment tool of choice:

$ virtualenv ~/.venv/fabric
$ . ~/.venv/fabric/bin/activate
(fabric)$ pip install fabric
(fabric)$ ln -s ~/.venv/bin/fabric /usr/local/bin/

This is fine if you only have one ‘tool’ to install, but something like docutils actually installs a whole stack of command line tools.

What we want, is something like:

  • create the virtualenv
  • get a list of items already in the <virtualenv>/bin
  • install the required tool (and any extra modules)
  • link all of the newly added commands in <virtualenv>/bin to /usr/local/bin

We could just add each <virtualenv>/bin to our path, but that would mean that the first virtualenv created would be used for pip, which I don’t want installed at all.

Additionally, it would be nice to be able to specify a required version of the package to install, and other (non-dependency) packages that should be installed. For instance, I want mercurial_keyring to be installed in the mercurial virtualenv.

This last one is probably less important, as you can just use that virtualenv’s pip to install them after. But the version number stuff might be nice.

virtualenv has the nice ability to be able to create bootstrap scripts, which will do other stuff (like install specific packages). We can co-opt this to build a tool for doing the automatic installation and linking:

import virtualenv, subprocess

data = """
import os, subprocess

def extend_parser(optparse_parser):
    optparse_parser.add_option(
        "--upgrade",
        action="store_true",
        dest="upgrade",
        default=False,
        help="Upgrade package",
    )
    optparse_parser.add_option(
        "--path",
        dest="path",
        default='~/.venv/',
        help="Parent path of virtualenvs"
    )
    optparse_parser.add_option(
        '--package',
        dest="packages",
        action="append",
        help="Other packages to install"
    )
    
def adjust_options(options, args):
    global package
    if not args: 
        return
    package = args[0]
    if '==' in args[0]:
        args[0], version = args[0].split('==', 1)
    args[0] = os.path.join(os.path.expanduser(options.path), args[0])

def after_install(options, home_dir):
    global package
    venv = os.path.join(os.path.expanduser(options.path), home_dir)
    before = os.listdir(os.path.join(venv, 'bin'))
    command = [os.path.join(venv, 'bin', 'pip'), 'install', package]
    if options.upgrade:
        command += ['--upgrade']
    if options.packages:
        command += options.packages
    subprocess.call(command)
    after = os.listdir(os.path.join(venv, 'bin'))
    
    for command in set(after).difference(before):
        subprocess.call([
            'ln', '-s', 
            os.path.join(venv, 'bin', command),
            '/usr/local/bin'
        ])
"""

output = virtualenv.create_bootstrap_script(data)
open('/usr/local/bin/pip-install', 'w').write(output)
subprocess.call(['chmod', '+x', '/usr/local/bin/pip-install'])

There is one caveat: if an existing file is found in /usr/local/bin that matches one that should be linked, it will be ignored. That is, it does not overwrite existing commands. I think this is preferable, as it is marginally safer.

Linking commands like this is better than copying them, as it means you can just do a pip install --upgrade <package> in the relevant virtualenv, and it will upgrade commands. You can also use pip-install <package>==<new-version>, and that should work too. However, if you unlink a command (or remove one that would have linked but failed), and do a pip-install, it will not link the commands that were already installed in that virtualenv.

Anyway, your mileage may vary. I’m using it now, and it seems good.

Duck-punch misbehaving software

Recently, I found myself having to interact with an API that uses SOAP. I’ve been using the SOAPpy package. Which has made it possible, but not exactly easy. But that’s not what I am going to write about right now.

In order to make the linking of data between my software and that system easier, I needed to get a dump of the other system’s data, in a CSV that I could send to the client.

So, since the SOAPpy module gives you something that looks dict-y, I thought I’d just be able to pass it to csv.DictWriter’s writerow() method.

Not quite.

See, whilst it supports the python dict-like [key] syntax, the SOAPpy.Types.structType doesn’t support the .get() method, that DictWriter uses to extract the data (indeed, it needs to, to be able to trap missing keys).

So, here is a simple duck-punch (does it quack like a duck? No? Punch it harder so, it quacks!) that enables you to pass a structType object to a DictWriter.writerow() method call:

from SOAPpy.Types import structType
structType.get = lambda x,y,z=None : x[y]

In this case, I was able to use this simple case, as I knew the keys it would be asked for would all exist, but you could make a slightly more complex one that checked to see the key exists, and if not, return z. You need all three arguments in the lambda though, since DictWriter passes them in.