Schinckel.net

Drone.io is a pretty neat (free for open-source projects) continuous integration server. The best feature from my perspective is that it works with BitBucket repositories.

It’s pretty nice having status badges indicating if a build is passing or failing, but even better is also getting a coverage report.

I’ve been using django-coverage for this, and ages ago manually created a set of drone.io style badges, and added a patch to copy the relevant file across. But then, shortly after, drone.io changed their status badge format. I never got around to redoing my badges, as it was pretty time consuming.

Enter Pillow.

from PIL import Image, ImageDraw, ImageFont

SIZE = (95, 18)

BACKGROUND = hex_colour('#4A4A4A')
SUCCESS = hex_colour('#94B944')
WARNING = hex_colour('#E4A83C')
ERROR = hex_colour('#B10610')

# You may need a different font filename if you aren't on a Mac
FONT = ImageFont.truetype(size=10, filename="/Library/Fonts/Arial.ttf")
FONT_SHADOW = hex_colour('#525252')

PADDING_TOP = 3

def build_image(percentage, colour):
    # Create a brand-new Image object, with the background
    # as the main badge colour.
    image = Image.new('RGB', SIZE, color=BACKGROUND)
    drawing = ImageDraw.Draw(image)
    
    # Write the word 'coverage' in our specified font.
    # Fake a text-shadow by drawing the text twice.
    # TODO: Make the text-shadow better.
    drawing.text((8, PADDING_TOP+1), 'coverage', font=FONT, fill=FONT_SHADOW)
    drawing.text((7, PADDING_TOP), 'coverage', font=FONT)
    
    # Do the percentage text.
    # TODO: Make the text-shadow better.
    # TODO: Make the text centred in the coloured box.
    drawing.rectangle([(55, 0), SIZE], colour, colour)
    drawing.text((63, PADDING_TOP+1), '%s%%' % percentage, font=FONT, fill=FONT_SHADOW)
    drawing.text((62, PADDING_TOP), '%s%%' % percentage, font=FONT)

Creating the required RGB tuple from a hex colour is also fairly easy:

def hex_colour(hex):
    if hex[0] == '#':
        hex = hex[1:]
    return int(hex[:2], 16), int(hex[2:4], 16), int(hex[4:6], 16)

Finally, you can just generate an image for every percentage point, and save them:

SUCCESS_CUTOFF = 85
WARNING_CUTOFF = 45

# range(101) -> [0, 1, 2, ..., 99, 100]
for i in range(101):
    file = open('%i.png' % i, 'wb')
    
    if i < WARNING_CUTOFF:
        build_image(i, ERROR).save(file)
    elif i < SUCCESS_CUTOFF:
        build_image(i, WARNING).save(file)
    else:
        build_image(i, SUCCESS).save(file)

It’s not quite perfect: that isn’t quite the font they use, but it will do for now.

Recently, I finally got around to re-installing OS X from scratch on my work machine. It was past time it needed to happen, to the extent where I would frequently be unable to wake machine from display sleep, and saving a file in a monitored directory would take the wsgi-monitor package tens of seconds to restart django.

One thing I wanted to do this time was only install stuff as necssary, but also put every pip installed command line tool in it’s own virtualenv. However, this has one drawback, in that it is a little repetitive.

For instance, to install Fabric, my deployment tool of choice:

$ virtualenv ~/.venv/fabric
$ . ~/.venv/fabric/bin/activate
(fabric)$ pip install fabric
(fabric)$ ln -s ~/.venv/bin/fabric /usr/local/bin/

This is fine if you only have one ’tool’ to install, but something like docutils actually installs a whole stack of command line tools.

What we want, is something like:

• create the virtualenv
• get a list of items already in the <virtualenv>/bin
• install the required tool (and any extra modules)
• link all of the newly added commands in <virtualenv>/bin to /usr/local/bin

We could just add each <virtualenv>/bin to our path, but that would mean that the first virtualenv created would be used for pip, which I don’t want installed at all.

Additionally, it would be nice to be able to specify a required version of the package to install, and other (non-dependency) packages that should be installed. For instance, I want mercurial_keyring to be installed in the mercurial virtualenv.

This last one is probably less important, as you can just use that virtualenv’s pip to install them after. But the version number stuff might be nice.

virtualenv has the nice ability to be able to create bootstrap scripts, which will do other stuff (like install specific packages). We can co-opt this to build a tool for doing the automatic installation and linking:

import virtualenv, subprocess

data = """
import os, subprocess

def extend_parser(optparse_parser):
    optparse_parser.add_option(
        "--upgrade",
        action="store_true",
        dest="upgrade",
        default=False,
        help="Upgrade package",
    )
    optparse_parser.add_option(
        "--path",
        dest="path",
        default='~/.venv/',
        help="Parent path of virtualenvs"
    )
    optparse_parser.add_option(
        '--package',
        dest="packages",
        action="append",
        help="Other packages to install"
    )
    
def adjust_options(options, args):
    global package
    if not args: 
        return
    package = args[0]
    if '==' in args[0]:
        args[0], version = args[0].split('==', 1)
    args[0] = os.path.join(os.path.expanduser(options.path), args[0])

def after_install(options, home_dir):
    global package
    venv = os.path.join(os.path.expanduser(options.path), home_dir)
    before = os.listdir(os.path.join(venv, 'bin'))
    command = [os.path.join(venv, 'bin', 'pip'), 'install', package]
    if options.upgrade:
        command += ['--upgrade']
    if options.packages:
        command += options.packages
    subprocess.call(command)
    after = os.listdir(os.path.join(venv, 'bin'))
    
    for command in set(after).difference(before):
        subprocess.call([
            'ln', '-s', 
            os.path.join(venv, 'bin', command),
            '/usr/local/bin'
        ])
"""

output = virtualenv.create_bootstrap_script(data)
open('/usr/local/bin/pip-install', 'w').write(output)
subprocess.call(['chmod', '+x', '/usr/local/bin/pip-install'])

There is one caveat: if an existing file is found in /usr/local/bin that matches one that should be linked, it will be ignored. That is, it does not overwrite existing commands. I think this is preferable, as it is marginally safer.

Linking commands like this is better than copying them, as it means you can just do a pip install --upgrade <package> in the relevant virtualenv, and it will upgrade commands. You can also use pip-install <package>==<new-version>, and that should work too. However, if you unlink a command (or remove one that would have linked but failed), and do a pip-install, it will not link the commands that were already installed in that virtualenv.

Anyway, your mileage may vary. I’m using it now, and it seems good.

Recently, I found myself having to interact with an API that uses SOAP. I’ve been using the SOAPpy package. Which has made it possible, but not exactly easy. But that’s not what I am going to write about right now.

In order to make the linking of data between my software and that system easier, I needed to get a dump of the other system’s data, in a CSV that I could send to the client.

So, since the SOAPpy module gives you something that looks dict-y, I thought I’d just be able to pass it to csv.DictWriter’s writerow() method.

Not quite.

See, whilst it supports the python dict-like [key] syntax, the SOAPpy.Types.structType doesn’t support the .get() method, that DictWriter uses to extract the data (indeed, it needs to, to be able to trap missing keys).

So, here is a simple duck-punch (does it quack like a duck? No? Punch it harder so, it quacks!) that enables you to pass a structType object to a DictWriter.writerow() method call:

from SOAPpy.Types import structType
structType.get = lambda x,y,z=None : x[y]

In this case, I was able to use this simple case, as I knew the keys it would be asked for would all exist, but you could make a slightly more complex one that checked to see the key exists, and if not, return z. You need all three arguments in the lambda though, since DictWriter passes them in.

Testing in Django is usually done using the unittest framework, which comes with Python. You can also test using doctest, with a little bit of work.

One advantage of doctest is that it’s super-easy to test for an exception: you just expect the traceback (which can be trimmed using \n ... \n).

In a unittest.TestCase, you can do a similar thing, but it’s a little more work.

Basically, you want to temporarily replace sys.stdout (or sys.stderr) with a StringIO instance, and set it back after the block you care about has finished.

Python has had a nice feature for some time called Context Managers. These enable you to ensure that cleanup code will be run, regardless of what happens in the block.

The syntax for running code within a context manager is:

with context_manager(thing) as other:
  # Code we want to run
  # Can use 'other' in here.

One place that you can see this syntax, in the context of testing using unittest is to check a specific exception is raised when a function that uses keyword arguments, or a statement that is not a callable is executed:

class FooTest(TestCase):
  def test_one_way(self):
    self.assertRaises(ExceptionType, callable, arg1, arg2)

  def test_another_way(self):
    with self.assertRaises(ExceptionType):
      callable(arg1, arg2)
      # Could also be:
      #     callable(arg1, arg2=arg2)
      # Or even:
      #     foo = bar + baz
      # Which are not possible in the test_one_way call.

So, we could come up with a similar way of calling our code that we want to capture the sys.stdout from:

class BarTest(TestCase):
  def test_and_capture(self):
    with capture(callable, *args, **kwargs) as output:
      self.assertEquals("Expected output", output)

And the context manager:

import sys
from cStringIO import StringIO
from contextlib import contextmanager

@contextmanager
def capture(command, *args, **kwargs):
  out, sys.stdout = sys.stdout, StringIO()
  try:
    command(*args, **kwargs)
    sys.stdout.seek(0)
    yield sys.stdout.read()
  finally:
    sys.stdout = out

It’s simple enough to do the same with sys.stderr.

Update: thanks to Justin Patrin for pointing out that we should wrap the command in a try:finally: block.

PyPI, formerly the CheeseShop is awesome. It’s a central repository of python packages. Knowing you can just do a pip install foo, and it looks on pypi for a package named foo is superb. Using pip requirements files, or setuptools install_requires means you can install all the packages you need, really simply.

And, the nice thing about pip is that it won’t bother downloading a package you already have installed, subject to version requirements, unless you specifically force it to. This is better than installing using pip install -e <scm>+https://... from a mercurial or git repository. This is a good reason to have published version numbers.

However, when installing into a new virtualenv, it still may take some time to download all of the packages, and not everything I do can be put onto pypi: quite a lot of my work is confidential and copyrighted by my employer. So, there is quite a lot of value to me to be able to have a local cache of packages.

You could use a shared (between all virtualenvs) --build directory, but the point of virtualenv is that every environment is isolated. So, a better option is a local cache server. And for publishing private packages, a server is required for this too. Being able to use the same workflow for publishing a private package as an open source package is essential.

Because we deploy using packages, our private package server is located outside of our office network. We need to be able to install packages from it on our production servers. However, this negates the other advantage of a pypi cache. It does mean we control all of the required infrastructure required to install: no more “We can’t deploy because github is down.”

So, the ideal situation is to actually have two levels of server: our private package server, and then a local cache server on each developer’s machine. You could also have a single cache server in the local network, or perhaps three levels. I’m not sure how much of a performance hit not having the cache on the local machine is.

To do this, you need two things. Your local cache needs to be able to use an upstream cache (no dicking around with /etc/hosts please), and your private server needs to be able to provide data to this.

The two tools I have been using handle neither of these. pypicache does not handle upstream caching, however this was easy to patch. My fork handles upstream caching, plus uses setuptools, enabling it to install it’s own dependencies.

localshop, however, will not work as an upstream cache, at least with pypicache, which uses some other APIs than those used by pip. However, it does have nice security features, and to move away from it would require me to extract the package data out. pypicache works to a certain extent with itself as an upstream cache, until you try to use it’s ‘requirements.txt caching’ feature. Which I tried to tonight.

Oh well.

I’ve spent a not insignificant amount of time working on a deployment function for within my fabfile.py (the configuration file used by Fabric). It’s well worth the investment, as being able to deploy with a single command (potentially to many servers) is not only faster, but much less prone to human error.

Currently, I’m using Mercurial as my source control. I’m also using it for deployment, but I’d like to get away from that.

My deployment process looks something like this:

1. Ensure the local repository has no uncommitted changes.
2. Ensure the requirements.txt file is exactly the same as the output from pip freeze.
3. Copy our public key to the remote server, for the user www-data, if it is not already installed there.
4. Create a virtualenv in the desired location on the server, if there is not one already there.
5. Ensure mercurial is installed on the server.
6. Push the local repository to the remote server. This will include any subrepositories. I do a little bit of fancy magic to ensure the remote subrepositories exist.
7. Update the remote server’s repository to the same revision as we are at locally. This means we don’t necessarily need to always deploy to tip.
8. Install the dependencies on the remote server.
9. Run some django management commands to ensure everything is setup correctly.
   • collect static files
   • sync the database
   • run migrations
   • ensure permissions are correct
   • compress static files
10. Restart the various services that need to be restarted.

This process is based around requirements files for a very good reason. pip is very good at recognising which packages are already installed, and not reinstalling them if the version requirements are met. I use pip freeze > requirements.txt to ensure that what will be deployed matches exactly with what I have been developing (and testing) against.

However, this process has some issues.

• Files must be committed to SCM before they can be deployed. This is fine for deployment to production, but is annoying for deploying to test servers. I have plenty of commits that turn on some extra debugging, and then a commit or two later, I turn it off.
• I have some packages that I have installed locally using pip install -e /path/to/package. To deploy these, I need to:
  1. Uninstall the editable installation.
  2. Package up a new version of the app.
  3. Push the package to my package repository (I use localshop).
  4. Install the package from the package repository.
  5. Run pip freeze > requirements.txt.
  6. Commit the changes.
  7. Deploy to the test server.
• Then, I usually need to develop further, so I reinstall using pip install -e ....

Today, I finally got around to spending some time looking at how pip can help improve this workflow.

With pip==1.3.1, we have a command that was not in pip==1.1, which was what I had been using until now. pip bundle.

My ‘deploy-to-development/test’ process now looks something like:

1. Get a list of packages installed as editable: pip list -e
2. Create a bundle, without dependencies, of these packages.
3. Get a list of all packages, other than those installed as editable: pip freeze | grep -v "^-e".
4. Ensure the server is set up (virtualenv, etc)
5. Push the local repository to the remote server.
6. Upload the bundle and requirements files.
7. Install from the requirements file on the server.
8. Force install from the bundle file on the server, without dependencies.
9. Repeat the post-installation stuff from above.

Some of this I’m never going to be able to avoid: ensuring we have the virtualenv, and the post-installation stuff. Migrations gotta migrate. However, I would like to move away from the pushing of the local repository.

My plan: turn my project into a package (complete with setup.py), so that it becomes just another entry in the requirements file. It will be editable, which means it will be bundled up for deployment.

However, it will mean I can get away from having the nested repositories that I currently have. Ultimately, I plan to be able to:

1. Build a bundle of editable packages.
2. Create a requirements file of non-editable packages.
3. Upload both of these files to the server.
4. Install the requirements.
5. Install the bundle.
6. Run the post installation tasks.

That would be bliss.

The Registry Pattern is something I use relatively frequently. In django, for instance, we see it used for the admin interface, and I used very derivative code for my first API generation tool: django-rest-api. For our integration with external POS and other systems, we need to register importers, so that the automated stats fetching is able to look for units that need to fetch data from an external system’s website, or parse incoming email headers for matching delivered data.

I had been using something similar to:

from base import BaseStatsImporter, register

class FooStatsImporter(BaseStatsImporter):
    # ...

register(FooStatsImporter)

This is all well and good, but it is annoying. I need to remember to register each class after I declare it.

Then I discovered the magic of __metaclass_, used with __new__:

class RegistryMetaClass(type):
    def __new__(cls, clsname, bases, attrs):
        new_class = super(cls, RegistryMetaClass).__new__(cls, clsname, bases, attrs)
        register(new_class)
        return new_class
        
class BaseStatsImporter(object):
    __metaclass__ = RegistryMetaClass
    
    # ...

As long as your subclasses don’t override __metaclass__, then every new subclass will be added to the registry.

Obviously, this is magic, and in some cases the explicit way would be better.

A recurring feature of #django is that someone asks about settings.py, and using a local_settings.py file. The standard advice is to have the following in your settings.py:

# More settings are above here.

try:
  from local_settings import *
except ImportError:
  pass

This is usually the last (or one of the last) things in the file. This can be used to override settings with sane values for the local environment.

However, this means that local_settings.py must be not in your source control system, or must not be deployed to other servers.

I like keeping everything in my source control system of choice (mercurial), and currently use an hg-based deployment. In my fabfile.py, instead of archiving up the current structure, I use hg to push the main repo, and any sub-repos, and update them to the version that is displayed locally.

This means I want to be able to control the content of production’s local_settings.py equivalent.

The other issue, and this was the one that came up today and gave me the idea of this post, is that someone wanted to add an app to settings.INSTALLED_APPS but only locally. I too have done this (still do, with django-devserver, amongst others).

I came up with the following solution. Instead of having a settings.py and local_settings.py, I have a settings module:

settings/
    __init__.py
    base.py
    development.py
    production.py
    testing.py

base.py contains what was normally in your main settings.py file. That is, settings that are common to all environments.

In development.py, production.py and testing.py, I have the following line at the top:

from base import *

Then, in each of those files, where I need to override or alter a setting, including appending to a list or tuple, I can just modify away. Some things that I do in development.py, for instance:

from base import *

DEBUG = True

DATABASES['default']['HOST'] = '127.0.0.1'

INSTALLED_APPS += (
  'devserver',
  'test_extensions',
  'test_utils' # really only for makfixture.
)

import getpass
EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
EMAIL_FILE_PATH =  PROJECT_ROOT / 'log' / 'email-messages-%s' % getpass.getuser()

This shows how you can set a value, alter a value of a dict given a specific key, and append to a tuple. I also have a nice little setup where I use a value set in the base.py file (PROJECT_ROOT) to determine where I want to log email messages to.

Finally, you need some way to say which of these files should be used. This is all done in __init__.py:

servers = {
  'development': [
    'darwin', 'boyd', 'arne'
  ],
  'testing': [
    'testing', 'debian'
  ],
  'production': [
    'staging', 'vps1', 'vps2', 'vps3'
  ]
}

def get_server_type():
  from socket import gethostname
  server_name = gethostname()
  for server_type, names in servers.items():
    if server_name in names:
      return server_type
  
  return 'production' # Or whatever you want the default to be.
                      # I usually have 'testing' here, because I tend to
                      # spin up test servers. If you spun up production
                      # servers lots, you'd use that.

exec("from %s import *" % get_server_type())

This method does require a little bit of maintainence: when you have a new server name, you need to add an entry to this file. If you are often creating testing servers (like I am) then you might want to use testing as the default server type.

Alternatively, you could use some sort of prefix to mean a particular server type.

Anyway, that’s how I do it. The only drawback is that it does mean that your SECRET_KEY, and any passwords you might have defined in settings.py are stored in your repository. We aren’t that fussed about that right now: our project is closed source, and only trusted people have access to the repository.

I mentioned a week or so ago about my django templatetag that will only display a menuitem or link if the logged in user has access to the view that it points to. In passing, I stated how it was rather complicated to do this test.

The complicated bit is finding out all of the decorators that are wrapping a given function. Fortunately, python is a dynamic language, and this type of introspection, whilst not completely simple, is possible.

The key lies in the property that all function objects in python have: func_closure. According to the python docs:

func_closure is None or a tuple of cells that contain binding for the function’s free variables

Depending upon where you read this, it may or may not be writable. Luckily, we don’t need to be able to write to this, only read from it.

If func_closure is not None, then it will be a tuple of <cell> objects. To do anything useful, we’ll need to look at their cell_contents attribute. If that is callable, then it’s a good candidate for a decorator.

Because of the way decorators work, if you have multiple decorators on a function, each one wraps the next one. Thus, we’ll need to have some recursion in there.

At a first draft, we might end up with something like:

def get_decorators(function):
  # If we have no func_closure, it means we are not wrapping any other functions.
  if not function.func_closure:
    return [function]
  decorators = []
  # Otherwise, we want to collect all of the recursive results for every closure we have.
  for closure in function.func_closure:
    decorators.extend(get_decorators(closure.cell_contents))
  return [function] + decorators

It’s important that we return the original function in the base case of the recursive function, as eventually, every closure will fold down to a single callable. It has the side effect that get_decorators will get all of the decorators, and the function they ultimately wrap.

You could probably also do this as a generator.

For basic django function views, this will work fine. For class-based views, we need to do something a little extra.

In the case I was writing this function for, I knew it would only be looking for a get() method on the class-based view, which makes things a little simpler. That, and the dispatch() method were the only places I would need to look on the class for decorators. Also, I only wanted decorators that were callable: since I would actually call a subset of them to test if the user could access the view.

def get_callable_cells(function):
  callables = []
  # Under some circumstances, I wound up with an object that has the name `view_func`: 
  # this is the view function I need to access.
  if not hasattr(function, 'func_closure'):
    if hasattr(function, 'view_func'):
      return get_callable_cells(function.view_func)
  # If we have no more function we are wrapping
  if not function.func_closure:
    return [function]
  for closure in function.func_closure:
    contents = closure.cell_contents
    # Class-based views have a dispatch method
    if hasattr(contents, 'dispatch'):
      callables.extend(get_callable_cells(contents.dispatch.__func__))
      if hasattr(contents, 'get'):
        callables.extend(get_callable_cells(contents.get.__func__))
    callables.extend(get_callable_cells(contents))
  return [function] + callables

The other trick I’m using there is to use the __func__ property of the dispatch and get methods of the view class.

From there, in my case, I wanted those callable cells that look like they take a user as their first argument:

def get_tests(function):
  return [
    x for x in get_callable_cells(function)
    if x.func_code.co_varnames[0] in ["user", "u"]
  ]

So, this works for things that look like:

class ViewClass(ParentClass):
  @login_required
  @permission_required('app_label.permission_name')
  def dispatch(self, *args, **kwargs):
    return super(ViewClass, self).dispatch(*args, **kwargs)
  
  @user_passes_test(lambda u: u.is_staff)
  def get(self, *args, **kwargs):
    # do stuff here
    pass

As well as:

@login_required
@permission_required('app_label.permission_name')
@user_passes_test(lambda u: u.is_staff)
@render('foo/bar.html')
def view_function(request, *args, **kwargs):
  # do stuff here

It will pick up that the @render decorator is not asking for a user.

It works with the @user_passes_test decorator, because that is passed a function that has the first argument of u. It works with the @permission_required decorator for basically the same reason, although that has a function inside a function that actually has the argument of user.

Finally, it works for the @login_required decorator, as that calls user_passes_test.

For future reference:

1
2
3
4
5
6

def decode_dreamweaver_password(encoded):
    output = ""
    for i in range(0, len(encoded), 2):
        val = int(data[i:i+1],16) - i/2
        output += chr(val)
    return output