Extending Django Wizards

I quite like the wizard pattern. Most people are somewhat familiar with the concept of having to complete a logically grouped set of form elements before moving onto the next one, with a final confirmation step, after which the operation is committed.

Howver, there are some perceieved problems within the Django formtools implementation, at least from the perspective of our use case.

The big one my team have identified: when you reload the page with a GET request, it clears out the data from the current run of the wizard. This is problematic when something happens to your internet, and you have to reload a page manually, for instance. Or if you just happen to reload it through another means.

Related to this, if you don’t have all of the information you require, there’s no way to “stash” your current iteration, and return to it later. This later could be after a logout and login (or even on a different computer).

Additionally, there’s no way to have “extra” steps inserted based on the selections made in a previous step. You could take the other approach, and define all of the steps initially (and then skip those that aren’t required), but in our case, we need a repeatable step (I call them a sub-step), where we perform an operation on a bunch of items that were selected in a previous step.

Let’s look at each of these in turn.


The default implementation automatically clears the storage whenever a GET request is handled:

class WizardView(TemplateView):

    def get(self, request, *args, **kwargs):
        self.storage.reset()
        self.storage.current_step = self.steps.first
        return self.render(self.get_form())

We can replace this with an implementation that will only reset when it detects a special GET parameter:

    def get(self, request, *args, **kwargs):
        if not request.GET.get('reset'):
            self.storage.reset()
            self.storage.current_step = self.steps.first
        
        step = self.storage.current_step or self.steps.first
        return self.render(self.get_form(
            step,
            data=self.storage.get_step_data(step),
            files=self.storage.get_step_files(step),
        ))

Not that we need to build up the form data to provide to the view - otherwise when you load up the wizard, it will render the first page as empty, but if you then use the navigation to select the first page it will correctly render it with data.

You’ll need to have an explicit link or button in your wizard template(s) to enable the user to restart the wizard if they need to.


Next up is persistent storage. For this we will need somewhere to store the data. The formtools implementation makes this easy to swap out - you can define the storage backend you want to use.

class MyWizard(WizardView):
    form_list = [...]
    storage_name = 'storage.wizard.DatabaseStorage'

That’s all you need to do to make your wizard use it - but now we need to build the storage class.

Let’s begin with a model to store the data in.

class WizardStorage(models.Model):
    user = models.ForeignKey('auth.User', related_name='wizard_storage', on_delete=models.CASCADE)
    prefix = models.TextField()
    data = models.JSONField()

We could use extra fields for the various parts, but that just complicates things.

Now let’s see a storage implementation:

class DatabaseStorage(BaseStorage):
    def __init__(self, prefix, request=None, file_storage=None):
        super().__init__(prefix, request, file_storage)
        self.init_data()
    
    def init_data(self):
        self.instance, _create = WizardStorage.objects.get_or_create(
            user=self.request.user,
            prefix=self.prefix,
            defaults={
                'data': {
                    self.step_key: None,
                    self.step_data_key: {},
                    self.step_files_key: {},
                    self.extra_data_key: {},
                }
            }
        )
    
    @property
    def data(self):
        return self.instance.data
    
    def update_response(self, response):
        if hasattr(self, 'instance'):
            self.instance.save()
      
    def reset(self):
        if hasattr(self, 'instance'):
            self.instance.delete()
            del self.instance

You can make it a bit more configurable so that the model could be swapped quite easily.

From here, we can use this in a wizard, and it will persist the step data to the database. I’ve still got a bit to do to ensure it can handle files, but this has not been a requirement of mine as yet.

There could be a bit of fun around having multiple stashes for a given wizard, and allowing the user to select which one they want to work on. As it stands, it just uses the user id and wizard prefix to determine where the data is stored.


The third improvement, allowing sub-steps, is a bit more complicated. To do that, you need to replace a bit more of the internal formtools code, rather than just subclassing/extending it. This involves a bunch of patching of the StepsHelper class from formtools - although you could replace this class by overriding WizardView.dispatch.

That’s beyond the scope of this post.

Testing WizardView in Django

Sometimes, you just have to resort to a multi-step view to simplify data entry for a user. Django had an “included” package for this: django.contrib.formtools, until some years ago, when it moved into it’s own repository. It’s still very useful though, although there is a bit of a learning curve.

I have some multi-step wizard views, for things where there are a bunch of different (sometimes dependent, but not necessarily) things we need to get from the user, and commit in an all-or-nothing way.

Writing tests for these views is quite cumbersome, but I’ve recently come up with a couple of things that can make this less painful. The reason it’s somewhat difficult is that the wizard, and the current step form both have form data, so to ensure there are no clashes between name fields, the forms are all prefixed. However, prefixes on forms mean that the data you need to pass to the form must also be prefixed in the correct way.

class TestWizard(TestCase):
    def test_wizard_view(self):
        # user has been created...

        self.client.force_login(user)

        # url is the url for this wizard view.

        response = self.client.get(url)
        self.assertEqual(200, response.status_code)

        response = self.client.post(url, {
            'wizard-current_step': 'first',
            'first-field_a': 'value',
            'first-field_b': 'value',
            'first-field_c': 'value',
        })
        self.assertEqual(200, response.status_code)
        # Because django by default returns a 200 on a form validation error,
        # we need another way to check this. If this is a re-rendering of the
        # same form it will be bound, otherwise it will not be.
        self.assertFalse(response.context['form'].is_bound)
        self.assertEqual(
            response.context['wizard']['management_form']['current_step'].value(),
            'second'
        )

        # This next one has a formset instead of a form!
        response = self.client.post(url, {
          'wizard-current_step': 'second',
          'second-0-field_a': 'a',
          'second-0-field_b': 'a',
          'second-1-field_a': 'a',
          'second-1-field_b': 'a',
          'second-INITIAL_FORMS': '0',
          'second-TOTAL_FORMS': '2'
        })
        self.assertEqual(200, response.status_code)
        self.assertFalse(response.context['form'].is_bound)
        # Okay, I'm bored with this already!

It would be much nicer if we could just do something like:

class TestWizard(TestCase):
    def test_wizard_view(self):
        # ...
        self.client.force_login(user)

        test_wizard(self, url, 'wizard', [
          ('first', {'field_a': 'value', 'field_b': 'value', ...}),
          ('second', [
            {'field_a': 'a', 'field_b': 'b'},
            {'field_a', 'a', 'field_b': 'a'},
          ])
        ])

The last part, the formset handling stuff is something I have touched on before, and I’m not sure if I’ll get to that bit now, but the first part is still useful.

def test_wizard(test, url, name, data):
    """
    Using the supplied TestCase, execute the wizard view installed at
    the "url", having the given name, with each item of data.
    """

    test.assertEqual(200, test.client.get(url).status_code)
    for step, step_data in data:
      step_data = {
        '{}-{}'.format(step, key): value
        for key, value in step_data.items()
      }
      step_data['{}-current_step'.format(name)] = step
      response = test.client.post(url, step_data, follow=True)
      test.assertEqual(200, response.status_code)
      if 'form' in response.context:
          test.assertFalse(response.context['form'].errors)

We might come back to the formset thing later.


But what about when we have a bunch of steps where we already have initial data, and we mostly just want to click next in each page of the wizard, possibly altering just a value here or there?

def just_click_next(test, url, override_data):
    response = test.client.get(url)
    while response.status_code != 302:
        form = response.context['form']
        test.assertFalse(form.errors)

        wizard = response.context['wizard']
        current_step = wizard['steps'].current

        # If we have a formset, then we need to do custom handling.
        if hasattr(form, 'forms'):
            current_step_data = step_data.get(current_step, [])
            data = {
                '{}-{}'.format(current_step, key): value
                for key, value in form.management_form.initial.items()
            }
            for i, _form in enumerate(form.forms):
                current_form_data = current_step_data[i] if len(current_step_data) > i else {}
                data.update({
                    '{}-{}'.format(_form.prefix, field): current_form_data.get(
                        field,
                        _form.initial.get(field)
                    )
                    for field in _form.fields
                    if field in current_form_data or field in _form.initial
                })
        else:
            current_step_data = step_data.get(current_step, {})
            data = {
                '{}-{}'.format(form.prefix, field): current_step_data.get(
                    field,
                    form.initial.get(field)
                )
                for field in form.fields
                if field in current_step_data or field in form.initial
            }

        # Add the wizard management form step data.
        data['{}-current_step'.format(wizard['management_form'].prefix)] = current_step

        response = test.client.post(url, data)

    return response

Those paying attention may notice that this one actually does handle formsets in a fairly nice way.

So how do we use it?

class TestViews(TestCase):
    def test_new_revision_wizard(self):
        # Setup required for wizard goes here.
        just_click_next(self, url, {
            'date': {'start_date': '2019-01-01'},
            'confirm': {'confirm': 'on'}
        })
        # Make assertions about the wizard having been run.

This replaces some 80 lines of a test: which actually becomes important when we need to test this with a bunch of subtly different preconditions.