Leveraging HTML and Django Forms: Pagination of Filtered Results

Django’s forms are fantastic for parsing user input, but I’ve come up with a nice way to use them, in conjunction with HTML forms, for pagination, using the inbuilt Django pagination features.

It all stems from the fact that I’ve begun using forms quite heavily for GET purposes, rather than just for POST. Basically, anytime you have a URL that may have some parts of the query string that may need to be built, it’s simpler to use a form element, than to manually build up the url in your template.

Thus, where you may have something like:

<a href="{% url 'foo' %}?page={{ page }}">

It may be better do do something more like:

<form action="{% url 'foo' %}">
  <input type="hidden" name="page" value="{{ page }}">
</form>

Indeed, you can even use named buttons for submission, which will refer to the page. That is the key to the process outlined below.


Django comes with lots of “batteries”, including form handling and pagination. The Class Based Views (CBV) that deal with collections of objects will include pagination, although it is possible to use this pagination in your own views. For simplicity, we’ll stick with a simple ListView.

Let’s begin with that simple view: in our views.py:

from django.views.generic import ListView, DetailView

from .models import Person

person_list = ListView.as_view(
    queryset=Person.objects.all(),
    template_name='person/list.html',
    paginate_by=10,
)

person_detail = DetailView.as_view(
    queryset=Person.objects.all(),
    template_name='person/detail.html',
)

Essentially, that’s all you need to do. You could use implied template names, but I almost never do this. The takeaway from this block is that we are stating the queryset that our ListView will use as the base, the template it should render, and the number of items per page.

I’ve stubbed out the person_detail view, just so we can refer to it in our urlconf, and then in turn in the template. Because of the simplicity of it, we could have just done all of this in our urls.py.

Speaking of our urls.py, we have something like:

from django.conf.urls import url

import views

urlpatterns = [
    url(r'^people/$', views.person_list, name='person_list'),
    url(r'^people/(?P<pk>\d+)/', views.person_detail, name='person_detail'),
]

Then, in our template, we can render it as (ignoring the majority of the page):

<ul class="people">
  {% for object in object_list %}
    <li>
      <a href="{% url 'person_detail' pk=object.pk %}">
        {{ object }}
      </a>
    </li>
  {% endfor %}
</ul>

But this doesn’t give us our pagination. It will only show the first ten results, with no way to access the others. All we need to do to access the others is to append ?page=X, but, as we will see, there is another way.

Typically, your pagination block might look something like:

<ul class="pagination">
  <li>
    {% if page_obj.has_previous %}
      <a href="?page={{ page_obj.previous_page_number }}">
        prev
      </a>
    {% else %}
      <span>prev</span>
    {% endif %}
  </li>

  {% for page_number in paginator.page_range %}
    {% if page_number = page_obj.number %}
      <li class="active">
        <span>{{ page_number }}</span>
      </li>
    {% else %}
      <li>
        <a href="?page={{ page_number }}">
          {{ page_number }}
        </a>
      </li>
    {% endif %}
  {% endfor %}


  <li>
    {% if page_obj.has_next %}
      <a href="?page={{ page_obj.next_page_number }}">
        next
      </a>
    {% else %}
      <span>next</span>
    {% endif %}
  </li>
</ul>

Depending upon your CSS framework, if you use one, there may already be some pre-prepared styles to help you out with this.

This is all well and good, until you want paginated search results. Then, you can no longer rely on being able to rely on using ?page=N, as this would remove any search terms you were already using. Also, if you were using ajax to fetch and display stuff, you may need to use the whole URL, rather than just the query string.

Instead, we can use a Django form for searching, and just add in the pagination bits.

We will build a page that displays an optionally filtered list of people.

Our Person model will be deliberately simple:

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=256)

Likewise, our form will be simple. All we need to do is have the form able to filter our queryset.

from django import forms

class PersonSearchForm(forms.Form):
    query = forms.CharField(label=_('Filter'), required=False)

    def filter_queryset(self, request, queryset):
        if self.cleaned_data['name']:
            return queryset.filter(name__icontains=self.cleaned_data['query'])
        return queryset

Finally, we will need to subclass a ListView. We’ll mixin from FormMixin, so we get the form-handling capabilities:

from django.views.generic.edit import FormMixin
from django.views.generic import ListView

class FilteredListView(FormMixin, ListView):
    def get_form_kwargs(self):
        return {
          'initial': self.get_initial(),
          'prefix': self.get_prefix(),
          'data': self.request.GET or None
        }

    def get(self, request, *args, **kwargs):
        self.object_list = self.get_queryset()

        form = self.get_form(self.get_form_class())

        if form.is_valid():
            self.object_list = form.filter_queryset(request, self.object_list)

        context = self.get_context_data(form=form, object_list=self.object_list)
        return self.render_to_response(context)

There’s a little bit to comment on there: we override the get_form_kwargs so we pull our form’s data from request.GET, instead of the default.

We also override get, so we filter results if the form validates (which it will if there was data provided). We delegate responsibility for the actual filtering to the form class.

Everything else is just standard.

We will want to actually use this view:

people_list = FilteredListView.as_view(
    form_class=PersonSearchForm,
    template_name='person/list.html',
    queryset=Person.objects.all(),
    paginate_by=10
)

Now we need to render this.

<form id="person-list-filter" action="{% url 'person_list' %}">
  <input name="{{ form.query.html_name }}" value="{{ form.query.value }}" type="search">
  <button type="submit" name="page" value="1">{% trans 'Search' %}</button>
</form>

<div class="results">
  {% include 'person/list-results.html' %}
</div>

You may notice that the search button will result in page=1 being used. This is deliberate.

Our person/list-results.html is just the same as what our person/list.html looked like before, with the addition of the pagination template inclusion.

{% include 'pagination.html' with form_target='person-list-filter' %}

<ul class="people">
  {% for object in object_list %}
    <li>
      <a href="{% url 'person_detail' pk=object.pk %}">
        {{ object }}
      </a>
    </li>
  {% endfor %}
</ul>

Our pagination.html is very similar to how our other template above looked too, but using <button> elements instead of <a>, and we will disable those that should not be clickable. Also, the buttons contain an attribute indicating which form they should be bound to.

<ul class="pagination">
  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_previous %}
        name="page"
        value="{{ page_obj.previous_page_number }}"
        type="submit"
      {% else %}
        disabled="disabled"
      {% endif %}>
      prev
    </button>
  </li>

  {% for page_number in paginator.page_range %}
    <li class="{% if page_number = page_obj.number %}active{% endif %}">
      <button
        name="page"
        value="{{ page_number }}"
        type="submit"
        form="{{ form_target }}"
        {% if page_number = page_obj.number %}
          disabled="disabled"
        {% endif %}>
        {{ page_number }}
      </button>
    </li>
  {% endfor %}

  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_next %}
        name="page"
        value="{{ page_obj.next_page_number }}"
        type="submit"
      {% else %}
        disabled="disabled"
      {% endif %}>
      next
    </button>
  </li>
</ul>

We are getting close now. This will be enough to have clicking on the next/previous or page number buttons resubmitting our search form, resulting in the page reloading with the correct results.

But we can do a bit better. We can easily load the results using AJAX, and just insert them into the page.

We just need one additional method on our View class:

class FilteredListView(FormMixin, ListView):
    # ...

    def get_template_names(self):
        if self.request.is_ajax():
            return [self.ajax_template_name]
        return [self.template_name]

    # ...

and one addition to our view declaration:

people_list = FilteredListView.as_view(
    form_class=PersonSearchForm,
    template_name='person/list.html',
    ajax_template_name='person/list-results.html',
    queryset=Person.objects.all(),
    paginate_by=10,
)

I’ll use jQuery, is it makes for easier to follow code:

// Submit handler for our form: submit it using AJAX instead.
$('#person-list-filter').on('submit', function(evt) {
  evt.preventDefault();

  var form = evt.target;

  $.ajax({
    url: form.action,
    data: $(form).serialize(),
    success: function(data) {
      $('#results').html(data)
    }
  });
});

// Because we are using buttons, which ajax submit will not send,
// we need to add a hidden field with the relevant page number
// when we send our request.
$('#person-list-filter').on('click', '[name=page]', function(evt) {
  var $button = $(evt.target).closest('button');
  var $form = $button[0].form;

  if (!$form.find('[type=hidden][name=page]')) {
    $form.append('<input type="hidden" name="page">');
  }

  $form.find('[type=hidden][name=page]').val($button.val());

  $form.submit();
});

That should do nicely.


There is another thing that we need to think about. If we leave the next/prev buttons, then we need to handle multiple clicks on those buttons, which fetch the subsequent page, and possibly cancel the existing AJAX request.

I do have a solution for this, too, although it complicates things a fair bit. First, we need to add some attributes to the next/prev buttons:

<ul class="pagination">
  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_previous %}
        name="page"
        value="{{ page_obj.previous_page_number }}"
        type="submit"
        data-increment="-1"
        data-stop-at="1"
      {% else %}
        disabled="disabled"
      {% endif %}>
      prev
    </button>
  </li>

  {% for page_number in paginator.page_range %}
    <li class="{% if page_number = page_obj.number %}active{% endif %}">
      <button
        name="page"
        value="{{ page_number }}"
        type="submit"
        form="{{ form_target }}"
        {% if page_number = page_obj.number %}
          disabled="disabled"
        {% endif %}>
        {{ page_number }}
      </button>
    </li>
  {% endfor %}

  <li>
    <button
      form="{{ form_target }}"
      {% if page_obj.has_next %}
        name="page"
        value="{{ page_obj.next_page_number }}"
        type="submit"
        data-increment="1"
        data-stop-at="{{ paginator.num_pages }}"
      {% else %}
        disabled="disabled"
      {% endif %}>
      next
    </button>
  </li>
</ul>

And our click handler changes a bit too:

$('#person-list-filter').on('click', 'button[name=page]', function() {
  var page = parseInt(this.value, 10);
  var $form = $(this.form);
  // Only update the value of the hidden form.
  if (!$form.find('[name=page][type=hidden]')) {
    $form.insert('<input name=page type=hidden>');
  }
  $form.find('[name=page][type=hidden]').val(page);
  // Increment any prev/next buttons values by their increment amount,
  // and set the disabled flag on any that have reached their stop-at
  $form.find('[data-increment]').each(function() {
    this.value = parseInt(this.dataset.increment, 10) + page;
    // We want to disable the button if we get to the 'stop-at' value,
    // but this needs to happen after any submit events have occurred.
    if (this.dataset.stopAt) {
      setTimeout(function() {
        this.disabled = (this.value == this.dataset.stopAt);
      }.bind(this), 0);
    }
  });

  $form.submit();
});

Since this was posted, I have written a number of pages that use this pattern. Some of the improvements that could be made are listed below:

It’s possible to have these results automatically update as the user types. Obviously, this only makes sense if we have AJAX submission happening!

$('#person-list-filter').on('keyup', function() {
  this.submit();
})

If you have lots and lots of results, you probably won’t want to show every button. Often you will see the first few, and a couple either side of the current page (and sometimes the last few). This is almost possible to do with pure CSS, but not quite. I do have a solution for this, but it’s probably worthy of a complete post of its own.

Another situation that is likely to happen is this:

  • User clicks on a page other than page 1 of results. Let’s say page N.
  • User enters text in search field which results in fewer than N pages of results being available.
  • User gets error message.

We can fix this with an overridden method:

class FilteredListView(FormMixin, ListView):
    # ...

    def paginate_queryset(self, queryset, page_size):
        try:
            return super(FilteredListView, self).paginate_queryset(queryset, page_size)
        except Http404:
            self.kwargs['page'] = 'last'
            return super(FilteredListView, self).paginate_queryset(queryset, page_size)

    # ...

You’ll also need to add in a get_prefix() method if you are using an old Django, but really you should just upgrade.


Updated: I’ve added in some more error checking into the templates, to prevent exceptions when attempting to render previous and next page links (thanks inoks).

Updated: I’ve changed to use the preferred urlpattern syntax. (thanks knbk).

Updated: Delegate to the form for filtering. Add discussion of other extensions. Add button[form] attributes.

Custom Element Form Submission

Custom elements are starting to get some traction in Web Development. There have been some really nice recent posts, the one which got me back on track was Web Components: Why You’re Already an Expert, but also Custom Elements: defining new elements in HTML and Performance and Custom Elements. Also, Polymer makes heavy use of Custom Elements, and I’ve had a bit of a look there too.

I really like KnockoutJS, so started playing around with that. I have a nicely defined DatePicker element, that would work well being turned into a Custom Element, and allow me to use the Shadow DOM to hide the internals of the code.

However, I soon hit a problem. Browsers, and jQuery, will only add data to a form when the element type is <input>, <select>, <textarea> and <keygen>. In jQuery, for instance, this is hard-coded in as rsubmittable, see source.

This means that you cannot just use a <x-date-picker> element, and have it submit. You still need to use some type of a hidden <input> element, and link that to the value. “Subclassing” <input> is not sufficient.

This seems to be a fairly large oversight, and I have not been able to find anything else on the internet that discusses this.

I haven’t tried “subclassing” <button> yet, to see if it is possible to create custom elements that can be used to submit forms, rather than provide data for them.

Django Fieldsets

HTML forms contain a construct called a fieldset. These are generally used to segment a form: splitting a form into groups of fields that are logically grouped. Each fieldset may also have a legend.

Django’s forms have no concept of a fieldset natively, but with a bit of patching, we can make every django form capable of rendering itself using fieldsets, yet still be backwards compatible with non-fieldset-aware templates.

Ideally, we would like to be able to render a form in a way similar to:

<form>
  {% for fieldset in form.fieldsets %}
  <fieldset>
    <legend>{{ fieldset.title }}</legend>
    
    <ul>
      {% for field in fieldset %}
        <li>
          {{ field.label_tag }}
          {{ field }}
          {{ field.help_text }}
          {{ field.errors }}
        </li>
      {% endfor %}
    </ul>
  </fieldset>
  {% endfor %}
  
  <!-- submit button -->
</form>

And, it would make sense to be able to declare a form’s fieldsets in a manner such as:

class MyForm(forms.Form):
  field1 = forms.BooleanField(required=False)
  field2 = forms.CharField()
  
  class Meta:
    fieldsets = (
      ('Fieldset title', {
        'fields': ('field1', 'field2')
      }),
    )

This is similar to how fieldsets are declared in the django admin.

We can’t just simply create a subclass of forms.Form, and do everything there, as the metaclass stuff doesn’t work correctly. Instead, we need to duck-punch.

First, we want to redefine the metaclass __init__ method, so it will accept the fieldsets attribute.

from django import forms
from django.forms.models import ModelFormOptions

_old_init = ModelFormOptions.__init__

def _new_init(self, options=None):
  _old_init(self, options)
  self.fieldsets = getattr(options, 'fieldsets', None)

ModelFormOptions.__init__ = _new_init

Next, we will need a Fieldset class:

class Fieldset(object):
  def __init__(self, form, title, fields, classes):
    self.form = form
    self.title = title
    self.fields = fields
    self.classes = classes
  
  def __iter__(self):
    # Similar to how a form can iterate through it's fields...
    for field in self.fields:
      yield field

And finally, we need to give every form a fieldsets method, which will yield each fieldset, as a Fieldset defined above:

def fieldsets(self):
  meta = getattr(self, '_meta', None)
  if not meta:
    meta = getattr(self, 'Meta', None)
  
  if not meta or not meta.fieldsets:
    return
  
  for name, data in meta.fieldsets:
    yield Fieldset(
      form=self,
      title=name,
      fields=(self[f] for f in data.get('fields',(,))),
      classes=data.get('classes', '')
    )

forms.BaseForm.fieldsets = fieldsets

I am using this code (or something very similar to it), in projects. It works for me, but your mileage may vary…

Django AJAX Forms

I think the more Django code I write, the more I like one particular feature.

Forms.

Simple as that. Forms are the reason I keep coming back to django, and discard other web frameworks in other languages, even though I really want to try them.

One pattern I have been using a fair bit, which was touched on in another post, is using AJAX to handle form submission, and displaying the response.

Before we continue, a quick recap on what Django’s forms offer us.

  • A declarative approach to defining the fields a form has, including validation functions.
  • Will render themselves to HTML input elements, as appropriate.
  • Handle validation of incoming form-encoded (or otherwise provided) data.
  • Fields can validate themselves, and can include validation error messages as part of the HTML output.
  • (Model forms) handle instantiation of and updating of model instances.

A normal form-submission cycle contains a POST or GET request to the server, which responds with a fresh HTML page, which the browser renders. The normal pattern for successful POST requests is to redirect to a GET afterwards, to prevent duplicate submission of forms.

By doing an ajax request instead of a full-page request means we can:

  • reduce the amount of data that is sent back from the server
  • improve apparent performance by only re-rendering the relevant data
  • reduce the amount of time spent rendering parts of the page that have not changed, such as menu, etc.

The way I have been doing it, in broad terms, is to have a template just for the form. If the request is an ajax request, then this will be rendered and returned. If it’s not an ajax request, then the full page will be returned.

Some example code, for one way to do this:

def view(request, pk):
  instance = MyModel.objects.get(pk=pk)
  
  if request.is_ajax():
    template = 'form.html'
  else:
    template = 'page.html'
  
  
  if request.method == 'POST':
    form = MyForm(request.POST, instance=instance)
    if form.is_valid():
      form.save()
      if not request.is_ajax():
        return redirect('redirect-name-here')
  else:
    form = MyForm(instance=instance)
  
  return render(request, template, {'form': form})

Our template files. page.html:

{% extends 'base.html' %}

{% block main %}
  {% include 'form.html' %}
{% endblock %}

{% block script %}
{# Assumes jQuery is loaded... #}
{# This should be in a seperate script file #}
<script>
$(document).on('submit', 'form.dynamic-form', function(form) {
  var $form = $(form);
  $.ajax({
    type: form.method,
    url: form.action,
    data: $form.serialize(),
    success: function(data) {
      $form.replace(data);
    }
  });
});
</script>
{% endblock %}

And form.html:

<form action="/path/to/url/" method="POST" class="dynamic-form">
  {% csrf_token %}
  {{ form }}
  <button type="input">Submit</button>
</form>

Obviously, this is a fairly cut-down example, but it gets the message across.

One thing I dislike in general about django is that failed form submissions are returned with a status code of 200: personally I think a 409 is more appropriate in most cases, but returning a 200 actually means this code is simpler.

jQuery dynamic forms

Some javascript, currently using jQuery, that will convert a form into an ajax form.

First up, a shim to enable FormData for non-compliant browsers. Based on formdata.js, but with some changes. Man, callback based code is a bitch when all you need to do is get some value, and have to work around that.

(function(w, $) {
  // Don't override if it is native.
  if (w.FormData)
    return;
      
  function FormData(form) {
    var fd = this;
    this.fake = true;
    this.boundary = "--------FormData" + Math.random();
    this._fields = [];
    this.contentType = 'multipart/form-data; boundary=' + this.boundary;
          
    if (form) {
      var $form = $(form);
      $.each($form.serializeArray(), function(i, obj){
        fd.append(obj.name, obj.value);
      });
      $form.find('[type=file]').each(function(i, file) {
        fd.append(file.name, file.files[0]);
      });
    }
  }
    
  // A listener to automatically add the binary version of the data. This may suck.
  $('[type=file]').change(function updateData(change) {
    var reader = new FileReader();
    reader.onload = function(load) {
      $(change.target.files[0]).data('binary-file-data', load.target.result);
    }
    reader.readAsBinaryString(change.target.files[0]);
  });
    
  // The interface FormData provides...
  FormData.prototype.append = function(key, value) {
    this._fields.push([key, value]);
  }
    
  // But, we will actually look more like a string to XMLHttpRequest.
  FormData.prototype.toString = function() {
    var boundary = this.boundary;
    var body = '';
    $.each(this._fields, function(i, field) {
      body += '--' + boundary + '\r\n';
      if (field[1].name) {
        var file = field[1];
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\"; filename=\""+ file.name +"\"\r\n";
        body += "Content-Type: "+ file.type +"\r\n\r\n";
        body += $(file).data('binary-file-data') + "\r\n";
      } else {
        body += "Content-Disposition: form-data; name=\""+ field[0] +"\";\r\n\r\n";
        body += field[1] + "\r\n";
      }
    });
          
    body += "--" + boundary +"--";
    return body;
  }
  w.FormData = FormData;
})(window, jQuery);

Now, for the jQuery code. This will override the submit event on all forms. If there are any <input type=file> elements, then it will use a FormData object, else it will just .serialize() the object. The response will override the .html() content of the form, but not the form itself.

$('form').submit(function(evt) {
  evt.preventDefault();
  var form = evt.target, $form = $(form), data;
  var options = {
    cache: false,
    type: form.method,
    url: form.action,
    done: function(data) {
      $form.html(data);
    },
    fail: function(xhr, status, error) {
      console.log(status, error, xhr);
    }
  };
    
  if ($form.find('[type=file]').length) {
    data = new FormData(form);
    // Native FormData objects set this automatically (how???), but we need to manually do it.
    options.contentType = data.contentType || false;
    options.processData = false;
  } else {
    data = $form.serialize();
  }
  options.data = data;
  $.ajax(options);
});

Pre-validating Many to Many fields.

Django’s form validation is great. You can rely on it to parse data that you got from the user, and ensure that the rules you have implemented are all applied. Model validation is similar, and I tend to use that in preference, as I often make changes from outside of the request-response cycle. Indeed, I’ve started to rewrite my API framework around using forms for serialisation as well as parsing.

One aspect of validation that is a little hard to grok is changes to many-to-many fields. For instance, the part of the system I am working on right now has Tags that are applied to Units, but a change to business requirements is that these tags need to be grouped, and a unit may only have one tag from a given TagGroup.

Preventing units from being saved with an invalid combination of Tags is simple if you use the django.db.models.signals.m2m_changed signal.

from django.db.models.signals import m2m_changed
from django.dispatch import receiver

@receiver(m2m_changed, sender=Tag.units.through)
def prevent_duplicate_tags_from_group(sender, instance, action, reverse, model, pk_set, **kwargs):
  if action != 'pre_add':
    return
  
  if reverse:
    # At this point, we know we are adding Tags to a Unit.
    tags = Tag.objects.filter(pk__in=pk_set).select_related('group')
    existing_groups = TagGroup.objects.filter(tags__units=instance).distinct()
    invalid_tags = set()
    for tag in tags:
      if tag.group in existing_groups:
        invalid_tags.add(tag)
      group_count = 0
      for other_tag in tags:
        if other_tag.group == tag.group:
          group_count += 1
      if group_count > 1:
        invalid_tags.add(tag)
    if invalid_tags:
      raise ValidationError(_(u'A unit may only have one Tag from a given Tag Group'))
  else:
    # At this point, we know we are adding Units to a Tag.
    units = Unit.objects.filter(pk__in=pk_set)
    group = instance.group
    invalid_units = []
    for unit in units:
      if unit.tags.exclude(pk=instance.pk).filter(group=group).exists():
        invalid_units.append(unit.name)
    if invalid_units:
      raise ValidationError(_(u'The unit%s "%s" already ha%s a Tag from group "%s"' % (
        "s" if len(invalid_units) > 1 else "",
        ", ".join(invalid_units),
        "ve" if len(invalid_units) > 1 else "s",
        group.name
      )))

Now, this on it’s own is nice enough. However, if you try to save invalid data from within the admin interface, then you will get an ugly trackback. If only there was a way to get this validation code to run during the validation phase of a form. i.e., when you are cleaning it…

So, we can create a form:

from django import forms
from models import Tag, prevent_duplicate_tags_from_group
class TagForm(forms.ModelForm):
  class Meta:
    model = Tag
    
  def clean_units(self):
    units = self.cleaned_data.get('units', [])
    if units:
      prevent_duplicate_tags_from_group(
        sender=self.instance.units,
        instance=self.instance,
        action="pre_add",
        reverse=False,
        model=self.instance.units.model,
        pk_set=units
      )
    return self.cleaned_data

You can create a complementary form on the other end (or, if you already have one, then just hook this into the field validator). The bonus here is that the validation errors will be put on the field with errors, in this case units.