Schinckel.net

Over the past years, I’ve worked really hard to get code coverage of my work project up to a reasonable level. I’m not attempting to get 100% coverage (although, I have made it there in some smaller projects), nor do I mistakenly believe that 100% coverage is a panacea. I’m aware that even 100% coverage doesn’t mean there are no bugs.

But there are a lot of reasons to have high test coverage. Having less than 100% coverage mean that the code not being covered is not automatically tested.

In a django project, there are several different types of code. Python code, HTML, CSS and Javascript. In my case, there’s also a lot of SQL. Some of it is doing things like defining views and functions - but I’m not going to deal with that now…

What I am going to look at is Javascript.

It would be awesome to write unit tests for all of our JS code, but we’ve also been writing Robot Framework tests for a lot of our functionality: what would be great is to capture the coverage that is already happening due to this. It would be even better if it was possible to get this coverage information for JS that is embedded in HTML templates, but that seems like it might be a harder job.

There are really only two things we need to do in order to capture the coverage data:

• Modify the scripts so they generate coverage data when they are executed
• Capture the coverage data before any navigation away from any page

The first one could be done in a number of different ways. You might be able to use a staticfiles backend to modify the files when running collectstatic, or perhaps a middleware that modified the files as they are served. That could possibly modify the inline script tags as well, although ideally you’d want to be doing that in the template rendering phase, as I think that could be the last point you get access to the template file names and locations.

There is a tool that you can use for the actual modification of the script files: nyc. This used to be called Istanbul, I believe, and it can “instrument” files. Then, when the script is executed by the browser, it stores the coverage inforamtion in window.__coverage__.

This then allows for capture of the coverage information, but how do we ensure this information is not lost when the page is navigated away from?

To do that in Django requires a small amount of JS, and a django view to capture it:

window.addEventListener('beforeunload', () => {
  if (window.__coverage__) {
    navigator.sendBeacon("{ % url 'collect-coverage' % }", JSON.stringify(window.__coverage__));
  }
})

Okay, that was easier than expected…what about the matching view?

import json
import uuid
from pathlib import Path

from django.http import HttpResponse

@csrf_exempt
def collect_coverage(request):
    coverage_dir = Path('js-coverage')
    if not coverage_dir.exists():
        coverage_dir.mkdir()

    try:
        json.dump(
            json.loads(request.data),
            (coverage_dir / 'coverage.{uuid.uuid4()}.json').open('w')
        )
    except ValueError:
        pass

    return HttpResponse('')

Oh, that wasn’t too bad either. It just checks that the incoming data is valid JSON, and then dumps it out to a uniquely named JSON file in js-coverage/.

We’ll need to install that in our urls.py:

if settings.DEBUG or settings.TESTING:
    path('coverage/', collect_coverage, name='collect-coverage')

Note well that you will need to ensure you don’t expose this view when not running your Selenium/Robot tests, else someone could very quickly fill up your disk with JSON files…

The last bit of the puzzle is to merge the various files together to generate the report:

nyc report --reporter lcov -t .js-coverage --report-dir .

This generates an lcov.info file that could be uploaded to a coverage analysis tool, and an lcov-report/ directory that contains an HTML report.

So, what about the tricky part, the bit that actually handles running the nyc instrument command on the files?

There are two types of scripts we need to be able to mutate: standalone files (ie, collectstatic), and inline scripts (within HTML templates).

Static files storage backends are relatively well-known, here’s some code that might work:

class InstrumentJSBackend(django.contrib.staticfiles.storage.StaticFilesStorage):
    def post_process(self, paths, dry_run=False, **options):
        if dry_run:
            return

        for path in paths:
            if path.endswith('.js'):
                # We need the original path name, not the one we have just copied...

            yield path, path, True

The latter can be done by using a custom template loader; it’s quite similar in construct to the supplied caching loader:

class Loader(BaseLoader):
    def __init__(self, engine, loaders):
        self.loaders = engine.get_template_loaders()
        super().__init__(engine)

    def get_dirs(self):
      for loader in self.loaders:
          if hasattr(loader, 'get_dirs'):
              yield from loader.get_dirs()

    def get_contents(self, origin):
        # Rewrite any script tags using `nyc instrument`
        contents = origin.loader.get_contents(origin)
        return self.rewrite(origin, contents)

    def get_template_sources(self, template_name):
        for loader in self.loaders:
            yield from loader.get_template_sources(template_name)

    def rewrite(self, origin, contents):

        return subprocess.check_output(['nyc', 'instrument', origin])