Django and Robot Framework

One of my colleagues has spent a bunch of time investigating and then implementing some testing using Robot Framework. Whilst at times the command line feels like it was written by someone who hasn’t used unix much, it’s pretty powerful. There are also some nice tools, like several Google Chrome plugins that will record what you are doing and generate a script based upon that. There are also other tools to help build testing scripts.

There is also an existing DjangoLibrary for integrating with Django.

It’s an interesting approach: you install some extra middleware that allows you to perform requests directly to the server to create instances using Factory Boy, or fetch data from Querysets. However, it requires that the data is serialised before sending to the django server, and the same the other way. This means, for instance, that you cannot follow object references to get a related object without a bunch of legwork: usually you end up doing another Query Set query.

There are some things in it that I do not like:

  • A new instance of the django runserver command is started for each Test Suite. In our case, this takes over 10 seconds to start as all imports are processed.
  • The database is flushed between Test Suites. We have data that is added through migrations that is required for the system to operate correctly, and in some cases for tests to execute. This is the same problem I’ve seen with TransactionTestCase.
  • Migrations are applied before running each Test Suite. This is unnecessary, and just takes more time.
  • Migrations are created automatically before running each Test Suite. This is just the wrong approach: at worst you’d want to warn that migrations are not up to date - otherwise you are testing migrations that may not have been committed: your CI would pass because the migrations were generated, but your system would fail in reality because those migrations do not really exist. Unless you are also making migrations directly on your production server and not committing them at all, in which case you really should stop that.

That’s in addition to having to install extra middleware.

But, back onto the initial issue: interacting with Django models.

What would be much nicer is if you could just call the python code directly. You’d get python objects back, which means you can follow references, and not have to deal with serialisation.

It’s fairly easy to write a Library for Robot Framework, as it already runs under Python. The tricky bit is that to access Django models (or Factory Boy factories), you’ll want to have the Django infrastructure all managed for you.

Let’s look at what the DjangoLibrary might look like if you are able to assume that django is already available and configured:

import importlib

from django.apps import apps
from django.core.urlresolvers import reverse

from robot.libraries.BuiltIn import BuiltIn


class DjangoLibrary:
    """

    Tools for making interaction with Django easier.

    Installation: ensure that in your `resource.robot` or test file, you have the
    following in your "***Settings***" section:

        Library         djangobot.DjangoLibrary     ${HOSTNAME}     ${PORT}

    The following keywords are provided:


    Factory:        execute the named factory with the args and kwargs. You may omit
                    the 'factories' module from the path to reduce the amount of code
                    required.

        ${obj}=     Factory     app_label.FactoryName       arg  kwarg=value
        ${obj}=     Factory     app_label.factories.FactoryName     arg  kwarg=value


    Queryset:       return a queryset of the installed model, using the default manager
                    and filtering according to any keyword arguments.

        ${qs}=      Queryset    auth.User       pk=1


    Method Call:    Execute the callable with tha args/kwargs provided. This differs
                    from the Builtin "Call Method" in that it expects a callable, rather
                    than an instance and a method name.

        ${x}=       Method Call     ${foo.bar}      arg  kwargs=value


    Relative Url:   Resolve the named url and args/kwargs, and return the path. Not
                    quite as useful as the "Url", since it has no hostname, but may be
                    useful when dealing with `?next=/path/` values, for instance.

        ${url}=     Relative Url        foo:bar     baz=qux


    Url:            Resolve the named url with args/kwargs, and return the fully qualified url.

        ${url}=     Url                 foo:bar     baz=qux


    Fetch Url:      Resolve the named url with args/kwargs, and then using SeleniumLibrary,
                    navigate to that URL. This should be used instead of the "Go To" command,
                    as it allows using named urls instead of manually specifying urls.

        Fetch Url   foo:bar     baz=qux


    Url Should Match:   Assert that the current page matches the named url with args/kwargs.

        Url Should Match        foo:bar     baz=qux

    """

    def __init__(self, hostname, port, **kwargs):
        self.hostname = hostname
        self.port = port
        self.protocol = kwargs.pop('protocol', 'http')

    @property
    def selenium(self):
        return BuiltIn().get_library_instance('SeleniumLibrary')

    def factory(self, factory, **kwargs):
        module, name = factory.rsplit('.', 1)
        factory = getattr(importlib.import_module(module), name)
        return factory(**kwargs)

    def queryset(self, dotted_path, **kwargs):
        return apps.get_model(dotted_path.split('.'))._default_manager.filter(**kwargs)

    def method_call(self, method, *args, **kwargs):
        return method(*args, **kwargs)

    def fetch_url(self, name, *args, **kwargs):
        return self.selenium.go_to(self.url(name, *args, **kwargs))

    def relative_url(self, name, *args, **kwargs):
        return reverse(name, args=args, kwargs=kwargs)

    def url(self, name, *args, **kwargs):
        return '{}://{}:{}'.format(
            self.protocol,
            self.hostname,
            self.port,
        ) + reverse(name, args=args, kwargs=kwargs)

    def url_should_match(self, name, *args, **kwargs):
        self.selenium.location_should_be(self.url(name, *args, **kwargs))

You can write a management command: this allows you to hook in to Django’s existing infrastructure. Then, instead of calling robot directly, you use ./manage.py robot

What’s even nicer about using a management command is that you can have that (optionally, because in development you probably will already have a devserver running) start runserver, and kill it when it’s finished. This is the same philosophy as robotframework-DjangoLibrary already does, but we can start it once before running out tests, and kill it at the end.

So, what could our management command look like? Omitting the code for starting runserver, it’s quite neat:

from __future__ import absolute_import

from django.core.management import BaseCommand, CommandError

import robot


class Command(BaseCommand):
    def add_arguments(self, parser):
        parser.add_argument('tests', nargs='?', action='append')
        parser.add_argument('--variable', action='append')
        parser.add_argument('--include', action='append')

    def handle(self, **options):
        robot_options = {
            'outputdir': 'robot_results',
            'variable': options.get('variable') or []
        }
        if options.get('include'):
            robot_options['include'] = options['include']

        args = [
            'robot_tests/{}_test.robot'.format(arg)
            for arg in options['tests'] or ()
            if arg
        ] or ['robot_tests']

        result = robot.run(*args, **robot_options)

        if result:
            raise CommandError('Robot tests failed: {}'.format(result))

I think I’d like to do a bit more work on finding tests, but this works as a starting point. We can call this like:

./manage.py robot foo --variable BROWSER:firefox --variable PORT:8000

This will find a test called robot_tests/foo_test.robot, and execute that. If you omit the test argument, it will run on all tests in the robot_tests/ directory.

I’ve still got a bit to do on cleaning up the code that starts/stops the server, but I think this is useful even without that.

Take that, Mr Morrison

People power still works.

Mr Morrison and his advisory panel still maintain that schools should remain open. For some reason, kids are immune to getting or spreading COVID-19 whilst at school, but if they visit their grandparents or a shopping centre, then all hell will break loose.

We, like many other parents around Australia, decided to withdraw our children from physical schooling (we are still doing remote learning, and our school has been very supportive of this) for nearly two weeks now.

A few days ago, we received an email from our school: the whole school will be remote-only; with exceptions for children of parents who are unable to perform remote learning, either because they work in essential jobs or whatever other reason. Those students will be managed at school, as of next week, but all students will be receiveng the same curriculum.

This is exactly what the NSW premier indicated her state’s schools would be doing. This makes perfect sense. Reduce the risk to teachers, and reduce the exposure of children to one another.

Now SA, and most other states, have announced early closure, and possible remote learning after the school holidays.

People voted by withdrawing their children from school. Maybe Mr Morrison should pay attention.

Human contact, or what the f*** are all these people doing

I went outside of my house today.

Initially I went to my office to get a few things that I’ll need to work from home (standing desk extension and a better trackpad were the main ones). But after that, I thought I’d swing by one of the smaller supermarkets in my area, and if it was quiet enough, pick up some supplies.

There were no customers when I went in. Good.

The first one who came in, when trying to get to a section near me, instead of taking the path that avoided me, instead took the path that went right by me. I quickly scrambled to get out of the way. Call me paranoid, but the fewer people I get close to, the smaller my chance of contracting the virus.

Eventually, too many people were in the store, so I left without everything I needed (I’d struggled to find some bits anyway). Having to get closer than I would have liked to people, I braved the checkout area, and made it outside.

So, for future reference, I came within a couple of metres of about 6 or 7 people at Drakes Mini between maybe 6:30 and 7pm.


The PM announced further measures, and said one thing I thought was interesting:

“It seems like lots of people want a complete lockdown”

Damn straight we do.

“Let me tell you, it won’t be for a short period of time.”

We are very aware of that: but it’s about public health. Man up, and lock it down. Have you not seen that China is able to start to release restrictions because they have started to control it?

Close Contacts and Community Transmission

The Australian government (and specifically the South Australian government) have made a big deal about how, so far at least, there has been virtually no “Community Transmission”. That it’s all been imports from overseas, other states, or “Close Contacts”.

But what do these terms “Close Contacts” and “Community Transmission” mean, in the scope of what normal people would assume they mean, and what the governments are using them to mean.

To me, a “Close Contact” would be a member of my immediate family, or perhaps one of the three people who share my office. Possibly even the other 6 people who work in my company who also work in the same place as me - since the air we all breathe is the same, and we share the same kitchen and bathroom facilities. Also, you’d probably count my extended family, although I don’t see them often, when I do see them we in close physical proximity. Indeed, even our close friends, who we don’t see as often as we would like, the sort of ones we catch up with every few months for dinner would count as close contacts.

A generous expansion of this might include the parents we see, and chat with every morning doing school drop-off. And of course, our kids’ close contacts would include their teacher and their closest friends. Note the word close.

So, you’d think that “Community Transmission” would include everyone else. People you come across in service roles, like the guy at the sushi bar, or your Barista. Maybe your neighbour that you chat to while walking your dog. The other parents who you stop briefly to and say hello. The Principal of the school, who you talk to probably once a week.

But, according to The Conversation AU, these people would be counted by the official tally as “Close Contacts”.

I think this belies how serious of a mismatch there is between the government’s current position and communication and reality there is. Keeping schools open will not, according to their definition, cause Community Transmission. It cannot, because by definition, this would count as Close Contacts.

Perhaps this is part of the reason they want to keep schools open. Being able to claim that cases are still being spread only through Close Contacts, when in reality the virus is being transmitted through the school COMMUNITY.

Maybe I’m turning into a tin-hatter, but I feel like our government is not trying hard enough to reduce the growth in number of cases. At this stage, the number of cases in South Australia is speeding up: it was growing at 23%, but now seems to be growing at 33%. Sure, some of this is better testing, but it doesn’t seem at this stage that many of the measures they have started to take have had much impact on the spread.

Lockdown: Day 6

So I guess this is becoming a bit of a journal. I missed yesterday though… ;)

The Australian government announced further economic aid to people who are newly unemployed or under-employed due to COVID-19. Notably, this included doubling the Jobseeker allowance (or whatever it’s currently called). This is surely a kick in the guts for the previously unemployed. Does it not state that the previous amount was way less than what an average person needs to survive? But now there are going to be significant numbers of people who until a week ago were fully-employed, the amount doubles.

Interesting.

We are going to take the boys out for a walk in the local National Park. I don’t know how draconian lockdown rules are going to be in the future. Keeping schools open seems to suggest they don’t really give a fuck though.

Our internet was pretty crappy all day yesterday; and a bit intermittent on Friday. Right now, it’s totally gone - allegedly there is an issue at some place that is up the chain from me: they have turned the power off there and are working on it. Hopefully it comes back on today.

Using Zoom for remote Cello lessons

With COVID-19 starting to cause significant lockdowns, our music lessons have moved to online. Our piano and cello teachers are both using Zoom for this.

Piano was fine, but the Cello teacher noticed that when my eldest son was playing some notes (especially the lower ones), the audio was cutting out. When he mentioned it was fine at the start of the note, but dropped in the middle, and then came back at the end, it triggered something in my brain.

Turns out Zoom has automatic background noise cancellation. This was picking up the low sounds of the Cello and blocking them.

This is fairly easy to fix:

SettingsAudioAdvanced

Then, set both types of background noise suppression to disabled:

Zoom Settings Audio Advanced

I’m not sure which one is more important: you’d think it was probably intermittent, but it was easy enough in our case just to leave them both set to disable.

Lockdown: Day 4

It’s Friday. End of our first week of working and schooling from home.

We bought a new iPad for the boys, which arrived within 25 hours. They are now using it attached to the trampoline; the old iPad was not working with that, so that’s a bonus. We also bought a water filter, so it’s been a productive week.

Working from home has been fine. I’m sure I’m not as productive as I was in previous weeks, but I don’t know how much of that is COVID-19 in general playing on my mind.

Australia’s infection rate is still above what my 23% compounding increase model suggests. https://covid-19-au.github.io is currently showing 854 confirmed cases, whilst my simple predictions said 835 by today. Watch it go over 1000 tomorrow.

According to the Guardian, our rates are slightly better than most other countries, although only Japan is an outlier there. Since that’s a logarithmic scale, we are perhaps even better off than it seems. Not sure how much of that is just insufficient testing though. On the same page, you can see that our total number of deaths is low: however, there is nothing that shows deaths-per-infection.

SA still sits at 50, which is about 30% better than predicted. However, we are not doing nearly as much testing as NSW.

Recommendations are to stay at home all weekend. I suspect I will be taking the boys somewhere - perhaps we could go for a walk in BNP.

Lockdown: Day 3

Took the boys down to the park this afternoon. They hadn’t left the house since Monday evening.

The school adjacent to the park was closed (due to a case of COVID-19), but otherwise, it didn’t feel that different. There were a couple of other groups of people there, and we ran/scooted/played for about an hour.

Getting them out of the house was really good - not only did they get to burn a bunch of energy, but it seems to have made them better able to play together this evening. They’ve been really good so far in terms of not fighting that much, but it is only day 3.

I’d be so happy if we turn out to have gone way too gung-ho on this - if South Australia (and Australia) manage to get this outbreak under control in weeks, rather than many months, I’ll be happy to cop shit for going overboard and pulling the kids out of school. But the number of cases has continued to grow at a rate higher than 23% per day.

My modelling, which is simplistic, looks like this:

    date    │ infected
────────────┼──────────
 2020-03-17 │      449
 2020-03-18 │      552
 2020-03-19 │      679
 2020-03-20 │      835
 2020-03-21 │     1027
 2020-03-22 │     1263
 2020-03-23 │     1553
 2020-03-24 │     1910
 2020-03-25 │     2349
 2020-03-26 │     2889
 2020-03-27 │     3553
 2020-03-28 │     4370
 2020-03-29 │     5375
 2020-03-30 │     6611
 2020-03-31 │     8132
 2020-04-01 │    10002

That’s stopping at 10k, by April 1st.

Currently, it’s 2020-03-19, and there are 691 confirmed cases.

Lockdown: Day 2

I really don’t know how this is going to play out.

It seems that the Australian government just doesn’t want to shut schools because they know when they do shut them, they won’t be re-opening them for many months. What they’d prefer to do is hold that off as long as possible, at the cost of people’s lives.

Currently, in Australia, the number of cases of COVID-19 appears to be growing at a rate exceeding 23% per day.

At that rate, we’ll have over 1000 cases by the weekend, and 10k within a fortnight or so. Within three weeks, there will no longer be any where near enough intensive care beds or ventilators to support the number of critically ill patients, and we’ll start to see Italy-like levels of mortality.

How will schools stay open when teachers are dying.


Our parents don’t seem to realise how dangerous this is.

My parents are apparently still planning on travelling from a rural town of 6000 people (they actually live on a farm, so that’s even better), where they are relatively safe, to Adelaide to attend a funeral. I understand that it’s one of their friends that died, but to be honest, there will be a lot more of those in the not too distant future.

And my partner’s mother was planning on going to the hairdresser tomorrow.


Personally, we have savings, and I have a stable job (for now). But considering the number of our customers who are service industry, quite a number of which I have little doubt will be out of business in short order.

There will be a new normal, we just don’t know what that is going to look like.

Lockdown: Day One

The Australian government is in denial.

It’s becoming clear that the global health crisis from COVID-19 is very likely to worsen quite quickly. From information I have come across from reliable medical sources (mostly from the ABC), it seems that our government is probably not doing enough to slow the growth of cases, and that the health risks to all age groups are significant, not just the elderly or vulnerable. Similarly, it feels like our health system has every likelihood of being overwhelmed by the increasing case load of critically ill people.

Because I have the “luxury” of being able to work from home (and still be gainfully employed), I am now doing so; additionally we are keeping our children home from school. Whilst there is probably very little practical risk from any two students attending one specific school, every person you come into contact with, and more so every person you remain in contact with for a significant period of time is an increased risk.

Other countries have been recommending that gatherings larger than 5 people are sufficient to continue the spread of the virus, and whilst it may seem that Australia (and Adelaide specifically) has less risk right now than some European and Asian countries, we have already seen Victoria declare a State of Emergency. Yet our government maintains that 500 people is still safe.

The number of confirmed cases in Australia has reached a higher proportion of the general population than that of the United States, which has many states and cities implementing strict shutdown policies.

The actual number of confirmed cases has started to grow exponentially, with an increase of almost 50% in one day (20 to 29 on March 16th). The school 400m away from where I live has a confirmed case - the principal of my children’s school has a child that attends there. The school where my niece attends school has a confirmed case.

Around 4000 doctors sent a petition to the government asking for strict lockdowns, fearing that our outcomes could be worse than that of Italy.

The government continues to maintain that this disease is just not that risky for young people.

This is just a flat-out lie.

The only reason we have not had deaths in Australia of young people is just a matter of numbers. Currently, anyone who has had severe respiratory issues has been able to receive artificial respiration. This will change, as there are only so many resipirators in Australian hospitals. As more people are infected, more will get serious complications, and some of them will miss out. Either on a respirator altogether, or later in the epidemic, perhaps even a bed.

Children appear not to be having severe complications, or even especially strong symptoms, but there is evidence that asymptomatic people shed a higher amount of the virus, and children continue to have the virus in stool samples well after they are deemed to be “clear” according to a throat swab.


So, my kids, partner and I stayed home today. I think it’s still fun for them - we have tried to help them understand the scale of this problem (“We could be here, just the four of us together for at least two weeks, maybe even longer”), but the novelty (oops) of being home-schooled has yet to wear off.

I’m thankful I have the means and opportunity to work from home, and have my immediate family here with me. We aren’t going to see the boys’ grandparents for some weeks, and I’m sure we’ll all go batshit insane from being around one another non-stop for an indefinite period of time.

Stay safe.