Schinckel.net

Wireguard and Dynamic Hostnames

2021-11-02T19:52:45+00:00

We use Wireguard at work for our VPN: this allows us to limit who can access our administration interface based on being on our company VPN.

I’ve set up Wireguard on my router so that I don’t need to connect to the VPN on each of my devices, but I also use a split tunnel so that only the IP addresses that are required go through there. I discuss in a previous post how I use VPN Policy Based Routing to ensure that the IP addresses are updated there.

However, I also run a different Wireguard interface allowing me to connect back to my LAN. This sits behind a dynamic IP address (even though it doesn’t change much). In most cases that’s okay - if my IP address happens to have changed, I can just reconnect my VPN, and in reality it’s very unlikely that I’d be actually using the VPN when the IP address changed.

However, I also have a device I’m going to put into the beachhouse, allowing me to manage the network from there. It would be nice not to have to maintain a seperate set of dynamic hostnames, but instead set up Wireguard to keep a connection between those two devices.

After a bunch of mucking around (turns out the OpenWRT UI page is misleading about the purpose of the AllowedIPs field. Pro tip: don’t have overlapping network ranges in the “server” configuration, else only one of them will work at a time), I finally got this working, and my old Raspberry Pi (which is still at my house) is all ready to go for a long holiday at the beach.

But, I’m still worried about losing the VPN when my IP address changes. There’s not always someone at the beachhouse, although there could be tenants at the other beachhouse (which shares the network, although it’s all segmented using VLANs). So, I may need to do stuff at any time.

There are a couple of tools out there that refresh the connection on a Wireguard peer, but I thought I’d have a go at writing my own script.

#! /bin/bash

update_endpoint() {
  local IFACE=$1
  local ENDPOINT=$(cat /etc/wireguard/${IFACE}.conf | grep '^Endpoint' | cut -d '=' -f 2)
  # No need to refresh if ne endpoint.
  [ -z ${ENDPOINT} ] && return 0;
  local HOSTNAME=$(echo ${ENDPOINT} | cut -d : -f 1)
  local PORT=$(echo ${ENDPOINT} | cut -d : -f 2)
  local PUBLIC_KEY="$(wg show ${IFACE} peers)"
  # No need to refresh if no handshake
  [ -z $(wg show ${IFACE} latest-handshakes | grep ${PUBLIC_KEY} | awk '{print $2}') ] && return 0;
  local ADDRESS=$(host -4 ${HOSTNAME} | grep 'has address' | awk '{print $4}')
  # Return if we don't find any matching lines here - that means our IP address matches.
  [ -z "$(wg show ${IFACE} endpoints | grep ${PUBLIC_KEY} | grep ${ADDRESS})" ] || return 0;
  wg set ${IFACE} peer ${PUBLIC_KEY} endpoint "${HOSTNAME}:${PORT}"
}


WG_IFS=$(wg show interfaces)

for WG_IF in $WG_IFS ; do 
  update_endpoint $WG_IF
done

This sweet little number iterates through each wireguard interface, and checks to see if the current IP address matches the hostname’s IP address. If not, it updates the running configuration.

Airport Express/Extreme and IoT Devices

2021-10-25T13:38:11+00:00

At some point (I think it was when I needed to track down what device on my network was consuming multi-gigabytes of data daily), I moved to OpenWRT on my gateway router. Prior to that, I’d just used an Airport Express, and several other Extreme and Express units around my house.

I still use the Express (and Extreme) units for WiFi. Until yesterday, I’d repurposed a couple of older Airport Express units that were running the IoT network themselves. I could have had those as an isolated network, but it’s convenient to be able to have some connection between the two. I do use a set of firewall rules to allow devices on LAN to access IOT, but not vice-versa. This allows me to update the firmware on my esphome devices without having to switch WiFi networks.

So, I had used a VLAN (and made it untagged on one port of the switch in the gateway router) for IOT. Then I had this connected to an older AirPort Express, which created the network. Then, and additional (only linked wirelessly) Airport Express extended this network.

I had thought that it would be neat to have the regular WiFi access points also creating the IOT WiFi network: indeed I nearly bought more Ubiquiti APs to install here just to do that (I put them in the beach houses).

I’m glad I didn’t: it turns out the the Guest Network functionality on the Airport devices can be repurposed for what I needed.

The magic bit of information is that the Airport base stations uses VLAN untagged ethernet frames for the main WiFi network, and VLAN 1003 tagged ethernet frames for the “Guest” network.

So all I needed to do was change the VLAN id I was using for IOT, and ensure that my router (and managed switch(es)) were passing through tagged frames for VLAN 1003 to all relevant devices. And then configure the IOT network on the existing Airport base stations. Oh, and unplug the two standalone Airport Express units.

The other thing I was able to do is add virtual Ethernet Adapters on some devices so they also exist on the IOT VLAN. This turned out to be much easier on a Mac than I thought it would be thanks to Apple Support than it was on Raspberry Pi, although not by much.

VPN Policy Based Routing for Hostnames

2021-10-25T10:42:19+00:00

I need to route traffic to our staging server (and the production server, so I can access our administration interface) through a VPN. But I don’t want to have to connect to the VPN all of the time, it should automatically stay up, but only route traffic that goes to the specific sites.

I’ve had limited success with using domain-name based VPN policy routing on OpenWRT. It feels like it doesn’t really work. It could be that it doesn’t add all of the IP addresses for each domain name, and if the IP address changes, then it stops working.

Turns out these are distinct problems, but it is possible to solve both of them.

Instead of trying to configure it using the UI, you can add a new “Custom User File Include” at the bottom. I have /etc/vpn-policy-routing.<name>.user, with the following:

#!/bin/sh

TARGET_IPSET='wg'
HOSTS='<production-server> <staging-server>'

for host in $HOSTS
do
  host $host | grep -v 'IPv6' | awk -v ipset="$TARGET_IPSET" '{print "add " ipset " " $(NF)}' | ipset restore
  host $host | grep 'IPv6' | awk -v ipset="$TARGET_IPSET" '{print "add " ipset "6 " $(NF)}' | ipset restore
done

Because the two ipset targets need to be different (wg for IPv4, and wg6 for IPv6), then we need to deal with them differently.

In this case, we just iterate through each hostname, and extract each matching IP address. We then create a string that looks like add wg 1.1.1.1 (or the IPv6 version), and pass these to ipset restore.

You will want to make sure that your IP addresses are only added using this method, as ipset restore will complain about duplicates.

You can execute this file directly, to see that it works.

Then, the other part of the problem is making sure this is kept up to date. I use a cron job to /etc/init.d/vpn-policy-routing reload every 10 minutes.

*/10 * * * * /etc/init.d/vpn-policy-routing reload

You will need to put this file back after updating OpenWRT, which is a bit annoying. Took me a while to figure out why my VPN connection was not routing correctly.

Automatically expire rows in Postgres

2021-09-09T14:16:36+00:00

Here’s a fun idea: how to make a database table where the data is kept only for a certain period of time?

One solution could be to have a column updated_at, which is set to the current timestamp each time a row is updated. Then you need a scheduled task that periodically clears out rows older than the threshold.

We can do this in a single trigger function:

CREATE OR REPLACE FUNCTION keep_for() RETURNS TRIGGER AS $$
  DECLARE
    primary_key_name TEXT = (
        SELECT attname
        FROM pg_index
        JOIN pg_attribute ON
            attrelid = indrelid
            AND attnum = ANY(indkey)
        WHERE indrelid = TG_RELID AND indisprimary
    );
    primary_key_value TEXT;
  BEGIN
    IF TG_WHEN <> 'BEFORE' THEN
      RAISE EXCEPTION 'keep_for() may only run as a BEFORE trigger';
    END IF ;

    IF TG_ARGV[0]::INTERVAL IS NULL THEN
      RAISE EXCEPTION 'keep_for() must be installed with an INTERVAL to keep data for';
    END IF;

    NEW.updated_at = now();

    IF TG_OP = 'INSERT' THEN
      EXECUTE 'DELETE FROM ' || quote_ident(TG_TABLE_NAME) || ' WHERE updated_at < now() - INTERVAL ' || quote_literal(TG_ARGV[0]::TEXT) || ';';
    ELSIF TG_OP = 'UPDATE' THEN

      EXECUTE 'DELETE FROM ' || quote_ident(TG_TABLE_NAME)
                             || ' WHERE updated_at < now() - INTERVAL ' || quote_literal(TG_ARGV[0]::TEXT)
                             || ' AND ' || quote_ident(primary_key_name) || ' <> ' || quote_literal(row_to_json(NEW) ->> primary_key_name) || ';';
    END IF;

    RETURN NEW;
  END;
$$ LANGUAGE plpgsql;

Now, we can install this on a table (that has a single primary key column):

CREATE TRIGGER keep_for_one_week
BEFORE UPDATE OR INSERT ON <target_table>
FOR EACH ROW
EXECUTE PROCEDURE keep_for('1 week');

This only works on tables that are updated somewhat regularly, and that are not too big. You’ll also want an index on that updated_by column.

This was a thought experiment (which I did implement), until my co-worker pointed out that there was no reason to actually store this data in the database to begin with, since we already use Redis we can just set the TTL on the key when we set it, and it will automatically expire without us having to do extra work.

Django Implied Relationship

2021-07-14T17:35:43+00:00

[This post was updated on 2021-07-28].

A little while ago, Alec McGavin put up a post on the Kogan blog about Custom Relationships in Django. This is a really cool way to get a relationship in Django that exists in the database, but cannot be modelled correctly in Django. For instance, this could be data in the database that does not have a Foreign Key, either because it’s legacy data, or because it’s possible either side of the relationship might be added to the database before the other, rather than the normal order in a database where the target table of the FK is always added to first.

However, I have another slighly different situation where an implied relationship exists, but should not be stored directly.

Consider the following data structures:

class Employee(models.Model):
    name = models.TextField()


class EmploymentPeriod(models.Model):
    employee = models.ForeignKey(
        Employee,
        related_name='employment_periods',
        on_delete=models.CASCADE,
    )
    valid_period = DateRangeField()

    class Meta:
        constraints = [
            ExclusionConstraint(
                name='employment_overlap',
                expressions=[
                    ('employee', RangeOperators.EQUAL),
                    ('valid_period', RangeOperators.OVERLAPS),
                ]
            )
        ]


class Shift(models.Model):
    employee = models.ForeignKey(
        Employee,
        related_name='shifts',
        on_delete=models.CASCADE,
    )
    date = models.DateField()
    start_time = models.TimeField()
    duration = models.DurationField()
    employment_period = Relationship(
        EmploymentPeriod,
        from_fields=['employee', 'date'],
        to_fields=['employee', 'valid_period'],
    )

    @property
    def start(self):
        return datetime.datetime.combine(self.date, self.start_time)

    @property
    def finish(self):
        return self.start + self.duration

Now, there is a direct relationship between Shift and Employee, and also between EmploymentPeriod and Employee, but there could be an inferred relatonship between Shift and EmploymentPeriod. Because of the exclusion constraint, we know there will only be one EmploymentPeriod for a given Shift.

It would be really nice to be able to create this relationship, so we can reference the employment period (or lack thereof) directly. The Relationship class above goes close, but tries to use an equality check between the date and date_range fields.

It turns out, we can add a bit to that class, and teach it how to handle the various ways this relationship can be accessed:

Shift().employment_period
EmploymentPeriod().shifts
Shift.objects.select_related(‘employment_period’)
EmploymentPeriod.objects.prefetch_related(‘shifts’)
Shift.objects.filter(employment_period=emp)
Shift.objects.filter(employment_period__in_=[emp1, emp2])
Shift.objects.filter(employment_period__isnull=True)
Shift.objects.filter(employment_period=None)
Shift.objects.filter(employment_period__isnull=False)
EmploymentPeriod.objects.filter(shifts__contains=shift)
EmploymentPeriod.objects.filter(shifts__contains=[shift1, shift2])
EmploymentPeriod.objects.filter(shifts__isnull=True)
EmploymentPeriod.objects.filter(shifts__isnull=False)

…plus there is also the inverse of a bunch of these - ie Shift.objects.exclude(employment_period=emp). In some cases these are equivalent, but that’s not always possible to determine.

Let’s have a look at the original class, and a new subclass for these non-direct relationships:

from django.db import models
from django.db.models.lookups import Lookup


class Relationship(models.ForeignObject):
    """
    Create a django link between models on a field where a foreign key isn't used.
    This class allows that link to be realised through a proper relationship,
    allowing prefetches and select_related.

    https://devblog.kogan.com/blog/custom-relationships-in-django
    """

    def __init__(self, model, from_fields, to_fields, **kwargs):
        super().__init__(
            model,
            on_delete=models.DO_NOTHING,
            from_fields=from_fields,
            to_fields=to_fields,
            null=True,
            blank=True,
            **kwargs,
        )

    def contribute_to_class(self, cls, name, private_only=False, **kwargs):
        # override the default to always make it private
        # this ensures that no additional columns are created
        super().contribute_to_class(cls, name, private_only=True, **kwargs)

There’s not much to this, which is part of the beauty of it. Django pretty-much handles composite-key relationships, it just won’t create actual ForeignKeys based on them. There have been noises about implementing this for years, and maybe eventually it will happen.

But what about a subclass that allows the implicit relationship described above?

class ImplicitRelationship(Relationship):
    """
    Create a django link between two models where at least one of the fields
    uses a containment (or other type of non-direct) relationship.

    Relationship should be used if this is just a composite key (or a single
    key that is not a real ForeignKey in the database).

    """

    def get_path_info(self, filtered_relation=None):
        """Get path from this field to the related model."""
        opts = self.remote_field.model._meta
        from_opts = self.model._meta
        self.related_fields
        return [
            PathInfo(
                from_opts=from_opts,
                to_opts=opts,
                target_fields=[rhs for lhs, rhs in self.other_related_fields],
                join_field=self,
                m2m=False,
                direct=False,
                filtered_relation=filtered_relation,
            )
        ]

    def get_reverse_path_info(self, filtered_relation=None):
        """Get path from the related model to this field's model."""
        opts = self.model._meta
        from_opts = self.remote_field.model._meta
        self.related_fields
        return [
            PathInfo(
                from_opts=from_opts,
                to_opts=opts,
                target_fields=[lhs for lhs, rhs in self.other_related_fields],
                join_field=self.remote_field,
                m2m=False,
                direct=False,
                filtered_relation=filtered_relation,
            )
        ]

    @cached_property
    def other_related_fields(self):
        return self.resolve_related_fields()

    @cached_property
    def related_fields(self):
        return []

    def get_local_related_value(self, instance):
        """
        Given an instance, determine the value that will be used as
        the key for this value in a dict of related items.

        This is where it starts to get tricky. Django only really expects
        keys to match exactly, but we may have a value that contains a
        date, that needs to be checked for inclusion in a DateRange.

        Whilst psycopg2 does not normalise Range values, it will handle
        <date> in <DateRange> correctly, so we can use that as the
        comparison.
        """
        parts = self.get_instance_value_for_fields(
            instance,
            [lhs for lhs, rhs in self.other_related_fields],
        )

        if not hasattr(self, '_known_instance_keys'):
            return parts

        if parts in self._known_instance_keys:
            return parts

        for keys in self._known_instance_keys:
            for part, key in zip(parts, keys):
                if part == key or getattr(key, '__contains__', None) and part in key:
                    return keys

    @property
    def get_foreign_related_value(self):
        """
        Because we need to use non-exact matching, we need to set up to store
        instances based on known keys. The Django code that uses this builds up
        a dict of keys/values, but since we need to do containment testing in
        get_local_related_value(instance), we have to store a local set of
        key values, which will be used for that containment checking.

        This is implemented as a property that returns a function, after clearing
        out the cache of known instances so that each queryset will have it's own
        cache. Otherwise, instances from the last run through would be matched
        in the next run.

        This uses knowledge of the Django internals, where this method is called
        before get_local_related_value, which really is not ideal, but there does
        not seem to be a better way to handle this.
        """
        self._known_instance_keys = set()

        def get_foreign_related_value(instance):
            values = self.get_instance_value_for_fields(
                instance, 
                [rhs for lhs, rhs in self.other_related_fields]
            )

            self._known_instance_keys.add(values)
            return values

        return get_foreign_related_value

    def get_extra_restriction(self, where_class, alias, remote_alias):
        """
        This method is used to get extra JOIN conditions.

        We don't need to include the exact conditions, only those
        that we filtered out from the regular related_fields.
        The exact conditinos will be already applied to the JOIN
        by the regular handling.
        """
        if not alias or not remote_alias:
            return

        if self.other_related_fields:
            cond = where_class()

            for local, remote in self.other_related_fields:
                local, remote = local.get_col(remote_alias), remote.get_col(alias)
                lookup_name = JOIN_LOOKUPS.get(get_key(local, remote), 'exact')
                lookup = local.get_lookup(lookup_name)(local, remote)
                cond.add(lookup, 'AND')

            return cond

    def get_extra_descriptor_filter(self, instance):
        """
        The sibling to get_extra_restriction, this one is used to get the extra
        filters that are required to limit to the correct objects.
        """
        extra_filter = {}
        for lhs, rhs in self.other_related_fields:
            lookup = JOIN_LOOKUPS.get(get_key(rhs.cached_col, lhs.cached_col), 'exact')
            extra_filter[f'{rhs.name}__{lookup}'] = getattr(instance, lhs.attname)
        return extra_filter

    def get_where(self, value, alias=None):
        constraint = WhereNode(connector=AND)

        values = self.get_instance_value_for_fields(
            value, 
            [remote for local, remote in self.other_related_fields]
        )

        for (source, target), value in zip(self.other_related_fields, values):
            key = (source.get_internal_type(), target.get_internal_type())
            lookup_type = JOIN_LOOKUPS.get(key, 'exact')
            lookup_class = source.get_lookup(lookup_type)
            lookup = lookup_class(target.get_col(alias or self.model._meta.db_table, source), value)
            constraint.add(lookup, AND)

        return constraint

    def get_exists_subquery_filters(self, inverted=False):
        filters = {}
        for source, target in self.other_related_fields:
            if inverted:
                source, target = target, source
            key = (target.get_internal_type(), source.get_internal_type())
            lookup = JOIN_LOOKUPS.get(key, 'exact')
            filters[f'{target.attname}__{lookup}'] = OuterRef(source.attname)
        return filters

    def exists_subquery(self, negated=False, inverted=False, **filters):
        if inverted:
            exists = Exists(
                self.model.objects.filter(
                    **self.get_exists_subquery_filters(inverted=True), 
                    **filters
                ).values('pk')
            )
        else:
            exists = Exists(
                self.related_model.objects.filter(
                    **self.get_exists_subquery_filters(), 
                    **filters
                ).values('pk')
            )
        return ~exists if negated else exists

    @property
    def target_field(self):
        raise FieldError()


# We should be able to add more pairs here as we need to handle them.
JOIN_LOOKUPS = {
    ('DateField', 'DateRangeField'): 'contained_by',
    ('DateRangeField', 'DateField'): 'contains',
}

def get_key(lhs, rhs):
    return (lhs.output_field.get_internal_type(), rhs.output_field.get_internal_type())

There’s actually a lot more code there than I really wanted, however it seems mostly to be necessary.

But wait, there’s more. We also need to teach Django how to handle the various lookups that can be performed on these relationships:

@ImplicitRelationship.register_lookup
class RelatedMultipleExact(Lookup):
    lookup_name = 'exact'
    """
    Apply each lookup type from each of the fields in an ImplicitRelationship.

    This is for querysets of the form:

    >>> Foo.objects.filter(relationship=instance)

    This gets the relevant operator for each of the lookups, based on
    the field type of the pair of (to/from) fields.
    """

    def as_sql(self, compiler, connection):
        field = self.lhs.field

        # If we only have a primary key here, and not an instance, then we
        # will need to push the querying back into the database - normally
        # a diroct lookup just uses the value as a PK, but here we need to
        # get the database to do a subquery to get the other values.
        if self.rhs_is_direct_value() and not isinstance(self.rhs, models.Model):
            # We can't unref the alias here, because Django will have also put in a IS NOT NULL
            # on the thing, which is referencing the wrong table.
            compiler.query.alias_map[self.lhs.alias] = compiler.query.alias_map[self.lhs.alias].promote()
            return field.exists_subquery(pk=self.rhs).resolve_expression(compiler.query).as_sql(compiler, connection)

        return field.get_where(self.rhs, alias=self.lhs.alias).as_sql(compiler, connection)


@ImplicitRelationship.register_lookup
class RelatedMultipleIn(Lookup):
    """
    Apply each lookup from each of the fields in an ImplicitRelationship.

    This is for querysets of the form:

    >>> Foo.objects.filter(relationship__in=[instance1, instance2])
    >>> Foo.objects.filter(relationship__not_in=[instance1, instance2])

    This builds an EXISTS() clause that uses a subquery to find
    if each instance matches - this is usually better than a bunch
    of clauses that would use OR, because that would preclude the
    use of indexes.
    """

    lookup_name = 'in'
    negated = False

    def as_sql(self, compiler, connection):
        field = self.lhs.field
        if self.negated:
            # We remove one reference to the joined table, so that if we only
            # have this reference, ie no columns, then we don't even join the
            # table in (as we'll be using an EXISTS in WHERE)
            compiler.query.unref_alias(self.lhs.alias)
        return (
            field.exists_subquery(
                negated=self.negated,
                pk__in=[getattr(x, 'pk', x) for x in self.rhs],
            )
            .resolve_expression(compiler.query)
            .as_sql(compiler, connection)
        )


@ImplicitRelationship.register_lookup
class RelatedMultipleNotIn(RelatedMultipleIn):
    negated = True
    lookup_name = 'not_in'


@ImplicitRelationship.register_lookup
class RelatedMultipleNull(Lookup):
    """
    Apply each lookup from each of the fields in an ImplicitRelationship.

    This is for querysets of the form:

    >>> Foo.objects.filter(relationship=None)
    >>> Foo.objects.exclude(relationship=None)
    >>> Foo.objects.filter(relationship__isnull=True)
    >>> Foo.objects.filter(relationship__isnull=False)

    """

    lookup_name = 'isnull'

    def as_sql(self, compiler, connection):
        field = self.lhs.field

        if not isinstance(field, Relationship):
            pk = field.related_model._meta.pk
            lookup = pk.get_lookup('isnull')(pk.get_col(self.lhs.alias), self.rhs)
            return lookup.as_sql(compiler, connection)

        # We remove one reference to the joined table, so that if we only
        # have this reference, ie no columns, then we don't even join the
        # table in (as we'll be using an EXISTS in WHERE)
        compiler.query.unref_alias(self.lhs.alias)
        return field.exists_subquery(negated=self.rhs).resolve_expression(compiler.query).as_sql(compiler, connection)


@ImplicitRelationship.register_lookup
class RelatedMultipleContains(Lookup):
    """
    Apply each lookup from each of the fields in an ImplicitRelationship.

    This is for querysets of the form:

    >>> Foo.objects.filter(reverse_relationship__contains=[x, y])
    >>> Foo.objects.filter(reverse_relationship__contains=x)

    """

    lookup_name = 'contains'
    negated = False

    def as_sql(self, compiler, connection):
        if isinstance(self.lhs.field, Relationship):
            raise TypeError(f'Unable to perform __{self.lookup_name} queries on Relationship, only on reversed')

        field = self.lhs.field.remote_field

        try:
            iter(self.rhs)
        except TypeError:
            value = getattr(self.rhs, 'pk', self.rhs)
            lookup = 'pk'
        else:
            if self.negated:
                raise ValueError('Unable to perform not_contains=list')
            value = [getattr(x, 'pk', x) for x in self.rhs]
            lookup = 'pk__in'

        # We remove one reference to the joined table, so that if we only
        # have this reference, ie no columns, then we don't even join the
        # table in (as we'll be using an EXISTS in WHERE)
        compiler.query.unref_alias(self.lhs.alias)

        return (
            field.exists_subquery(negated=self.negated, inverted=True, **{lookup: value})
            .resolve_expression(compiler.query)
            .as_sql(compiler, connection)
        )


@ImplicitRelationship.register_lookup
class RelatedMultipleNotContains(RelatedMultipleContains):
    lookup_name = 'not_contains'
    negated = True

There is one caveat to this - these lookups (under certain conditions) will change the list of FROM tables that are required to be joined into the query. In some cases this adds a required JOIN, in others it actually removes the need for a JOIN because the WHERE clause is all within a subquery, and should not contain a join to the original table.

However, until https://github.com/django/django/pull/14683 is merged, these lookups will not always work - if the list of tables is mutated by the lookup, the SQL query that is generated will not contain this required mutation. Hopefully I can get that PR merged, but it is possible to use patchy to patch your local installation until that is done.

Optus Billing Misleading Chart

2021-07-10T18:30:42+00:00

I really hate when people use charts or graphs in a misleading manner. It’s even worse when it’s big companies, and in my latest bill from Optus, I noticed a particularly egregious chart.

For those that don’t notice at first (it’s taken me a bunch of billing months to notice, actually), the middle line is slightly longer than the other two. In that week, I went over my data plan, and had to pay an extra $10 to get more data.

I don’t have a problem with this, but I do have a problem with the scaling on the axis.

The shorter ones are months where I was billed $35. The longer one is where I was billed $45.

That’s a difference of $10, which is ~30%. However, the middle bar in the chart is not even close to 20% longer. It looks to me to be around 20 pixels longer, or around 3.5%

So, when you overspend on your bill, it looks like you are only paying a small amound extra, when in fact I am paying a significant amount more on that given bill.

The only way you could have scaling like this would be if you had a logarithmic scale. Which, granted, many more people are familiar with now than before the pandemic, but you don’t use a logarithmic scale when your values are all of the same order of magnitude, unless you want to mislead people.

I find it pretty unlikely that a high proportion of Optus’s customers have a monthly billing variance that differs by orders of magnitude. I also find it pretty unlikely that this is a mistake - someone at Optus (and I suspect not someone in their tech department, great work dumping on them chat support person) has decided that their charts should be tweaked so that overspends look like they are “not that big”.

Who wants to bet Optus never get back to me about my complaint?

KnockoutJS Watchdog

2021-06-07T18:15:49+00:00

I first discovered KnockoutJS many years ago. It’s a really neat way of managing dependencies between values, and also updating HTML accordingly. Recently, I rewrote my fronius-dashboard to use a Quart app and KnockoutJS instead of a Phoenix LiveView, as the latter was much harder to build (and dependency changes seem to break that fairly frequently).

One thing I did notice is that from time to time I’ll come back to a pinned tab version of the app and it will no longer be updating. I’m not sure exactly why that is, but it seems to be connected still (the EventSource object says it is still connected, anyway).

So, I thought maybe a watchdog process would be in order.

Basically, if there are no updates for a certain period of time, then it should just reload the page.

KnockoutJS doesn’t have anything like this built in, but it does have the facility to throttle updates, and and specifically, to use the “only update when changes stop” facility. As such, we can build a watchdog fairly easily using just that:

let watchdog = ko.computed(() => viewModel.grid()).extend({
  rateLimit: {
    timeout: 10000,
    method: "notifyWhenChangesStop"
  }
});
watchdog.subscribe(() => {
  document.location.reload();
});

In this case, I have one value that will almost certainly update every second (or more frequently than this), called grid.

If there are no updates for 10 seconds, then it will reload the page.

I’m not sure that this will actually solve my problem though - I suspect that the stoppage happens because the browser tab is not frontmost, and the JS is suspended, and doesn’t un-suspend properly. Time will tell, I guess.

Extending Django Wizards

2021-04-23T20:12:51+00:00

I quite like the wizard pattern. Most people are somewhat familiar with the concept of having to complete a logically grouped set of form elements before moving onto the next one, with a final confirmation step, after which the operation is committed.

Howver, there are some perceieved problems within the Django formtools implementation, at least from the perspective of our use case.

The big one my team have identified: when you reload the page with a GET request, it clears out the data from the current run of the wizard. This is problematic when something happens to your internet, and you have to reload a page manually, for instance. Or if you just happen to reload it through another means.

Related to this, if you don’t have all of the information you require, there’s no way to “stash” your current iteration, and return to it later. This later could be after a logout and login (or even on a different computer).

Additionally, there’s no way to have “extra” steps inserted based on the selections made in a previous step. You could take the other approach, and define all of the steps initially (and then skip those that aren’t required), but in our case, we need a repeatable step (I call them a sub-step), where we perform an operation on a bunch of items that were selected in a previous step.

Let’s look at each of these in turn.

The default implementation automatically clears the storage whenever a GET request is handled:

class WizardView(TemplateView):

    def get(self, request, *args, **kwargs):
        self.storage.reset()
        self.storage.current_step = self.steps.first
        return self.render(self.get_form())

We can replace this with an implementation that will only reset when it detects a special GET parameter:

    def get(self, request, *args, **kwargs):
        if not request.GET.get('reset'):
            self.storage.reset()
            self.storage.current_step = self.steps.first
        
        step = self.storage.current_step or self.steps.first
        return self.render(self.get_form(
            step,
            data=self.storage.get_step_data(step),
            files=self.storage.get_step_files(step),
        ))

Not that we need to build up the form data to provide to the view - otherwise when you load up the wizard, it will render the first page as empty, but if you then use the navigation to select the first page it will correctly render it with data.

You’ll need to have an explicit link or button in your wizard template(s) to enable the user to restart the wizard if they need to.

Next up is persistent storage. For this we will need somewhere to store the data. The formtools implementation makes this easy to swap out - you can define the storage backend you want to use.

class MyWizard(WizardView):
    form_list = [...]
    storage_name = 'storage.wizard.DatabaseStorage'

That’s all you need to do to make your wizard use it - but now we need to build the storage class.

Let’s begin with a model to store the data in.

class WizardStorage(models.Model):
    user = models.ForeignKey('auth.User', related_name='wizard_storage', on_delete=models.CASCADE)
    prefix = models.TextField()
    data = models.JSONField()

We could use extra fields for the various parts, but that just complicates things.

Now let’s see a storage implementation:

class DatabaseStorage(BaseStorage):
    def __init__(self, prefix, request=None, file_storage=None):
        super().__init__(prefix, request, file_storage)
        self.init_data()
    
    def init_data(self):
        self.instance, _create = WizardStorage.objects.get_or_create(
            user=self.request.user,
            prefix=self.prefix,
            defaults={
                'data': {
                    self.step_key: None,
                    self.step_data_key: {},
                    self.step_files_key: {},
                    self.extra_data_key: {},
                }
            }
        )
    
    @property
    def data(self):
        return self.instance.data
    
    def update_response(self, response):
        if hasattr(self, 'instance'):
            self.instance.save()
      
    def reset(self):
        if hasattr(self, 'instance'):
            self.instance.delete()
            del self.instance

You can make it a bit more configurable so that the model could be swapped quite easily.

From here, we can use this in a wizard, and it will persist the step data to the database. I’ve still got a bit to do to ensure it can handle files, but this has not been a requirement of mine as yet.

There could be a bit of fun around having multiple stashes for a given wizard, and allowing the user to select which one they want to work on. As it stands, it just uses the user id and wizard prefix to determine where the data is stored.

The third improvement, allowing sub-steps, is a bit more complicated. To do that, you need to replace a bit more of the internal formtools code, rather than just subclassing/extending it. This involves a bunch of patching of the StepsHelper class from formtools - although you could replace this class by overriding WizardView.dispatch.

That’s beyond the scope of this post.

Django modify_settings and receivers

2021-03-31T18:47:08+00:00

Sometimes, tests expose weird behaviour.

In this instance, I have a Makefile command that calls a cookiecutter command to create a new integration with a payroll system. We’d noticed that over time, this infrequntly used code had become slightly less than perfect.

So, I wrote a test that runs this code. Not directly using the make new-payroll-system command, but rather using a test case from within Django.

This worked great in development. I was able to have code that was generated, some tests on that code were run (including generating migrations for the new app), and then the code was removed.

def test_cookie_cutter(self):
    from cookiecutter.main import cookiecutter
    
    try:
        cookiecutter(
            'integrations/__template__', 
            extra_context={'system_name': 'new_thing'}, 
            output_dir='integrations/systems',
            no_input=True,
        )
        with modify_settings(INSTALLED_APPS={'append': 'integrations.systems.new_thing'}):
            call_command('makemigrations', 'new_thing', no_input=True, verbosity=0)
    finally:
        shutil.rmtree('integrations/systems/new_thing')

But this failed in CI.

Turns out that our Codeship-based testing infrastructure doesn’t allow for writing the files in the expected location.

Never mind, we can use tempfile.TemporaryDirectory() instead. That will handle the cleanup for us, which is better than removing files ourselves:

def test_cookie_cutter(self):
    from cookiecutter.main import cookiecutter
    
    with tempfile.TemporaryDirectory() as dirname:
        # Put our new temporary directory on the PYTHONPATH.
        sys.path.insert(0, dirname)
        
        cookiecutter(
            'integrations/__template__', 
            extra_context={'system_name': 'new_thing'}, 
            output_dir=dirname,
            no_input=True,
        )
        with modify_settings(INSTALLED_APPS={'append': 'integrations.systems.new_thing'}):
            call_command('makemigrations', 'new_thing', no_input=True, verbosity=0)

There’s a little more to my code, but it’s not really relevant.

What is revelant is that, whilst this test was working fine, there is a subsequent test that was failing. Because, as part of the cookie-cutter template, we install a signal handler by default, and this signal handler was still connected even after the app was removed by the end of the modify_settings context manager.

So, how can we remove the signal handler when we are done? It should be possible to, in the context manager, look at which signals exist before we run, and then compare that to the ones that are connected when we are exiting…

Turns out, we don’t need to. In this context, because we aren’t really running the code from the new django app, we can just prevent the signal handlers being connected in the first place:

@patch('django.dispatch.receiver')
def test_cookie_cutter(self, _receiver):
    from cookiecutter.main import cookiecutter
    
    with tempfile.TemporaryDirectory() as dirname:
        # Put our new temporary directory on the PYTHONPATH.
        sys.path.insert(0, dirname)
        
        cookiecutter(
            'integrations/__template__', 
            extra_context={'system_name': 'new_thing'}, 
            output_dir=dirname,
            no_input=True,
        )
        with modify_settings(INSTALLED_APPS={'append': 'integrations.systems.new_thing'}):
            call_command('makemigrations', 'new_thing', no_input=True, verbosity=0)

That one change of mocking out the @receiver decorator means that it won’t attach the signals that it comes across when doing the modify_settings, which is exactly what I want to happen in this case.

Docker + Makefile

2021-02-12T21:27:42+00:00

I rewrote one of my projects (Fronius Dashboard) in Python - more because I was no longer able to get the build to work correctly under Elixir. As a side-effect, the image size went down a lot.

Part of this process is to build multiple-architectures, and publish these manifests. As an aside, being able to version them is also nice.

We’ll start with the versioning, because that’s a bit simpler.

Take a file VERSION. Put the current version into this file (x.y.z format is the only one supported so far).

Now, we can have a few tools to handle this:

.PHONY: bump-major bump-minor

requirements.txt: poetry.lock pyproject.toml
	poetry export -o requirements.txt

bump-major:
	cat VERSION | awk -F. '{print $$1 + 1 ".0.0"}' | tee VERSION
	
bump-minor:
	cat VERSION | awk -F. '{print $$1 "." $$2 + 1  ".0"}' | tee VERSION
	
VERSION: app.py requirements.txt
	cat VERSION | awk -F. '{print $$1 "." $$2 "." $$3 + 1}' | tee VERSION

This allows us to have make bump-major that adds one to the existing major version, and resets the minor and patch versions. And another version that adds one to the minor version, and resets the patch version.

There is no bump-patch, instead every file that could possibly affect the code is included in the make VERSION dependencies list. This means that make VERSION will only run if any files have changed, and in that case it will increment the patch version.

So, that’s the versioning. How about publishing docker images?

There are a couple of things that we need to do:

build for a number of platforms (amd64, armv6 and armv7, because I run stuff on Raspberry Pi hardware).
create and push a manifest of all images
also create and push a tagged version (ie, not just latest).

IMAGE := <image-name-goes-here>

.PHONY: release bump-major bump-minor
	
release: Dockerfile VERSION
	docker buildx build . -t $(IMAGE):armv6 --platform linux/arm/v6 --push
	docker buildx build . -t $(IMAGE):armv7 --platform linux/arm/v7 --push
	docker buildx build . -t $(IMAGE):amd64 --platform linux/amd64 --push
	
	docker pull $(IMAGE):armv6
	docker pull $(IMAGE):armv7
	docker pull $(IMAGE):amd64
	
	docker image rm --force $(IMAGE):latest
	
	docker manifest create $(IMAGE):latest \
		$(IMAGE):armv6 \
		$(IMAGE):armv7 \
		$(IMAGE):amd64 \
		--amend
	
	docker manifest annotate $(IMAGE):latest $(IMAGE):armv6 --variant v6l
	docker manifest annotate $(IMAGE):latest $(IMAGE):armv7 --variant v7l
	
	docker manifest create $(IMAGE):$(shell cat VERSION) \
		$(IMAGE):armv6 \
		$(IMAGE):armv7 \
		$(IMAGE):amd64 \
		--amend
	
	docker manifest annotate $(IMAGE):$(shell cat VERSION) $(IMAGE):armv6 --variant v6l
	docker manifest annotate $(IMAGE):$(shell cat VERSION) $(IMAGE):armv7 --variant v7l
	
	docker manifest push $(IMAGE):$(shell cat VERSION)
	docker manifest push $(IMAGE):latest

requirements.txt: poetry.lock pyproject.toml
	poetry export -o requirements.txt

bump-major:
	cat VERSION | awk -F. '{print $$1 + 1 ".0.0"}' | tee VERSION
	
bump-minor:
	cat VERSION | awk -F. '{print $$1 "." $$2 + 1  ".0"}' | tee VERSION
	
VERSION: app.py requirements.txt
	cat VERSION | awk -F. '{print $$1 "." $$2 "." $$3 + 1}' | tee VERSION

There is a bit of repetition - I’m sure I could do something using Makefile expansion, but this works for now.

Cleaning up a required process when you quit

2021-01-27T14:32:53+00:00

Some time ago, I moved a bunch of our devops tools into a Makefile. I’m not sure exactly what the benefit was then, but there have been some challenges.

One of these was related to having to run multiple processes when performing a task, but cleaning up the dependencies when we finish our task. Specifically, we run a websocket server in production that uses Redis for PubSub. We can safely assume the developer is running redis already, since the rest of the platform also depends upon it. However, to collect the changes from Postgres and publish them into Redis, we use a custom django management command. All this does is listen for postgres notifications and forward them to redis.

$ ./manage.py relay

Mostly we don’t keep this running in development, unless you need to be running the websocket code locally for testing. It is a hassle to remember to start a process and then kill it when you are done, so it would be convenient to start the relay command, then start our websocket worker, and clean up both processes when we quit the worker.

This can be done in a Makefile command:

websocket:
    $(shell trap 'kill 0' SIGINT ; \
        ./manage.py relay & \
        gunicon \
          -k geventwebsocket.gunicorn.workers.GeventWebsocketWorker \
          -workers=1 \
          -reload \
          webocket:application \
    )

The trap command performs the magic - it will kill the background command when the gunicorn worker is killed with Ctrl-C.

I have some other commands which start the background process, and then use kill -0 $(shell ps ax | grep ... | head -n 1 | cut -d ' ' -f 1) to ensure they kill the process on quit, but this sometimes does not work: I should get around to changing them over to this…

ESPHome variables from Home Assistant

2021-01-24T21:28:00+00:00

One of my key tenets of Home Automation is that as much stuff should be done locally as possible. Whilst with HomeAssistant, that means “in the local network”, I feel that “in the device” is even better. However, sometimes values should still be configurable (from within Home Assistant), but the device should still work even if there is not (and has never been) a connection to Home Assistant since boot.

For instance, I have a sensor light. The device contains a PIR sensor, with configurable delay, sensitivity and lux settings. This in turn drives a relay connected to a pair of PAR bulbs, and additionally has a green LED. This device usually only has the ESP8266 controlling if the thing should ever turn on at all, but I hacked it so that it exposes the PIR sensor, and allows triggering the relay.

But the delay between the motion no longer being detected and the light turning off should be configurable. I don’t want to have to re-flash the device if I decide to change this value, so I wanted to make it that the value will be fetched from Home Assistant (if it is set).

This turned out to work really well. There are a few parts that need to be set up in the YAML (this is not the complete file):

esphome:
  name: ${device_name}
  platform: ESP8266
  board: esp01_1m
  on_boot:
    - logger.log:
        level: DEBUG
        format: 'Light ON time is set to %d'
        args: ['id(light_on_time)']

globals:
  - id: light_on_time
    type: int
    restore_value: true
    initial_value: '30'

sensor:
  - platform: homeassistant
    id: on_time
    entity_id: input_number.${device_name}
    on_value:
      then:
        - globals.set:
            id: light_on_time
            value: !lambda 'return int(x);'

It really is that simple. The first boot will set the light_on_time variable to 30. Then, when it connects to Home Assistant, it will look for an input_number.<device_name> (which matches the device name). If it finds one (or is ever told about this value changing), then it will commit that new value to the flash, and this will be be restored after a reboot.

There is one other thing we could do here, to make it so that we don’t write the value to the flash if it has not changed (and prevent wear to that, since it is limited to a number of writes):

sensor:
  - platform: homeassistant
    id: on_time
    entity_id: input_number.${device_name}
    on_value:
      then:
        if:
          condition:
            lambda: 'return id(light_on_time) != int(x);'
          then:
            - logger.log:
                level: DEBUG
                format: 'Light ON time changed from %d to %d seconds'
                args: ['id(light_on_time)', 'int(x)']
            - globals.set:
                id: light_on_time
                value: !lambda 'return int(x);'

With regards to a similar problem, detecting if it is dark enough to turn the light on should be something like “a bit before sunset to a bit after sunrise”. I could set the lux threshold on the device, but it would be nice to have motion detection work during the day too.

Here, we can have a global variable sun_is_down that defaults to being true, and is only set to false when the sun comes up. However, we would also want this to trigger when first connecting to Home Assistant and the sun is already up.

We can use a wait_until in our on_boot handler to trigger this:

globals:
  - id: sun_is_down
    type: bool
    restore_value: false
    initial_value: 'true'

esphome:
  ...
  on_boot:
    - wait_until:
        api.connected:
    - if:
        condition:
          sun.is_above_horizon:
        then:
          - logger.log:
              level: DEBUG
              format: 'Sun is up, light will not turn on.'
          - globals.set:
              id: sun_is_down
              value: 'false'
        else:
          - logger.log:
              level: DEBUG
              format: 'Sun is down, light will turn on.'

sun:
  on_sunrise:
    - elevation: 10°
      then:
        - logger.log:
            format: 'The sun is now down, light will turn on.'
        - globals.set:
            id: sun_is_down
            value: 'false'
  on_sunset:
    - elevation: 10°
      then:
        - logger.log:
            format: 'The sun is now up, light will not turn on.'
        - globals.set:
            id: sun_is_down
            value: 'true'

I’ve used a 10° elevation to trigger the light to turn on a bit before sunset and still turn on a bit after sunrise when motion is detected. I haven’t figured out a way to get the same check to apply in the sun.is_above_horizon: check.

Mute the rest of this track.

2021-01-22T20:43:18+00:00

Working from home means that I don’t use my AirPods as much as I used to: I’m happy to mostly listen to the music my partner (who has worked from home for over a decade) wants to listen to. Lately, that has been NonStop80s.

We play this through the IKEA Symphonisk stereo pair in the dining room, several metres away from the office. The sound is good, and it’s nice not having it come out of the computer speaker.

Because I’m a nut, I have all of this hooked up to Home Assistant, and have one nice automation that will turn off the music that is playing on that speaker when the TV in that room is turned on.

Sometimes the tracks that are played on NonStop80s are really bad. For instance, Snooker Loopy, by Chas and Dave is an absolute piece of shit song. If it weren’t a streaming station, I’d just hit skip.

What would be nice is to be able to have the speakers stop playing, and start playing again when the next track comes on.

Enter “Mute the rest of this track”.

alias: Mute the rest of this track
sequence:
  - condition: state
    entity_id: media_player.dining_room_sonos
    state: playing
  - condition: template
    value_template: ''
  - service: media_player.volume_mute
    data:
      is_volume_muted: true
    entity_id: media_player.dining_room_sonos
  - delay: '1'
  - wait_for_trigger:
      - platform: state
        entity_id: media_player.dining_room_sonos
    timeout: ''
  - service: media_player.volume_mute
    data:
      is_volume_muted: false
    entity_id: media_player.dining_room_sonos
mode: single
icon: 'mdi:volume-mute'
description: Mute the media player until the media_title attribute changes.
fields:
  timeout:
    description: Amount of time to stay muted if the track does not change
    example: 300

This is a script that you can use in Home Assistant to do just that. It will (iff the dining_room_sonos is unmuted, and currently playing), mute that speaker until something else on the speaker changes.

This could be a track change, or a volume change, or anything else. It’s deliberately loose, so that if it was muted, and someone changes the volume, it unmutes immediately.

After a configurable (default 300 seconds) timeout, it will unmute.

I was trying to make it so that you could provide the named media_player entity, but there still seem to be places where you can’t use templates (but need to provide the string directly). I was able to get the conditions to use the variable, but was not able to figure out how to use them in the service calls…and to be honest, I don’t really need it to apply to other players.

Scheduling Celery Tasks in the (far) future

2021-01-19T19:59:47+00:00

We used to make use of the fact that a celery task can be scheduled at some time in the future to auto-punch-out staff members who failed to punch out 24 hours after their shift started. This was as simple as scheduling a task with an eta=86400.

However, as Adam points out here (see number 5). This is not a great idea. For one, it will bog down your celery workers (and make them use a bunch more memory). Secondly, something Adam doesn’t mention is that if your queue is corrupted, then all of those future tasks will not be executed.

Discussing this in IRC today, I thought of a simple mechanism for scheduling these tasks and processing them at some point after they are due.

We will start with a model to store our task:

class ScheduledTask(models.Model):
    task_name = models.TextField()
    task_args = models.JSONField(default=list)
    task_kwargs = models.JSONField(default=dict)
    due_date = models.DateTimeField()

    objects = ScheduledTaskQuerySet.as_manager()

    @property
    def task(self):
        module, task = task_name.rsplit('.', 1)
        return getattr(importlib.import_module(module), task)

    def execute(self):
        self.task.apply_async(args=self.args, kwargs=self.kwargs)

We have a custom queryset defined, that allows us to see which tasks are due using a nice queryset method:

class ScheduledTaskQuerySet(models.query.QuerySet):
    def due(self):
        return self.due_date < datetime.datetime.utcnow()

    def schedule(self, task, when, *args, **kwargs):
        self.create(task_name=task_name, task_args=args, task_kwargs=kwargs, due_date=when)

Finally, we need a task that will enqueue the due tasks:

@app.task
def enqueue_due_tasks():
    for task in ScheduledTask.objects.due():
        task.execute()
        task.delete()

As it stands, with this code, to schedule a task, you need to create a model instance:

ScheduledTask.objects.schedule(
    'foo.tasks.bar',
    datetime.datetime(2525, 11, 11, 9, 30),
    'x',
    y='z'
)

But, it would be nice if we could use the task to schedule itself:

foo.tasks.bar.schedule(args=[], kwargs={}, eta=X, countdown=Y)

Or, even better:

foo.tasks.bar.s(arg, kwarg=value).schedule(datetime.timedelta(days=365))

The first one we should be able to do by using custom tasks (and implementing a schedule method):

class Scheduled(celery.Task):
    def schedule(self, *, args=None, kwargs=None, eta=None, countdown=None):
        if not eta and not countdown:
            raise ValueError('One of eta and countdown must be supplied')
        if eta and countdown:
            raise ValueError('Only one of eta and countdown must be supplied')
        if eta:
            ScheduledTask.objects.schedule(self.name, eta, *args, **kwargs)
        else:
            ScheduledTask.objects.schedule(
                self.name,
                datetime.datetime.utcnow() + datetime.timedelta(countdown),
                *args, **kwargs
            )

Then, as long as a task is defined as using the base class, we can schedule it:

@app.task(base=Schedule)
def task_name(x, y=None):
    pass

But what about mucking with the .s() or .signature() calls? Now we are getting a bit experimental, but it still might be fun:

from celery.canvas import Signature

def schedule(self, when=None):
    if when:
        if isinstance(when, datetime.timedelta):
            when = datetime.datetime.utcnow() + when
    else:
        if self.options.countdown:
            when = datetime.datetime.utcnow() + datetime.timedelta(seconds=self.options.countdown)
        elif self.options.eta:
            when = self.otions.eta

    ScheduledTask.objects.create(
        task_name=self.task,
        task_args=self.args,
        task_kwargs=self.kwargs,
        due_date=when,
    )

Signature.schedule = schedule

This is ~~totally~~ mostly untested, and I’m not sure it’s a really good idea.

A better idea might be to have a backend that doesn’t even tell the workers about the tasks until after they are due…that way you would not have to duck-punch celery at all.

Logging WiFi client Signal Strength

2021-01-19T18:58:47+00:00

One of the cool things about the old macOS version of Airport Utility is that it would show you the WiFi signal of the various connected clients. At some point, they removed this feature, bringing it down to where the iOS app is in terms of functionality.

Whilst I still have a Mac that can (only) run that version next to me on the desk, mostly I don’t use that, and having to switch computers and open it up is a pain. It would be really nice if I could get that data in a more configurable format (possibly with the client data from multiple base stations).

It turns out that this data is available over SNMP, but you need the old version of Airport Utility to enable it. And, as we shall see shortly, there is another caveat too.

Before we get to that though, my main router is actually an OpenWRT device, so it might be nice to grab the handful of clients that connect to that (it being in the garage and all).

We can get a list of the WiFi interfaces using iwinfo | awk '/ESSID/ {print $1}'. In my case, that gives me wlan0 and wlan1, but the values are not relevant, other than the fact we will need them later.

Now, for each each WiFi interface, we want to get the MAC address, which we can get similarly: iwinfo $iw assoclist | awk '/Access Point:/ {print $3}.

It might be nice to also get the “connection speed”, although in some cases this is not known. In those cases, we want to use null, since we’ll have a numeric value otherwise.

  iwinfo $iw assoclist | awk '/expected/ {if ($3 == "unknown") print "null"
  else
   print $3
  }'

Okay, now we are ready to get the signal/noise/MAC data:

  iwinfo $iw assoclist \
    | awk -v rate=$RATE -v mac=$MAC '/dBm/ { \
      print "mosquitto_pub -t \"wifi-signal/" substr($1, 1) "\"", \
      " -m \x27{ \
        \"signal\":" $2 ", \
        \"noise\":" $5 ", \
        \"rate\":" rate ", \
        \"station\":\"" mac "\" \
      }\x27 -h mqtt.lan"}' \
    | bash

Oh, there’s a bit in there, we might unpack that a bit:

We get the list of associated clients, and then pipe that through awk. To make it easier, we want to stick our previous environment variable captures of our rate and MAC into awk variables, and then we grab the relevant columns, and build it all up into a shell command that will look something like:

mosquitto_pub -t "wifi-signal/98:AF:65:E3:16:F1"  -m '{
        "signal":-70,
        "noise":-102,
        "rate":null,
        "station":"B0:95:75:E8:C8:32"
      }' -h mqtt.lan

Because awk has no way to escape a single quote, we need to use \x27 to get it to emit one of them.

Finally, this is passed to bash to execute the command(s).

And, putting this all together:

#! /bin/sh

for iw in $(iwinfo | awk '/ESSID/ {print $1}')
do
  MAC=$(iwinfo $iw info | awk '/Access Point:/ {print $3}')
  RATE=$(iwinfo $iw assoclist | awk '/expected/ {if ($3 == "unknown") print "null"
else
   print $3
 }')
  iwinfo $iw assoclist \
    | awk -v rate=$RATE -v mac=$MAC '/dBm/ { \
      print "mosquitto_pub -t \"wifi-signal/" substr($1, 1) "\"", \
      " -m \x27{ \
        \"signal\":" $2 ", \
        \"noise\":" $5 ", \
        \"rate\":" rate ", \
        \"station\":\"" mac "\" \
      }\x27 -h mqtt.lan"}' \
    | bash
done

Okay, so that’s the easy one. What about getting the data from SNMP?

Since this code is running not on a Mac, I can’t rely on the MIB data being available. Instead, we’ll use the raw numeric codes. It turns out that we can get all of the data we need (and then some!) using one command:

snmpwalk -v 2c -c public hostname.local iso.3.6.1.4.1.63.501.3.2.2.1

This outputs a whole bunch of data - the ones we care about have the values 1, 6, 7 and 8 in position 13, so we can extract those values. But, we want to retain a bit more context, so we can sort the data and ensure we have a consistent ordering to our data: MAC address, signal, noise and then rate for each client. To ensure we have a stable sorting, we need to get the last two parts of the MIB (which seemed to be enough in my case to generate a unique value), and then the 13th column. Then we want to get the “value” that was returned for this row. If it was a STRING value, then it will already have quotes around it.

  snmpwalk -v 2c -c public $station $WIFI_DATA \
    | awk -F. 'match($31, /: /) {print $30, substr($31, 0, 3), $13, substr($31, RSTART + 2)}' \
    | sort \
    | grep '^.. .. [1678] '

This will build up the data in the correct format, from here we now need to put each client’s data on a single line:

  snmpwalk ...
    | <as-above>
    | awk '/^.. .. 1/{ if (NR!=1) print "";}{printf $4 " "}'

And then turn it into a string that we’ll be able to execute with bash (like before). Putting it all together:

  MAC=$(/usr/sbin/arp -a $station | awk '{print $4}')

  snmpwalk -v 2c -c public $station iso.3.6.1.4.1.63.501.3.2.2.1 \
    | awk -F. 'match($31, /: /) {print $30, substr($31, 0, 3), $13, substr($31, RSTART + 2)}' \
    | sort \
    | grep '^.. .. [1678] ' \
    | awk '/^.. .. 1/ {if (NR!=1) print "";}{printf $4 " "}' \
    | awk -v mac=$MAC '{if ($2 != "") \
      print "mosquitto_pub -t \"wifi-signal/" substr($1, 2), \
        "-m \x27{\
          \"signal\":" $2 ", \
          \"noise\":" $3 ", \
          \"rate\":" $4 ", \
          \"station\": \"" mac "\" \
    }\x27 -h mqtt.lan" \
    }' | bash

Note that I also needed to check that there was a signal value - one of the stations was reporting this as empty, so we skip over any that don’t have a signal value reported.

But… this doesn’t quite work. I mean, it works, but for some reason (after having left it running every minute overnight), it never updates the signal/noise/rate data. It seems that the Airport Express/Extreme SNMP data is not refreshed.

You can, however, force it to update by asking for the number of connected clients!

snmpget -v 2c -c public $station iso.3.6.1.4.1.63.501.3.2.1.0

Thus, our full script can look like:

#! /bin/bash

stations="study.lan
bedroom.lan
dining-room.lan"

CLIENT_COUNT=iso.1.3.6.1.4.1.63.501.3.2.1.0
WIFI_DATA=iso.3.6.1.4.1.63.501.3.2.2.1

for station in ${stations}
do
  MAC=$(/usr/sbin/arp -a $station | awk '{print $4}')

  snmpget -v 2c -c public $station $CLIENT_COUNT > /dev/null

  snmpwalk -v 2c -c public $station $WIFI_DATA \
    | awk -F. 'match($31, /: /) {print $30, substr($31, 0, 3), $13, substr($31, RSTART + 2)}' \
    | sort \
    | grep '^.. .. [1678] ' \
    | awk '/^.. .. 1/ {if (NR!=1) print "";}{printf $4 " "}' \
    | awk -v mac=$MAC '{if ($2 != "") \
      print "mosquitto_pub -t \"wifi-signal/" substr($1, 2), \
        "-m \x27{\
          \"signal\":" $2 ", \
          \"noise\":" $3 ", \
          \"rate\":" $4 ", \
          \"station\": \"" mac "\" \
    }\x27 -h mqtt.lan" \
    }' | bash
done

These scripts could be more configurable - I’ve hard-coded the MQTT broker name, for instance, but that will be fine for my purposes. I’m also not actually doing anything with this data yet - I will eventually look at pulling this into Home Assistant or somewhere else, and graphing the data.

Locking rows in PostgreSQL

2021-01-13T13:37:53+00:00

Someone asked a question today in #postgresql about locking rows for a long running task, where multiple tasks could be updating rows. There is a flag in each row that indicates whether a row has been processed, but not one indicating if it is currently being processed.

One solution is to use advisory locks to mark that we are currently working on a given row. The advantage of this over having a column that indicates it is currently being processed (or mis-using the “processed” column by setting it before we’ve finished building the data) is that if our task that generates the data fails, the lock is released when the session is closed.

So, let’s look at the things we need to make this happen. We’ll start with something that represents our long-running process:

CREATE OR REPLACE FUNCTION expensive_operation(INTEGER)
RETURNS JSONB AS $$

  BEGIN
    PERFORM pg_sleep(10);
    RETURN JSONB_BUILD_OBJECT(
        'foo', 'bar',
        'baz', $1
    );
  END;

$$ LANGUAGE plpgsql;

And we need our table to put the tasks in (and some unprocessed rows):

CREATE TABLE tasks (
    id SERIAL PRIMARY KEY,
    processed BOOLEAN DEFAULT FALSE,
    data JSONB
);

INSERT INTO tasks (data) VALUES (NULL), (NULL), (NULL);

Finally, we are ready to process our tasks.

WITH next_task AS (
  SELECT t.id, pg_advisory_lock(t.id)
    FROM (
      SELECT id
        FROM tasks
       WHERE NOT processed
         AND id NOT IN (SELECT objid
                          FROM pg_locks
                         WHERE locktype = 'advisory')
       LIMIT 1
    ) t
)
UPDATE tasks
   SET data = expensive_operation(id),
       processed = true
 WHERE id IN (SELECT id FROM next_task)
   AND NOT processed;

The CTE will get a task.id that has not been processed (and is not currently being processed). It will take an advisory lock on that value, which marks the task as in-progress.

Then, when it has the lock, it performs the expensive operation and updates the data (and the processed status).

I’m not totally sure if the AND NOT processed at the bottom of the query is necessary - it seems to me that there could be a situation where two workers try to get the lock at the same time (and therefore are both asking for the same lock), and one of them gets it. The other one, when it finally gets the lock, would now see that row as processed, and so would not update anything.

I’m happy to see a better way to do this.

RobotFramework, Chromedriver and Docker

2020-12-18T23:42:26+00:00

One of my team implemented RobotFramework support for automated browser testing of our platform a while ago. At the time, we were using Codeship Basic, and I built a helper to run a robot test suite within a tox environment. It was all good, because chromedriver and all it’s dependencies were already installed.

But time passes, and we needed to move to Codeship Pro. Which has some neater features, but required me to build docker images for everything. We already use docker for deployment, but I didn’t really want to build a bunch of distinct images just for testing that re-implemented the same stuff that we have in our deployment images. Even just appending new stuff to them means that things could turn out to be a pain in the arse to manage.

And getting chromedriver installed into a docker image is not neat.

I did find a docker image that just has an instance of chromedriver, and exposes that. But getting that to work with robot was still a bunch of work. After much experimentation, I was able to get the connections between everything to work.

First, we need to have the chromedriver container running:

$ docker run -p 4444:4444 CHROMEDRIVER_WHITELISTED_IPS='' robcherry/docker-chromedriver:latest

Then, there are a few moving parts that need to be in place to get things to work. Using my djangobot management command (which I had to extend a bit here), a single command can be used to spin up a Django runserver command, apply migrations (if necessary), and then run the robot commands. The trick is you need to teach Robot to speak to the remote WebDriver instance, which then in turn speaks to the running django webserver.

First, the RobotFramework commands; my resource.robot file which is referenced by all of my robot test suites contains:

*** Variables ***

${HOSTNAME}         127.0.0.1
${PORT}             8000
${SCHEME}           http
${SERVER}           ${SCHEME}://${HOSTNAME}:${PORT}
${BROWSER}          headlesschrome
${TIMEOUT}          30
${REMOTE_URL}

*** Settings ***

Documentation   A resource file with reusable keywords and variables.
Library         SeleniumLibrary             timeout=${TIMEOUT}      implicit_wait=1
Library         Collections
Library         DebugLibrary
Library         DateTime
Library         String
Library         djangobot.DjangoLibrary     ${HOSTNAME}     ${PORT}

*** Keywords ***

Create Remote Webdriver
    ${chrome_options} =     Evaluate    sys.modules['selenium.webdriver'].ChromeOptions()    sys, selenium.webdriver
    Call Method    ${chrome_options}   add_argument    headless
    Call Method    ${chrome_options}   add_argument    disable-gpu
    Call Method    ${chrome_options}   add_argument    no-sandbox
    ${options}=     Call Method     ${chrome_options}    to_capabilities

    Create Webdriver    Remote   command_executor=${REMOTE_URL}    desired_capabilities=${options}
    Open Browser    ${SERVER}   ${BROWSER}  remote_url=${REMOTE_URL}    desired_capabilities=${options}

Start Session
    Run Keyword If  '${REMOTE_URL}'    Create Remote Webdriver
    Run Keyword If  '${REMOTE_URL}' == ''    Open Browser    ${SERVER}   ${BROWSER}

    Set Window Size     2048  2048
    Fetch Url       login
    Add Cookie      robot   true

    Register Keyword To Run On Failure    djangobot.DjangoLibrary.Dump Error Data

End Session
    Close Browser

Logout
    Fetch Url     logout

Notice that the Start Session keyword determines which type of browser to open - either a local or remote one.

Thus, each *.robot file starts with:

*** Settings ***
Resource                resource.robot
Suite Setup             Start Session
Suite Teardown          End Session
Test Setup              Logout

Because the requests will no longer be coming from localhost, you need to ensure that your runserver is listening on the interface the requests will be coming from. If you can’t detect this, and your machine is not exposed to an insecure network, then you can use 0.0.0.0 to get the django devserver to listen on all interfaces. You will also need to supply the hostname that you will be using for the requests (which won’t be localhost anymore), and ensure this is in your Django settings.ALLOWED_HOSTS.

In my case, I needed to make my robot command allow all this, but ultimately I can now do:

$ ./manage.py robot --runserver 0 \
                    --listen 0.0.0.0 \
                    --hostname mymachine.local \
                    --remote-url http://localhost:4444 \
                    --include tag

This runs against the database I already have prepared, but in my codeship-steps.yml I needed to do a bit more, and hook it up to the other containers:

coverage run --branch --parallel \
    /app/manage.py robot --migrate \
                         --server-url=http://web.8000 \
                         --remote-url=http://chromedriver:4444 \
                         --tests-dir=/app/robot_tests/ --output-dir=/coverage/robot_results/ \
                         --exclude skip  --exclude expected-failure

Now, if only Codeship’s jet tool actually cached multi-stage builds correctly.

Maybe I neeed to try this.

Update value only if present

2020-12-18T22:44:41+00:00

We have a bunch of integrations with external systems, and in most of these cases we are unable to use Oauth, or other mechanisms that don’t require us to store a username password pair. So, we have to store that information (encrypted, because we need to use the value, rather than just being able to store a hashed value to compare an incoming value with).

Because this data is sensitive, we do not want to show this value to the user, but we do need to allow them to change it. As such, we end up with a form that usually contains a username and a password field, and sometimes a URL field:

class ConfigForm(forms.ModelForm):
    class Meta:
        model = ExternalSystem
        fields = ('username', 'password', 'url')

But this would show the password to the user. We don’t want to do that, but we do want to allow them to include a new password if it has changed.

In the past, I’ve done this on a per-form basis by overridding the clean_password method:

class ConfigForm(forms.ModelForm):
    class Meta:
        model = ExternalSystem
        fields = ('username', 'password', 'url')

    def clean_password(self):
        return self.cleaned_data.get('password') or self.instance.password

But this requires implementing that method on every form. As I mentioned before, we have a bunch of these. And on at least one, we’d missed this method. We could subclass a base form class that implements this method, but I think there is a nicer way.

It should be possible to have a field that handles this. The methods that look interesting are clean, and has_changed. Specifically, it would be great if we could just override has_changed:

class WriteOnlyField(forms.CharField):
    def has_changed(self, initial, data):
        return bool(data) and initial != data

However, it turns out this is not used until the form is re-rendered (or perhaps not at all by default, it’s very likely my code calls this to get a list of changed fields to mark as changed as a UI affordance).

The clean method in a CharField does not have access to the initial value, and there really is not a nice way to get this value attached to the field (other than doing it in the has_changed method, which is not called).

But it turns out this behaviour (apply changes only when a value is supplied) is the same behaviour that is used by FileField: and as such, it gets a special if statement in the form cleaning process, and is passed both the initial and the new values.

So, we can leverage this and get a field class that does what we want:

class WriteOnlyField(forms.CharField, forms.FileField):
    def clean(self, value, initial):
        return value or initial

    def has_changed(self, initial, data):
        return bool(data) and initial != data

We can even go a bit further, and rely on the behaviour of forms.PasswordInput() to hide the value on an unbound form:

class WriteOnlyField(forms.CharField, forms.FileField):
    def __init__(self, *args, **kwargs):
        defaults = {
            'widget': forms.PasswordInput(),
            'help_text': _('Leave blank if unchanged'),
        }
        defaults.update(**kwargs)
        return super().__init__(*args, **defaults)

    def clean(self, value, initial):
        return value or initial

    def has_changed(self, initial, data):
        return bool(data) and initial != data

Then we just need to override that field on our form definition:

class ConfigForm(forms.ModelForm):
    password = WriteOnlyField()

    class Meta:
        model = ExternalSystem
        fields = ('username', 'password', 'url')

Please note that this technique should not be used in the situation where you don’t need the user to be able to change a value, but instead just want to render the value. In that case, please omit the field from the form, and just use `` instead - you can even put that in a disabled text input widget if you really want it to look like the other fields.

I also use a JavaScript affordance on all password fields that default to hiding the value, but allows clicking on a control to toggle the visibility of the value: UIkit Password Field.

Preventing Model Overwrites in Django and Postgres

2020-09-23T19:39:15+00:00

I had an idea tonight while helping someone in #django. It revolved around using a postgres trigger to prevent overwrites with stale data.

Consider the following model:

class Person(models.Model):
    first_name = models.TextField()
    last_name = models.TextField()

If we had two users attempting to update a given instance at around the same time, Django would fetch whatever it had in the database when they did the GET request to fetch the form, and display that to them. It would also use whatever they sent back to save the object. In that case, the last update wins. Sometimes, this is what is required, but it does mean that one user’s changes would be completely overwritten, even if they had only changed something that the subsequent user did not change.

There are a couple of solutions to this problem. One is to use something like django-model-utils FieldTracker to record which fields have been changed, and only write those back using instance.save(update_fields=...). If you are using a django Form (and you probably should be), then you can also inspect form.changed_data to see what fields have changed.

However, that may not always be the best behaviour. Another solution would be to refuse to save something that had changed since they initially fetched the object, and instead show them the changes, allow them to update to whatever it should be now, and then resubmit. After which time, someone else may have made changes, but then the process repeats.

But how can we know that the object has changed?

One solution could be to use a trigger (and an extra column).

class Person(models.Model):
    first_name = models.TextField()
    last_name = models.TextField()
    _last_state = models.UUIDField()

And in our database trigger:

CREATE EXTENSION "uuid-ossp";

CREATE OR REPLACE FUNCTION prevent_clobbering()
RETURNS TRIGGER AS $prevent_clobbering$

BEGIN
  IF NEW._last_state != OLD._last_state THEN
    RAISE EXCEPTION 'Object was changed';
  END IF;
  NEW._last_state = uuid_generate_v4();
  RETURN NEW;
END;

$prevent_clobbering$
LANGUAGE plpgsql STRICT IMMUTABLE;

CREATE TRIGGER prevent_clobbering
BEFORE UPDATE ON person_person
FOR EACH ROW EXECUTE PROCEDURE prevent_clobbering();

You’d also want to have some level of handling in Django to capture the exception, and re-display the form. You can’t use the form/model validation handling for this, as it needs to happen during the save.

To make this work would also require the _last_state column to have a DEFAULT uuid_generate_v4(), so that newly created rows would get a value.

This is only a prototype at this stage, but does work as a mechanism for preventing overwrites. As usual, there’s probably more work in the application server, and indeed in the UI that would need to be required for displaying stale/updated values.

What this does have going for it is that it’s happening at the database level. There is no way that an update could happen (unless the request coming in happened to guess what the new UUID was going to be).

What about drawbacks? Well, there is a bit more storage in the UUID, and we need to regenerate a new one each time we save a row. We could have something that checks the other rows looking for changes.

Perhaps we could even have the hash of the previous row’s value stored in this field - that way it would not matter that there had been N changes, what matters is the value the user saw before they entered their changes.

Another drawback is that it’s hard-coded to a specific column. We could rewrite the function to allow defining the column when we create the trigger:

CREATE TRIGGER prevent_clobbering
BEFORE UPDATE ON person_person
FOR EACH ROW EXECUTE PROCEDURE prevent_clobbering('_last_state_');

But that requires a bit more work in the function itself:

CREATE OR REPLACE FUNCTION prevent_clobbering()
RETURNS TRIGGER AS $prevent_clobbering$

BEGIN
  IF to_jsonb(NEW)->TG_ARGV[0] != to_jsonb(OLD)->TG_ARGV[0] THEN
    RAISE EXCEPTION 'Object was changed';
  END IF;
  NEW._last_state = uuid_generate_v4();
  RETURN NEW;
END;

$prevent_clobbering$
LANGUAGE plpgsql STRICT IMMUTABLE;

Django properties from expressions, or ComputedField part 2

2020-09-15T14:21:42+00:00

I’ve discussed the concept of a ComputedField in the past. On the weekend, a friend pointed me towards SQL Alchemy’s Hybrid Attributes. The main difference here is that in a ComputedField, the calculation is always done in the database. Thus, if a change is made to the model instance (and it is not yet saved), then the ComputedField will not change it’s value. Let’s look at an example from that original post:

class Person(models.Model):
    first_name = models.TextField()
    last_name = models.TextField()
    display_name = ComputedField(
        Concat(F('first_name'), Value(' '), F('last_name')),
        output_field=models.TextField()
    )

We can use this to query, or as an attribute:

Person.objects.filter(display_name__startswith='foo')
Person.objects.first().display_name

But, if we make changes, we don’t see them until we re-query:

person = Person(first_name='Fred', last_name='Jones')
person.display_name  # This is not set

So, it got me thinking. Is it possible to turn a django ORM expression into python code that can execute and have the same output?

And, perhaps the syntax SQL Alchemy uses is nicer?

class Person(models.Model):
    first_name = models.TextField()
    last_name = models.TextField()

    @shared_property
    def display_name(self):
        return Concat(
            F('first_name'),
            Value(' '),
            F('last_name'),
            output_field=models.TextField(),
        )

The advantage to using the decorator approach is that you could have a more complex expression - but perhaps that is actually a disadvantage. It might be nice to ensure that the code can be turned into a python function, after all.

The first step is to get the expression we need to convert to a python function. Writing a python decorator will give us access to the “function” object - we can just call this, as long as it does not refer to self at all, this can be done without an instance:

class shared_property(object):
    def __init__(self, function):
        expression = function(None)

This gives us the expression object. Because this is a python object, we can just look at it directly, and turn that into an AST. Having a class for parsing this makes things a bit simpler. Let’s look at a parser that can handle this expression.

import ast


class Parser:
    def __init__(self, function):
        # Make a copy, in case this expression is used elsewhere, and we change it.
        expression = function(None).copy()
        tree = self.build_expression(expression)
        # Need to turn this into code...
        self.code = compile(tree, mode='eval', filename=function.func_code.co_filename)

    def build_expression(self, expression):
        # Dynamically find the method we need to call to handle this expression.
        return getattr(self, 'handle_{}'.format(expression.__class__.__name__.lower()))(expression)

    def handle_concat(self, concat):
        # A Concat() contains only one source expression: ConcatPair().
        return self.build_expression(*concat.get_source_expressions())

    def handle_concatpair(self, pair):
        left, right = pair.get_source_expressions()
        return ast.BinOp(
            left=self.build_expression(left),
            op=ast.Add(),
            right=self.build_expression(right),
        )

    def handle_f(self, f):
        # Probably some more work here around transforms/lookups...
        # Set this, because without it we get errors. Will have to
        # figure out a better way to handle this later...
        f.contains_aggregate = False
        return ast.Attribute(
            value=ast.Name(id='self'),
            attr=f.name,
        )

    def handle_value(self, value):
        if value.value is None:
            return ast.Name(id='None')

        if isinstance(value.value, (str, unicode)):
            return ast.Str(s=value.value)

        if isinstance(value.value, (int, float)):
            return ast.Num(n=value.value)

        if isinstance(value.value, bool):
            return ast.Name(id=str(value.value))

        # ... others?
        raise ValueError('Unable to handle {}'.format(value))

There’s a bit more “noise” required in there (every node must have a ctx, and a filename, lineno and col_offset), but they make it a bit harder to follow.

So, we have our expression, and we have turned that into an equivalent python expression, and compiled it…except it won’t compile. We need to wrap it in an ast.Expression(), and then we can compile it (and call it).

Roughly, we’ll end up with a code object that does:

self.first_name + (' ' + self.last_name)

We can call this with our context set:

eval(code, {'self': instance})

But, before we head down that route (I did, but you don’t need to), it’s worth noticing that not all ORM expressions can be mapped directly onto a single python expression. For instance, if we added an optional preferred_name field to our model, our display_name expression may look like:

@shared_property
def display_name(self):
    return Case(
        When(preferred_name__isnull=True, then=Concat(F('first_name'), Value(' '), F('last_name'))),
        When(preferred_name__exact=Value(''), then=Concat(F('first_name'), Value(' '), F('last_name'))),
        default=Concat(F('first_name'), Value(' ('), F('preferred_name'), Value(') ') F('last_name')),
        output_field=models.TextField()
    )

Since this will roughly translate to:

@property
  def display_name(self):
      if all([self.preferred_name is None]):
          return self.first_name + ' ' + self.last_name
      elif all([self.preferred_name == '']):
          return self.first_name + ' ' + self.last_name
      else:
          return self.first_name + ' (' + self.preferred_name + ') ' + self.last_name

Whilst this is still a single ast node, it is not an expression (and cannot easily be turned into an expression - although in this case we could use a dict lookup based on self.preferred_name, but that’s not always going to work). Instead, we’ll need to change our code to generate a statement that contains a function definition, and then evaluate that to get the function object in the context. Then, we’ll have a callable that we can call with our model instance to get our result.

There are a few hitches along the way though. The first is turning our method into both a private field and a property. That is the relatively straightforward part:

class shared_property:
    def __init__(self, function):
        self.parsed = Parser(function)
        context = {}
        eval(self.parsed.code, context)
        self.callable = context[function.func_code.co_name]

    def __get__(self, instance, cls=None):
        # Magic Descriptor method: this method will be called when this property
        # is accessed on the instance.
        if instance is None:
            return self
        return self.callable(instance)

    def contribute_to_class(self, cls, name, private_only=False):
        # Magic Django method: this is called by django on class instantiaton, and allows
        # us to add our field (and ourself) to the model. Mostly this is the same as
        # a normal Django Field class would do, with the exception of setting concrete
        # to false, and using the output_field instead of ourself.
        field = self.parsed.expression.output_field
        field.set_attributes_from_name(name)
        field.model = cls
        field.concrete = False
        # This next line is important - it's the key to having everything work when querying.
        field.cached_col = ExpressionCol(self.parsed.expression)
        cls._meta.add_field(field, private=True)
        if not getattr(cls, field.attname, None):
            setattr(cls, field.attname, self)

There are a few things to note in that last method.

We use the output_field from the expression as the added field.
We mark this field as a private, non-concrete field. This prevents django from writing it back to the database, but it also means it will not appear in a .values() unless we explicitly ask for it. That’s actually fine, because we want the python property to execute instead of just using the value the database gave us.
The cached_col attribute is used when generating queries - we’ll look more at that now.

When I previously wrote the ComputedField implementation, the place I was not happy was with the get_col() method/the cached_col attribute. Indeed, to get that to work, I needed to use inspect to sniff up the stack to find a query instance to resolve the expression.

This time around though, I took a different approach. I was not able to use the regular resolve_expression path, because fields are assumed not to require access to the query to resolve to a Col expression. Instead, we can delay the resolve until we have something that gives us the query object.

class ExpressionCol:
    contains_aggregate = False
    def __init__(self, expression):
        self.expression = expression
        self.output_field = expression.output_field

    def get_lookup(self, name):
        return self.output_field.get_lookup(name)

    def get_transform(self, name):
        return self.output_field.get_transform(name)

    def as_sql(self, compiler, connection):
        resolved = self.expression.resolve_expression(compiler.query)
        return resolve_expression.as_sql(compiler, connection)

    def get_db_converters(self, connection):
      return self.output_field.get_db_converters(connection) + \
             self.expression.get_db_converters(connection)

This doesn’t need to be a full Expression subclass, because it mostly delegates things to the output field, but when it is turned into SQL, it can resolve the expression before then using that resolved expression to build the SQL.

So, let’s see how this works now (without showing the new Nodes that are handled by the Parser):

Person.objects.filter(display_name__startswith='Bob')

Yeah, that correctly limits the queryset. How about the ability to re-evaluate without a db round trip?

person = Person(first_name='Fred', last_name='Jones')
person.display_name  # -> 'Fred Jones'
person.preferred_name = 'Jonesy'
person.display_name  # -> 'Fred (Jonesy) Jones'

Success!

This project is not done yet: I have improved the Parser (as implied) to support more expressions, but there is still a bit more to go. It did occur to me (but not until I was writing this post) that the ComputedField(expression) version may actually be nicer. As hinted, that requires the value to be an expression, rather than a function call. It would be possible to create a function that references self, for instance, and breaks in all sorts of ways.

Redirect all DNS traffic to the pi.hole

2020-07-08T20:23:06+00:00

This is more to remind me than anything else, but I figured out how to configure my firewall to redirect all DNS traffic (except from the pihole itself) to the pihole.

My pihole has an IP address of 10.1.1.3:

iptables -t nat -A PREROUTING -i br-lan ! -s 10.1.1.3 -p tcp --dport 53 -j DNAT --to 10.1.1.3
iptables -t nat -A PREROUTING -i br-lan ! -s 10.1.1.3 -p udp --dport 53 -j DNAT --to 10.1.1.3
iptables -t nat -A POSTROUTING -j MASQUERADE

In OpenWrt, this needs to be pasted into Network → Firewall → Custom Rules, and then possibly reboot the router.

It is likely that a reboot is not necessary: the MASQUERADE line made me think I was still hitting the external DNS server, but it was transparently being handled by my pihole.

Certificate Expiry Dates without extra software

2020-06-27T14:32:04+00:00

I’ve got my Home Assistant set up and running, and have obtained a Lets Encrypt certificate to allow me to serve it all over HTTPS.

One of the things that you can do is set it up to notify you about expiring certificates. However, this requires the installation of a specific package. Since I’m running Home Assistant in a docker image, I can’t really do this.

However, the tools you need to determine a certificate’s expiry date are already in most systems (otherwise how would they be able to tell if the certificate from a site is still valid?).

echo | \
  openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

This gives the very useful:

notBefore=Nov 28 00:00:00 2018 GMT
notAfter=Dec  2 12:00:00 2020 GMT

We can manipulate this using some other commands to get just the expiry date:

echo | \
  openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -dates | \
  tail -n 1 | \
  cut -d '=' -f 2

Now, we want to turn this into a number of days from today. Bash can do arithmetic, we just need to make the values in the right format. In this case, we’ll get date to give us an epoch value, and divide this by 3600 * 24.

echo $(( ($(date +%s --date "2020-12-02 12:00:00") - $(date +%s)) / (3600 * 24) ))

That gives us 158 days from the day I wrote this. Now let’s put our command instead of the fixed date:

echo $((
  (
    $(date +%s --date "$(echo | \
  openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -dates | \
  tail -n 1 | \
  cut -d '=' -f 2)") - $(date +%s)
  ) / (3600 * 24)
))

Okay, we still get our 158. That’s a good sign.

Now, to put this into a Home Assistant sensor, we need to edit our configuration.yaml. Note that I needed to change the date parsing format inside the docker container to %b %d %H:%M:%S %Y GMT.

sensor:
  - platform: command_line
    name: SSL Certificate Expiry
    unit_of_measurement: days
    scan_interval: 10800
    command: echo $((
      (
        $(date +%s --date "$(echo | openssl s_client -connect example.com:443 2>/dev/null
                                  | openssl x509 -noout -dates
                                  | tail -n 1
                                  | cut -d '=' -f 2)"
                   -D "%b %d %H:%M:%S %Y GMT") - $(date +%s) ) / (3600 * 24) ))

This should give us a sensor that we can then use to create an automation, as seen in the original post.

Don’t forget to change the domain to your Home Assistant hostname!

Smart Devices Aren't (or why connected devices suck)

2020-06-25T11:37:52+00:00

I love tinkering with gadgets. I’ve put a bunch of sensors around my house, so I can see the temperature in various places, and have a couple of smart light and power devices too. At the moment, they are limited to my laundry (where the hard-wired switch is in the wrong place, due to the moving of a door), my workbench (because the overhead lights there run from a power point, so it was trivial to put in a smart switch), and the lounge room (where I had room in the light fitting to put a Sonoff Mini).

In all of these cases, with the exception of the Laundry, since the switch is not really accessible, I have taken great care to ensure that the physical switches still toggle the light. In that case I have an Ikea bulb connected to an Ikea dimmer.

In my study, I have a desk lamp that has a smart (dimmable) bulb in it, and it irks me no end that I have to use a smart device or computer to turn it on or off. I will be getting some more of the Ikea dimmers to alleviate this, but in the mean time, it’s a pain to turn on or off.

Having said that, I love the option of being able to automate power and lighting, or turn things off from a distance. I just don’t like that being the only way.

I installed Home Assistant on the weekend. But, in order to fit that onto my Raspberry Pi, I needed to use a bigger Micro SD card.

Which meant I needed to clone the old one.

Which took several hours.

I’d already installed Home Assistant before running out of space, and had converted a couple of my esphome devices to use the API instead of just MQTT for connection, including the lounge room light.

Now, it turns out, by default there is an “auto reboot if API not found in 15 minutes” setting, which meant that during the four or five hours it took to create an image of the Micro SD, verify this, copy to a new SD card, and then verify that, my lights (and a powerboard in my office) would flick off every 15 minutes. Likewise, if they cannot connect to a WiFi access point they will power cycle. I believe this second one can be resolved using a Captive AP setting that will mean if they can’t connect to a network, they will create their own.

Which really got me thinking. Smart devices should continue to work in every way possible when they don’t have access to the local network, or the internet. In my case, my smart devices do not have access to the internet anyway, because they don’t need to. However, the point is the same.

In situations where a network connection, or even worse, a working connection to a server that you don’t control, is no longer available, you dont’ want your lights or god forbid, your coffee machine to not be able to perform their simple task.

This worries me somewhat about the current trends in smart homes. At some point, companies will stop supporting their devices (this has already happened), and they will become less useful than their dumb counterparts. And add further to our global waste problems.

But having a significant system outage (even an intentional one, like in my case), made me think about other aspects of my home automation as well.

I’ve been using NodeRED for a couple of automation tasks. One of them was to have different grind lengths for my coffee grinder, and making this available to Siri.

However, with the device running NodeRED not operating, I was no longer able to rely on this.

I was heading this way philosophically before, but (OMG NO COFFEE) this just cemented something else in my mind. Automations, where they don’t rely on interaction between multiple devices, should live on the local device where possible. Further to this, where the interaction between devices is required for the automation (like the PIR sensor in the laundry I have that turns on the Ikea lightbulb), the devices should connect directly to one another, without requiring some other mechanism to trigger the automation.

In my case, I have a physical button that I press to trigger a long grind. But, the grind only stops if the NodeRED server tells it to. And, I had no way to (when NodeRED was not running), trigger a short grind.

I was able to fix this: I now have a short press triggering a long grind, and a long press triggering a short grind. That seems backwards, but since I mostly do a long grind in the morning before I’ve had time to properly wake up, I want that the easiest one to trigger…

Having to program this in my esphome firmware instead of NodeRED made for an interesting exercise. Because we need to turn off the device after a period of time, but need to be aware of other events that have happened in the meantime, we need to use scripts.

script:
  - id: short_grind
    then:
      - switch.turn_on: relay
      - delay: 13s
      - switch.turn_off: relay
  - id: long_grind
    then:
      - switch.turn_on: relay
      - delay: 17s
      - switch.turn_off: relay

Whenever our relay turns on, we want to start our long grind script, so that even if the relay was triggered some other way than through the script, it will turn off after 17s if not before. Whenever it turns off, we want to stop any instances of our scripts running. We can also use Template Switches to have logical devices we can use to trigger the different scripts, either from Home Assistant, or from button presses:

switch:
  - platform: gpio
    id: relay
    pin: GPIO2
    restore_mode: ALWAYS_OFF
    on_turn_on:
      - script.execute: long_grind
    on_turn_off:
      - script.stop: short_grind
      - script.stop: long_grind
  - platform: template
    name: "Grind a Single"
    optimistic: true
    id: grind_a_single
    icon: mdi:coffee-outline
    turn_on_action:
      - script.execute: short_grind
      - script.wait: short_grind
      - switch.template.publish:
          id: grind_a_single
          state: OFF
    turn_off_action:
      - switch.turn_off: relay
  - platform: template
    name: "Grind a Double"
    optimistic: true
    id: grind_a_double
    icon: mdi:coffee
    turn_on_action:
      - script.execute: long_grind
      - script.wait: long_grind
      - switch.template.publish:
          id: grind_a_double
          state: OFF
    turn_off_action:
      - switch.turn_off: relay

Both of these template switches will also turn off the grinder when toggled off if they are currently on.

There’s only one more bit of logic that’s required, and that’s the handling of the physical button. I wanted this to trigger either length based on the amount of time that the button is held down for, but I also want a UX affordance of knowing when you have held it down long enough to trigger the alternate action. Finally, if it’s on, any type of press should turn it off, and not trigger a new grind.

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO14
      inverted: true
      mode: INPUT_PULLUP
    on_press:
      - light.turn_on:
          id: led
          transition_length: 0s
      - delay: 500ms
      - light.turn_off:
          id: led
          transition_length: 0s
    on_click:
      - max_length: 350ms
        then:
          - if:
              condition:
                - switch.is_on: relay
              then:
                - switch.is_off: relay
              else:
                - script.execute: long_grind
      - min_length: 500ms
        max_length: 2s
        then:
          - if:
              condition:
                - switch.is_on: relay
              then:
                - switch.is_off: relay
              else:
                - script.execute: short_grind

Remember that the turning off of the relay will stop any running scripts.

So, now, when you hold down the button, when the light turns off, you can release it and it will trigger a short grind. If you just tap the switch and release it immediately, it will trigger a long grind. Any button press when the grinder is already running will turn it off.

Hacking Arlec's 'Smart' sensor light

2020-06-06T18:54:20+00:00

Quite some time ago, I purchased one of the Arlec Smart Security Lights from Bunnings. The big draw for me was that these devices have an ESP8266, and run a Tuya firmware, which can be trivially flashed without opening up the unit.

In this case, though, the device was not really that capable. Whilst there is a PIR sensor (and an LDR for tuning if the light should turn on yet or not), the status of the PIR is not exposed at all. Instead, the firmware allows toggling between three states: “ON”, “OFF”, and “SENSOR”.

That’s not actually that useful. For one, it makes using it in conjunction with a physical switch really inconvenient. The behaviour I would prefer is:

Light ON/OFF state can be toggled by network request.
Light ON/OFF state can be toggled by physical switch. It must not matter which state the switch is in, toggling it must toggle the light.
PIR ON turns ON the light.
PIR OFF turns OFF the light, but only if it was turned ON by PIR.

As the last point indicates, the only time the PIR turning off should turn the light off is if the light was turned on by the PIR. That is, either physical switch actuation or a network request should turn the light into manual override ON.

There is no manual override OFF.

Most of this was already present in a firmware I wrote for adding a PIR to a light strip. However, the ability to also toggle the state using a physical switch is important to me: not in the least because there is a switch inside the door where I want to mount this, and I’m very likely to accidentally turn it on when I go outside: it’s also a much better solution to a manual override than just having to use HomeKit. I’ll possibly add that feature back into the aforementioned project.

Like all of the other Grid Connect hardware I’ve used so far, it was easy to flash using tuya convert. But I did run into a bunch of problems from that point onwards. The base contains the ESP8266, one of the TYWE2S units, and it’s possible to see without opening up the sensor unit that this has three GPIO pins connected: GPIO4, GPIO5 and GPIO12.

With a custom firmware, it was possible to see that GPIO5 is connected to the green LED near the PIR sensor, but the other two appear to be connected to another IC on the PCB. I thought perhaps this could be accessed using the TuyaMCU protocol, but had no luck.

As it turns out, I’d prefer not to have to use that. There are two more wires, it would be great if I could connect one of them to the relay, and the other to the PIR.

Indeed, with limited rewiring (I did have to cut some tracks and run wires elsewhere), I was able to connect GPIO12 to the point on the PCB that the output from the other IC that triggered the relay, and GPIO4 to the input of the IC that was sensing the PIR output.

I also ran an extra pair of wires from GPIO14 and GND, to use to connect to the physical switch. These will only transmit low voltage.

Unfortunately, I forgot to take photos before putting it all back together and having it mounted on the wall.

Then we just need the firmware:

esphome:
  name: $device_name
  platform: ESP8266
  board: esp01_1m

globals:
  - id: manual_override
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: mqtt_triggered
    type: bool
    restore_value: no
    initial_value: 'false'

sensor:
  - platform: wifi_signal
    name: "WiFi signal sensor"
    update_interval: 60s

binary_sensor:
  - platform: gpio
    pin: GPIO4
    id: pir
    device_class: motion
    filters:
      - delayed_off: 15s
    on_press:
      - light.turn_on: green_led
      - mqtt.publish:
          topic: HomeKit/${device_name}/MotionSensor/MotionDetected
          payload: "1"
      - switch.turn_on: relay
    on_release:
      - light.turn_off: green_led
      - mqtt.publish:
          topic: HomeKit/${device_name}/MotionSensor/MotionDetected
          payload: "0"
      - if:
          condition:
            lambda: 'return id(manual_override);'
          then:
            logger.log: "Manual override prevents auto off."
          else:
            switch.turn_off: relay

  - platform: gpio
    pin:
      number: GPIO14
      mode: INPUT_PULLUP
    name: "Toggle switch"
    filters:
      - delayed_on_off: 100ms
    on_state:
      - switch.toggle: relay
      - globals.set:
          id: manual_override
          value: !lambda "return id(relay).state;"

ota:

logger:

output:
  - platform: esp8266_pwm
    id: gpio5
    pin:
      number: GPIO5
    inverted: False

switch:
  - platform: gpio
    id: relay
    pin:
      number: GPIO12
      # inverted: True
      # mode: INPUT_PULLDOWN_16
    on_turn_on:
      - if:
          condition:
            lambda: 'return id(mqtt_triggered);'
          then:
            logger.log: "No MQTT message sent"
          else:
            mqtt.publish:
              topic: HomeKit/${device_name}/Lightbulb/On
              retain: ON
              payload: "1"
    on_turn_off:
      - if:
          condition:
            lambda: 'return id(mqtt_triggered);'
          then:
            logger.log: "No MQTT message sent"
          else:
            mqtt.publish:
              topic: HomeKit/${device_name}/Lightbulb/On
              retain: ON
              payload: "0"


light:
  - platform: monochromatic
    id: green_led
    output: gpio5
    restore_mode: ALWAYS_OFF
    default_transition_length: 100ms

mqtt:
  broker: "mqtt.lan"
  discovery: false
  topic_prefix: esphome/${device_name}
  on_message:
    - topic: HomeKit/${device_name}/Lightbulb/On
      payload: "1"
      then:
        - globals.set:
            id: mqtt_triggered
            value: 'true'
        - switch.turn_on: relay
        - globals.set:
            id: mqtt_triggered
            value: 'false'
        - globals.set:
            id: manual_override
            value: !lambda "return !id(pir).state;"
    - topic:  HomeKit/${device_name}/Lightbulb/On
      payload: "0"
      then:
        - globals.set:
            id: mqtt_triggered
            value: 'true'
        - switch.turn_off: relay
        - globals.set:
            id: manual_override
            value: 'false'
        - globals.set:
            id: mqtt_triggered
            value: 'false'

I’ve also implemented a filter on sending the state to MQTT: basically, we don’t want to send a message to MQTT if we received the same message. There is a race condition that can occur where this results in fast toggling of the relay as each toggle sends a message, but then receives a message with the alternate state. I’ve had this on my Sonoff Mini firmware too.

Extracting values from environment variables in tox

2020-06-02T13:03:35+00:00

Tox is a great tool for automated testing. We use it, not only to run matrix testing, but to run different types of tests in different environments, enabling us to parallelise our test runs, and get better reporting about what types of tests failed.

Recently, we started using Robot Framework for some automated UI testing. This needs to run a django server, and almost certainly wants to run against a different database. This will require our tox -e robot to drop the database if it exists, and then create it.

Because we use dj-database-url to provide our database settings, our Codeship configuration contains an environment variable set to DATABASE_URL. This contains the host, port and database name, as well as the username/password if applicable. However, we don’t have the database name (or port) directly available in their own environment variables.

Instead, I wanted to extract these out of the postgres://user:password@host:port/dbname string.

My tox environment also needed to ensure that a distinct database was used for robot:

[testenv:robot]
setenv=
  CELERY_ALWAYS_EAGER=True
  DATABASE_URL={env:DATABASE_URL}_robot
  PORT=55002
  BROWSER=headlesschrome
whitelist_externals=
  /bin/sh
commands=
  sh -c 'dropdb --if-exists $(echo {env:DATABASE_URL} | cut -d "/" -f 4)'
  sh -c 'createdb $(echo {env:DATABASE_URL} | cut -d "/" -f 4)'
  coverage run --parallel-mode --branch manage.py robot --runserver={env:PORT}

And this was working great. I’m also using the $PG_USER environment variable, which is supplied by Codeship, but that just clutters things up.

However, when merged to our main repo, which has it’s own codeship environment, tests were failing. It would complain about the database not being present when attempting to run the robot tests.

It seems that we were using a different version of postgres, and thus were using a different port.

So, how can we extract the port from the $DATABASE_URL?

commands=
  sh -c 'dropdb --if-exists \
                -p $(echo {env:DATABASE_URL} | cut -d "/" -f 3 | cut -d ":" -f 3) \
                $(echo {env:DATABASE_URL} | cut -d "/" -f 4)'

Which is all well and good, until you have a $DATABASE_URL that omits the port…

dropdb: error: missing required argument database name

Ah, that would mean the command being executed was:

$ dropdb --if-exists -p  <database-name>

Eventually, I came up with the following:

sh -c 'export PG_PORT=$(echo {env:DATABASE_URL} | cut -d "/" -f 3 | cut -d ":" -f 3); \
              dropdb --if-exists \
                     -p $\{PG_PORT:-5432} \
                     $(echo {env:DATABASE_URL} | cut -d "/" -f 4)'

Whew, that is a mouthful!

We store the extracted value in a variable PG_PORT, and then use bash variable substitution (rather than tox variable substitution) to put it in, with a default value. But because of tox variable substitution, we need to escape the curly brace to allow it to be passed through to bash: $\{PG_PORT:-5432}. Also note that you’ll need a space after this before a line continuation, because bash seems to strip leading spaces from the continued line.

Django and Robot Framework

2020-05-27T17:34:01+00:00

One of my colleagues has spent a bunch of time investigating and then implementing some testing using Robot Framework. Whilst at times the command line feels like it was written by someone who hasn’t used unix much, it’s pretty powerful. There are also some nice tools, like several Google Chrome plugins that will record what you are doing and generate a script based upon that. There are also other tools to help build testing scripts.

There is also an existing DjangoLibrary for integrating with Django.

It’s an interesting approach: you install some extra middleware that allows you to perform requests directly to the server to create instances using Factory Boy, or fetch data from Querysets. However, it requires that the data is serialised before sending to the django server, and the same the other way. This means, for instance, that you cannot follow object references to get a related object without a bunch of legwork: usually you end up doing another Query Set query.

There are some things in it that I do not like:

A new instance of the django runserver command is started for each Test Suite. In our case, this takes over 10 seconds to start as all imports are processed.
The database is flushed between Test Suites. We have data that is added through migrations that is required for the system to operate correctly, and in some cases for tests to execute. This is the same problem I’ve seen with TransactionTestCase.
Migrations are applied before running each Test Suite. This is unnecessary, and just takes more time.
Migrations are created automatically before running each Test Suite. This is just the wrong approach: at worst you’d want to warn that migrations are not up to date - otherwise you are testing migrations that may not have been committed: your CI would pass because the migrations were generated, but your system would fail in reality because those migrations do not really exist. Unless you are also making migrations directly on your production server and not committing them at all, in which case you really should stop that.

That’s in addition to having to install extra middleware.

But, back onto the initial issue: interacting with Django models.

What would be much nicer is if you could just call the python code directly. You’d get python objects back, which means you can follow references, and not have to deal with serialisation.

It’s fairly easy to write a Library for Robot Framework, as it already runs under Python. The tricky bit is that to access Django models (or Factory Boy factories), you’ll want to have the Django infrastructure all managed for you.

Let’s look at what the DjangoLibrary might look like if you are able to assume that django is already available and configured:

import importlib

from django.apps import apps
from django.core.urlresolvers import reverse

from robot.libraries.BuiltIn import BuiltIn


class DjangoLibrary:
    """

    Tools for making interaction with Django easier.

    Installation: ensure that in your `resource.robot` or test file, you have the
    following in your "***Settings***" section:

        Library         djangobot.DjangoLibrary     ${HOSTNAME}     ${PORT}

    The following keywords are provided:


    Factory:        execute the named factory with the args and kwargs. You may omit
                    the 'factories' module from the path to reduce the amount of code
                    required.

        ${obj}=     Factory     app_label.FactoryName       arg  kwarg=value
        ${obj}=     Factory     app_label.factories.FactoryName     arg  kwarg=value


    Queryset:       return a queryset of the installed model, using the default manager
                    and filtering according to any keyword arguments.

        ${qs}=      Queryset    auth.User       pk=1


    Method Call:    Execute the callable with tha args/kwargs provided. This differs
                    from the Builtin "Call Method" in that it expects a callable, rather
                    than an instance and a method name.

        ${x}=       Method Call     ${foo.bar}      arg  kwargs=value


    Relative Url:   Resolve the named url and args/kwargs, and return the path. Not
                    quite as useful as the "Url", since it has no hostname, but may be
                    useful when dealing with `?next=/path/` values, for instance.

        ${url}=     Relative Url        foo:bar     baz=qux


    Url:            Resolve the named url with args/kwargs, and return the fully qualified url.

        ${url}=     Url                 foo:bar     baz=qux


    Fetch Url:      Resolve the named url with args/kwargs, and then using SeleniumLibrary,
                    navigate to that URL. This should be used instead of the "Go To" command,
                    as it allows using named urls instead of manually specifying urls.

        Fetch Url   foo:bar     baz=qux


    Url Should Match:   Assert that the current page matches the named url with args/kwargs.

        Url Should Match        foo:bar     baz=qux

    """

    def __init__(self, hostname, port, **kwargs):
        self.hostname = hostname
        self.port = port
        self.protocol = kwargs.pop('protocol', 'http')

    @property
    def selenium(self):
        return BuiltIn().get_library_instance('SeleniumLibrary')

    def factory(self, factory, **kwargs):
        module, name = factory.rsplit('.', 1)
        factory = getattr(importlib.import_module(module), name)
        return factory(**kwargs)

    def queryset(self, dotted_path, **kwargs):
        return apps.get_model(dotted_path.split('.'))._default_manager.filter(**kwargs)

    def method_call(self, method, *args, **kwargs):
        return method(*args, **kwargs)

    def fetch_url(self, name, *args, **kwargs):
        return self.selenium.go_to(self.url(name, *args, **kwargs))

    def relative_url(self, name, *args, **kwargs):
        return reverse(name, args=args, kwargs=kwargs)

    def url(self, name, *args, **kwargs):
        return '{}://{}:{}'.format(
            self.protocol,
            self.hostname,
            self.port,
        ) + reverse(name, args=args, kwargs=kwargs)

    def url_should_match(self, name, *args, **kwargs):
        self.selenium.location_should_be(self.url(name, *args, **kwargs))

You can write a management command: this allows you to hook in to Django’s existing infrastructure. Then, instead of calling robot directly, you use ./manage.py robot

What’s even nicer about using a management command is that you can have that (optionally, because in development you probably will already have a devserver running) start runserver, and kill it when it’s finished. This is the same philosophy as robotframework-DjangoLibrary already does, but we can start it once before running out tests, and kill it at the end.

So, what could our management command look like? Omitting the code for starting runserver, it’s quite neat:

from __future__ import absolute_import

from django.core.management import BaseCommand, CommandError

import robot


class Command(BaseCommand):
    def add_arguments(self, parser):
        parser.add_argument('tests', nargs='?', action='append')
        parser.add_argument('--variable', action='append')
        parser.add_argument('--include', action='append')

    def handle(self, **options):
        robot_options = {
            'outputdir': 'robot_results',
            'variable': options.get('variable') or []
        }
        if options.get('include'):
            robot_options['include'] = options['include']

        args = [
            'robot_tests/{}_test.robot'.format(arg)
            for arg in options['tests'] or ()
            if arg
        ] or ['robot_tests']

        result = robot.run(*args, **robot_options)

        if result:
            raise CommandError('Robot tests failed: {}'.format(result))

I think I’d like to do a bit more work on finding tests, but this works as a starting point. We can call this like:

./manage.py robot foo --variable BROWSER:firefox --variable PORT:8000

This will find a test called robot_tests/foo_test.robot, and execute that. If you omit the test argument, it will run on all tests in the robot_tests/ directory.

I’ve still got a bit to do on cleaning up the code that starts/stops the server, but I think this is useful even without that.

Take that, Mr Morrison

2020-03-27T11:30:57+00:00

People power still works.

Mr Morrison and his advisory panel still maintain that schools should remain open. For some reason, kids are immune to getting or spreading COVID-19 whilst at school, but if they visit their grandparents or a shopping centre, then all hell will break loose.

We, like many other parents around Australia, decided to withdraw our children from physical schooling (we are still doing remote learning, and our school has been very supportive of this) for nearly two weeks now.

A few days ago, we received an email from our school: the whole school will be remote-only; with exceptions for children of parents who are unable to perform remote learning, either because they work in essential jobs or whatever other reason. Those students will be managed at school, as of next week, but all students will be receiveng the same curriculum.

This is exactly what the NSW premier indicated her state’s schools would be doing. This makes perfect sense. Reduce the risk to teachers, and reduce the exposure of children to one another.

Now SA, and most other states, have announced early closure, and possible remote learning after the school holidays.

People voted by withdrawing their children from school. Maybe Mr Morrison should pay attention.

Human contact, or what the f*** are all these people doing

2020-03-24T22:11:33+00:00

I went outside of my house today.

Initially I went to my office to get a few things that I’ll need to work from home (standing desk extension and a better trackpad were the main ones). But after that, I thought I’d swing by one of the smaller supermarkets in my area, and if it was quiet enough, pick up some supplies.

There were no customers when I went in. Good.

The first one who came in, when trying to get to a section near me, instead of taking the path that avoided me, instead took the path that went right by me. I quickly scrambled to get out of the way. Call me paranoid, but the fewer people I get close to, the smaller my chance of contracting the virus.

Eventually, too many people were in the store, so I left without everything I needed (I’d struggled to find some bits anyway). Having to get closer than I would have liked to people, I braved the checkout area, and made it outside.

So, for future reference, I came within a couple of metres of about 6 or 7 people at Drakes Mini between maybe 6:30 and 7pm.

The PM announced further measures, and said one thing I thought was interesting:

“It seems like lots of people want a complete lockdown”

Damn straight we do.

“Let me tell you, it won’t be for a short period of time.”

We are very aware of that: but it’s about public health. Man up, and lock it down. Have you not seen that China is able to start to release restrictions because they have started to control it?

Close Contacts and Community Transmission

2020-03-23T19:01:12+00:00

The Australian government (and specifically the South Australian government) have made a big deal about how, so far at least, there has been virtually no “Community Transmission”. That it’s all been imports from overseas, other states, or “Close Contacts”.

But what do these terms “Close Contacts” and “Community Transmission” mean, in the scope of what normal people would assume they mean, and what the governments are using them to mean.

To me, a “Close Contact” would be a member of my immediate family, or perhaps one of the three people who share my office. Possibly even the other 6 people who work in my company who also work in the same place as me - since the air we all breathe is the same, and we share the same kitchen and bathroom facilities. Also, you’d probably count my extended family, although I don’t see them often, when I do see them we in close physical proximity. Indeed, even our close friends, who we don’t see as often as we would like, the sort of ones we catch up with every few months for dinner would count as close contacts.

A generous expansion of this might include the parents we see, and chat with every morning doing school drop-off. And of course, our kids’ close contacts would include their teacher and their closest friends. Note the word close.

So, you’d think that “Community Transmission” would include everyone else. People you come across in service roles, like the guy at the sushi bar, or your Barista. Maybe your neighbour that you chat to while walking your dog. The other parents who you stop briefly to and say hello. The Principal of the school, who you talk to probably once a week.

But, according to The Conversation AU, these people would be counted by the official tally as “Close Contacts”.

I think this belies how serious of a mismatch there is between the government’s current position and communication and reality there is. Keeping schools open will not, according to their definition, cause Community Transmission. It cannot, because by definition, this would count as Close Contacts.

Perhaps this is part of the reason they want to keep schools open. Being able to claim that cases are still being spread only through Close Contacts, when in reality the virus is being transmitted through the school COMMUNITY.

Maybe I’m turning into a tin-hatter, but I feel like our government is not trying hard enough to reduce the growth in number of cases. At this stage, the number of cases in South Australia is speeding up: it was growing at 23%, but now seems to be growing at 33%. Sure, some of this is better testing, but it doesn’t seem at this stage that many of the measures they have started to take have had much impact on the spread.

Lockdown: Day 6

2020-03-22T14:21:21+00:00

So I guess this is becoming a bit of a journal. I missed yesterday though… ;)

The Australian government announced further economic aid to people who are newly unemployed or under-employed due to COVID-19. Notably, this included doubling the Jobseeker allowance (or whatever it’s currently called). This is surely a kick in the guts for the previously unemployed. Does it not state that the previous amount was way less than what an average person needs to survive? But now there are going to be significant numbers of people who until a week ago were fully-employed, the amount doubles.

Interesting.

We are going to take the boys out for a walk in the local National Park. I don’t know how draconian lockdown rules are going to be in the future. Keeping schools open seems to suggest they don’t really give a fuck though.

Our internet was pretty crappy all day yesterday; and a bit intermittent on Friday. Right now, it’s totally gone - allegedly there is an issue at some place that is up the chain from me: they have turned the power off there and are working on it. Hopefully it comes back on today.

Using Zoom for remote Cello lessons

2020-03-21T12:21:16+00:00

With COVID-19 starting to cause significant lockdowns, our music lessons have moved to online. Our piano and cello teachers are both using Zoom for this.

Piano was fine, but the Cello teacher noticed that when my eldest son was playing some notes (especially the lower ones), the audio was cutting out. When he mentioned it was fine at the start of the note, but dropped in the middle, and then came back at the end, it triggered something in my brain.

Turns out Zoom has automatic background noise cancellation. This was picking up the low sounds of the Cello and blocking them.

This is fairly easy to fix:

Settings → Audio → Advanced

Then, set both types of background noise suppression to disabled:

I’m not sure which one is more important: you’d think it was probably intermittent, but it was easy enough in our case just to leave them both set to disable.

Lockdown: Day 4

2020-03-20T18:54:58+00:00

It’s Friday. End of our first week of working and schooling from home.

We bought a new iPad for the boys, which arrived within 25 hours. They are now using it attached to the trampoline; the old iPad was not working with that, so that’s a bonus. We also bought a water filter, so it’s been a productive week.

Working from home has been fine. I’m sure I’m not as productive as I was in previous weeks, but I don’t know how much of that is COVID-19 in general playing on my mind.

Australia’s infection rate is still above what my 23% compounding increase model suggests. https://covid-19-au.github.io is currently showing 854 confirmed cases, whilst my simple predictions said 835 by today. Watch it go over 1000 tomorrow.

According to the Guardian, our rates are slightly better than most other countries, although only Japan is an outlier there. Since that’s a logarithmic scale, we are perhaps even better off than it seems. Not sure how much of that is just insufficient testing though. On the same page, you can see that our total number of deaths is low: however, there is nothing that shows deaths-per-infection.

SA still sits at 50, which is about 30% better than predicted. However, we are not doing nearly as much testing as NSW.

Recommendations are to stay at home all weekend. I suspect I will be taking the boys somewhere - perhaps we could go for a walk in BNP.

Lockdown: Day 3

2020-03-19T19:50:37+00:00

Took the boys down to the park this afternoon. They hadn’t left the house since Monday evening.

The school adjacent to the park was closed (due to a case of COVID-19), but otherwise, it didn’t feel that different. There were a couple of other groups of people there, and we ran/scooted/played for about an hour.

Getting them out of the house was really good - not only did they get to burn a bunch of energy, but it seems to have made them better able to play together this evening. They’ve been really good so far in terms of not fighting that much, but it is only day 3.

I’d be so happy if we turn out to have gone way too gung-ho on this - if South Australia (and Australia) manage to get this outbreak under control in weeks, rather than many months, I’ll be happy to cop shit for going overboard and pulling the kids out of school. But the number of cases has continued to grow at a rate higher than 23% per day.

My modelling, which is simplistic, looks like this:

    date    │ infected
────────────┼──────────
 2020-03-17 │      449
 2020-03-18 │      552
 2020-03-19 │      679
 2020-03-20 │      835
 2020-03-21 │     1027
 2020-03-22 │     1263
 2020-03-23 │     1553
 2020-03-24 │     1910
 2020-03-25 │     2349
 2020-03-26 │     2889
 2020-03-27 │     3553
 2020-03-28 │     4370
 2020-03-29 │     5375
 2020-03-30 │     6611
 2020-03-31 │     8132
 2020-04-01 │    10002

That’s stopping at 10k, by April 1st.

Currently, it’s 2020-03-19, and there are 691 confirmed cases.

Lockdown: Day 2

2020-03-18T21:08:37+00:00

I really don’t know how this is going to play out.

It seems that the Australian government just doesn’t want to shut schools because they know when they do shut them, they won’t be re-opening them for many months. What they’d prefer to do is hold that off as long as possible, at the cost of people’s lives.

Currently, in Australia, the number of cases of COVID-19 appears to be growing at a rate exceeding 23% per day.

At that rate, we’ll have over 1000 cases by the weekend, and 10k within a fortnight or so. Within three weeks, there will no longer be any where near enough intensive care beds or ventilators to support the number of critically ill patients, and we’ll start to see Italy-like levels of mortality.

How will schools stay open when teachers are dying.

Our parents don’t seem to realise how dangerous this is.

My parents are apparently still planning on travelling from a rural town of 6000 people (they actually live on a farm, so that’s even better), where they are relatively safe, to Adelaide to attend a funeral. I understand that it’s one of their friends that died, but to be honest, there will be a lot more of those in the not too distant future.

And my partner’s mother was planning on going to the hairdresser tomorrow.

Personally, we have savings, and I have a stable job (for now). But considering the number of our customers who are service industry, quite a number of which I have little doubt will be out of business in short order.

There will be a new normal, we just don’t know what that is going to look like.

Lockdown: Day One

2020-03-17T19:35:12+00:00

The Australian government is in denial.

It’s becoming clear that the global health crisis from COVID-19 is very likely to worsen quite quickly. From information I have come across from reliable medical sources (mostly from the ABC), it seems that our government is probably not doing enough to slow the growth of cases, and that the health risks to all age groups are significant, not just the elderly or vulnerable. Similarly, it feels like our health system has every likelihood of being overwhelmed by the increasing case load of critically ill people.

Because I have the “luxury” of being able to work from home (and still be gainfully employed), I am now doing so; additionally we are keeping our children home from school. Whilst there is probably very little practical risk from any two students attending one specific school, every person you come into contact with, and more so every person you remain in contact with for a significant period of time is an increased risk.

Other countries have been recommending that gatherings larger than 5 people are sufficient to continue the spread of the virus, and whilst it may seem that Australia (and Adelaide specifically) has less risk right now than some European and Asian countries, we have already seen Victoria declare a State of Emergency. Yet our government maintains that 500 people is still safe.

The number of confirmed cases in Australia has reached a higher proportion of the general population than that of the United States, which has many states and cities implementing strict shutdown policies.

The actual number of confirmed cases has started to grow exponentially, with an increase of almost 50% in one day (20 to 29 on March 16th). The school 400m away from where I live has a confirmed case - the principal of my children’s school has a child that attends there. The school where my niece attends school has a confirmed case.

Around 4000 doctors sent a petition to the government asking for strict lockdowns, fearing that our outcomes could be worse than that of Italy.

The government continues to maintain that this disease is just not that risky for young people.

This is just a flat-out lie.

The only reason we have not had deaths in Australia of young people is just a matter of numbers. Currently, anyone who has had severe respiratory issues has been able to receive artificial respiration. This will change, as there are only so many resipirators in Australian hospitals. As more people are infected, more will get serious complications, and some of them will miss out. Either on a respirator altogether, or later in the epidemic, perhaps even a bed.

Children appear not to be having severe complications, or even especially strong symptoms, but there is evidence that asymptomatic people shed a higher amount of the virus, and children continue to have the virus in stool samples well after they are deemed to be “clear” according to a throat swab.

So, my kids, partner and I stayed home today. I think it’s still fun for them - we have tried to help them understand the scale of this problem (“We could be here, just the four of us together for at least two weeks, maybe even longer”), but the novelty (oops) of being home-schooled has yet to wear off.

I’m thankful I have the means and opportunity to work from home, and have my immediate family here with me. We aren’t going to see the boys’ grandparents for some weeks, and I’m sure we’ll all go batshit insane from being around one another non-stop for an indefinite period of time.

Stay safe.

Modelling a residential battery in Postgres

2020-03-10T22:31:54+00:00

Having five-minute interval generation and consumption data from my PV system means that, in theory at least, I should be able to do some predictive modelling of what adding a battery to the system will do to my import and export levels, and more importantly, determine how much money having a battery of a specific capacity and efficiency would save me.

There are several approaches that could be used to model a battery. They all boil down to the same main idea:

The battery has a fixed capacity
The battery has a current level
If generating more power than consuming, this excess could be fed into the battery (assuming the battery is not full)
If the battery is full, any excess generation is exported
If we are consuming more power than generating, and we have battery level > 0, then we draw power from the battery
Any power we are drawing from the battery is divided by the round-trip efficiency of the battery
Any excess consumption that cannot be supplied by the battery results in imports

There are some assumptions that this model makes.

There is no current (as in Amperes, not “now”) limit to either exporting or importing
If we are both importing and exporting, the battery can handle both of these. In practice, most batteries take some time to switch between charging and discharging
The usable capacity is the same as the battery capacity

In all of these cases, the limitations we are not addressing would more than likely result in a poorer outcome than the model will predict.

So, on to the technical details of creating the model. In the first instance, I tried using a recursive view (using in this case the 13.5kWh/90% efficient Tesla Powerwall):

CREATE OR REPLACE RECURSIVE VIEW battery_status(
  timestamp,
  capacity,
  efficiency,
  level,
  import,
  export
) AS

SELECT '2019-03-07'::TIMESTAMP,
       13500 AS capacity,
       0.90 AS efficiency,
       0::NUMERIC AS level,
       0::NUMERIC AS import,
       0::NUMERIC AS export

 UNION ALL

SELECT battery.timestamp + INTERVAL '5 minutes',
       battery.capacity,
       battery.efficiency,
       GREATEST(
         0,
         battery.level + delta.amount_to_battery - delta.amount_from_battery
       )::NUMERIC(12, 3) AS level,
       (summary.import - delta.amount_from_battery)::NUMERIC(8, 3) AS import,
       (summary.export - delta.amount_to_battery)::NUMERIC(8, 3) AS export
  FROM battery_status battery
  LEFT OUTER JOIN summary ON (summary.timestamp = battery.timestamp + INTERVAL '5 minutes')
 INNER JOIN LATERAL (
   SELECT LEAST(summary.import, battery.level / battery.efficiency) AS amount_from_battery,
          LEAST(summary.export, battery.capacity - battery.level) AS amount_to_battery
 ) delta ON (true)

Whilst this sort-of worked, it was unsatisfactory in a couple of ways. Firstly, it took ages to run. Way longer than I expected, even when I used materialised views, and changed the interval to hourly instead of 5 minutely.

Additionally, because I have some gaps in my data (for instance, when my power was disconnected, or the inverter was off because I was cleaning the panels, or whatever), the view stopped recursing at this point, so I was never actually able to get a result that went for more than a couple of months. Even generating missing values seemed to be insufficient, so at some point I gave up on this.

I even tried the same approach on a daily summary - this was never going to give me anything close to an accurate result, but at that point I was stretching thin, so tried a bunch of things.

It seemed unusual that it was taking a long as it did (dozens of seconds, and that was only to build a few months of data). It should only need to pass over the summary data once, and store…that gave me an idea. I could write a fairly simple python generator function to perform the modeling.

def battery_status(capacity, efficiency):
    result = plpy.execute('SELECT * FROM summary')
    energy = 0

    for row in result:
        export_amount = row['export']
        if export_amount and energy < capacity:
            energy += export_amount
            if energy > capacity:
                export_amount = energy - capacity
                energy = capacity
            else:
                export_amount = 0
        import_amount = row['import']
        if import_amount and energy > 0:
            energy -= import_amount / efficiency
            if energy < 0:
                import_amount = -energy * efficiency
                energy = 0
            else:
                import_amount = 0
        yield (row['timestamp'], energy, import_amount, export_amount)

Following it through, we can see some benefits to this approach. We don’t need to pass the capacity and efficiency (or the current energy level in the battery) through to each iteration: because this is a generator, the state remains in the function, and those variables are just re-used over and over again.

The only thing that might be unusual there is the plpy.execute('SELECT * FROM summary'): this hints that this code is actually going inside a plpythonu function and living inside the database. We can use a composite type to make the return values easier to deal with:

CREATE TYPE battery_status AS (
    timestamp TIMESTAMP,
    stored NUMERIC,
    import NUMERIC,
    export NUMERIC
);

CREATE FUNCTION battery_status(capacity NUMERIC, efficiency NUMERIC)
RETURNS SETOF battery_status AS $$

result = plpy.execute('SELECT * FROM summary')
energy = 0

for row in result:
    export_amount = row['export']
    if export_amount and energy < capacity:
        energy += export_amount
        if energy > capacity:
            export_amount = energy - capacity
            energy = capacity
        else:
            export_amount = 0
    import_amount = row['import']
    if import_amount and energy > 0:
        energy -= import_amount / efficiency
        if energy < 0:
            import_amount = -energy * efficiency
            energy = 0
        else:
            import_amount = 0
    yield (row['timestamp'], energy, import_amount, export_amount)

$$ LANGUAGE plpythonu STRICT IMMUTABLE;

Again, I’ll reiterate that in this context, we apply both charging and discharging in the same period (which could happen), but assumes that there is zero turnaround time required between charge and discharge, and that there is no limit to the charge and discharge rate. We could probably improve the model to take those two into account fairly easily though.

To revisit what the other data stores look like, let’s have a look at my summary view:

CREATE OR REPLACE VIEW summary AS

SELECT "timestamp",
       COALESCE(generation.energy, 0::double precision) AS generation,
       COALESCE(import.energy, 0) AS import,
       COALESCE(export.energy, 0) AS export,
       import.energy::double precision
           + generation.energy
           - export.energy::double precision AS consumption
  FROM generation
  JOIN import USING ("timestamp")
  JOIN export USING ("timestamp")

My generation table just contains the generation for each interval: the import and export are both views that use the cumulative import/export totals and calculate the value by subtracting the previous row:

CREATE OR REPLACE VIEW import AS

SELECT import_cumulative."timestamp",
       import_cumulative.energy
           - lag(import_cumulative.energy, 1) OVER (ORDER BY import_cumulative."timestamp")
           AS energy
  FROM import_cumulative
 ORDER BY import_cumulative."timestamp";

We can create a materialised view of the battery status per-day, since that’s all we will need for the purposes of cost calculations:

CREATE MATERIALIZED VIEW daily_summary_battery AS

SELECT battery_status."timestamp"::date AS "timestamp",
       sum(battery_status.import) AS import,
       sum(battery_status.export) AS export
  FROM battery_status(13500::numeric, 0.9)
       battery_status("timestamp", stored, import, export)
 GROUP BY (battery_status."timestamp"::date)
 ORDER BY (battery_status."timestamp"::date);

So, lets have a look at what results we get.

Refreshing the materialised view takes me around 9 seconds (this is quite stable), and I currently have about 106000 records (370 days of data, 288 records per day if we got them all). This is nice: it’s not immediate, but not going to be a deal-breaker if I need to run a bunch of different models of different battery capacity or efficiencies.

I can use the same cost functions from my actual cost calculator to look at what this battery model will do to my costs:

CREATE VIEW daily_battery_cost AS
SELECT daily_summary_battery."timestamp" AS date,
       daily_summary_battery.import,
       daily_summary_battery.export,
       cost(retailer.*,
            daily_summary_battery.import::integer,
            daily_summary_battery.export::integer) AS cost,
       billing_periods.period
  FROM daily_summary_battery
  JOIN billing_periods ON daily_summary_battery."timestamp" <@ billing_periods.period
  JOIN retailer ON retailer.name = billing_periods.retailer
 ORDER BY daily_summary_battery."timestamp";

And for comparisons in my billing periods:

CREATE VIEW billing_period_battery_costs AS
SELECT daily_battery_cost.period,
       sum(daily_battery_cost.cost) AS cost
  FROM daily_battery_cost
 GROUP BY daily_battery_cost.period
 ORDER BY daily_battery_cost.period;

CREATE VIEW billing_period_battery_savings AS
SELECT costs.period,
       costs.cost - battery.cost AS savings
  FROM billing_period_costs costs
  JOIN billing_period_battery_costs battery USING (period)
 ORDER BY costs.period;

solar=# SELECT * FROM billing_period_battery_savings ;
         period          |            savings
-------------------------+-------------------------------
 [2019-03-07,2019-06-01) | 101.6372820000000000000000000
 [2019-06-01,2019-06-24) |      15.592432500000000000000
 [2019-06-24,2019-07-24) |  20.4731725000000000000000000
 [2019-07-24,2019-08-24) |      17.898683500000000000000
 [2019-08-24,2019-09-24) |  18.5995855000000000000000000
 [2019-09-24,2019-10-24) |  35.1526490000000000000000000
 [2019-10-24,2019-11-24) |  38.4160700000000000000000000
 [2019-11-24,2019-12-24) |  37.7519350000000000000000000
 [2019-12-24,2020-01-24) |  39.2153900000000000000000000
 [2020-01-24,2020-02-24) |  39.5369135000000000000000000
 [2020-02-24,2020-03-24) |  21.2207440000000000000000000
(11 rows)

And for a per-day saving: it works out to around $1.05 per day.

Which is ~$380 per year.

A Tesla Powerwall 2 battery would cost me (excluding installation, but including the SA battery rebate), in excess of $5000. That’s still almost a 15 year payback time.

What’s quite amusing is that I distinctly remember calculating what I thought a battery would save me shortly after getting my PV system installed. I have such a strong recollection of it that I can remember exactly where I was: I was out running at lunch time, and was partway up the hill at the back of Flinders Uni.

And my in-head calculations were that it would save me $1 per day.

I’m actually more interested in a DC coupled battery system, partly because of the reduction in round-trip efficiency losses, but more so because I should be able to stick more panels in that would just feed the battery. I’m especially interested in the Fronius Gen24 inverters and a compatible battery, however there aren’t clear details on what that would cost. The benefit of this is that in SA there is currently a 10kW inverter limit; this includes the inverter in an AC coupled battery like the Tesla Powerwall.

These are likely to come midway through this year. I’ll be interested to see the pricing.

I think I’ll need to do a bit more on my calculations to be able to model this: my panel_factor calculations have a 5kW limit - this goes away, and instead the limit applies after taking the battery charging into account. But this can wait for another day.

QuerySets of various models

2020-03-09T10:36:53+00:00

In general, the first thing we try to do to reduce the load time of a page in Django is to reduce the number of queries we are making to the database. This often has the greatest impact, sometimes orders of magnitude greater than other improvements.

One problem I recently hit with a specific set of pages was that there are potentially seven different models that may have zero or more items for a given request. This could mean we do seven queries that are all empty.

These are for all distinct models, but in this case, they are used for the same purpose, and this case, we always need all of them, if there are any relevant records. Whilst there are seven models, there may be more in the future.

Each of these models has a specific method, that validates some data against the rules that are applicable to that model, and the stored attributes. But each model has a different set of attributes, and future models will more than likely have different attributes.

There are at least three different ways we could have solved this problem. It turns out we have solved similar problems in the past the first two ways, and the third one is the one I’ve used in this case.

Solution 1: Store all data in the same model, and use a JSON field to store the attributes specific to this “class”.

This also requires a “type” field of some sort. Then, when loading data, we have the model use this type field to work out which attributes should apply.

This has a bunch of problems. First and foremost is that it becomes far more difficult to get the database (postgres, in this case) to apply integrity constraints. It’s not impossible, but it is much harder to read a constraint that checks a field and performs a JSON expression constraint. Changing a constraint, assuming it’s done using a check constraint and not a trigger, is still possible, but is likely to be harder to understand.

Secondly, it no longer becomes “automatic” to get Django Model Form support. It’s not impossible to use a Django Model Form, but you need to work a bit harder to get the initial data in, and ensure that the cleaned data for the fields is applied to the JSON field correctly.

Finally, as hinted above, using a “type” field means the logic for building the checks is more complex, unless you use class swizzling and proxy models or similar to have a different class for each type value. If an instance was accidentally updated to the wrong type, then all sorts of things could go wrong.

This was the first solution we used for our external integrations, and whilst convenient at some level, turned out to be much harder to manage than distinct model classes. It is not subject to the problem that is the basis of this article: we can always fetch objects of different logical types, as it’s all the same model class. Indeed, to only fetch a single class, we need to perform extra filtering.

Solution 2: Use concrete/multi-table inheritance.

This is the solution we moved to with our external integrations, and has been much nicer than the previous solution. Instead of having a JSON field with an arbitrary blob of data in it, we have distinct fields. This makes it much easier to have unique constraints, as well as requiring values, or values of a specific type. Knowing that the database is going to catch us accidentally putting a text value into the external id field for a system that requires an integer is reassuring.

This overcomes the second problem. We can now just use a Django ModelForm, and this makes writing views much easier. The validation for a given field or set of fields lives on the model, and where possible also in the database, as an exclusion or check constraint.

It also overcomes the third problem. We have distinct classes, which can have their own methods. We don’t need to try to use some magical proxy model code, and it’s easy for new developers to follow.

Finally, thanks to django-model-utils InheritanceManager, we can fetch all of our objects using the concrete parent model, and the .select_subclasses() method to downcast to our required class.

There are a couple of drawbacks to using concrete inheritance. Any fetch of an instance will perform a JOIN in your database, but more importantly, it’s impossible to perform a bulk_create() for models of these types.

Solution 3: Use a Postgres VIEW and JSONB to perform one query, and reconstruct models.

In the problem I’ve recently solved, we had a bunch of different models that, although being used in similar ways, didn’t have that much similarity. They were pre-existing, and it wasn’t worth the effort to move them to concrete inheritance, and using JSON fields for all data is not something I would repeat.

Instead, I came up with an idea that seems novel, based on some previous work I’ve done converting JSON data into models:

CREATE VIEW combined_objects AS

SELECT 'foo.Foo' AS model,
       foo.id,
       foo.tenant_id,
       TO_JSONB(foo) AS data
  FROM foo

 UNION ALL

SELECT 'foo.Bar' AS model,
       bar.id,
       baz.tenant_id,
       TO_JSONB(bar) AS data
  FROM bar
 INNER JOIN baz ON (baz.id = bar.baz_id)

 UNION ALL

SELECT 'qux.Qux' AS model,
       qux.id,
       tenant.id,
       TO_JSONB(qux) AS data
  FROM tenant
  JOIN qux ON (true)

This builds up a postgres view that contains all of the data from the model, but in a generic way. It also contains a tenant_id, which in this case was the mechanism that we’ll be using to filter the ones that are required at any given time. This can be a field on a model (as shown in the first subquery), or a field on a related model (as shown in the second). It could even be every object in a table for every tenant, as shown in the third.

From there, we need a model that will recreate the model instances correctly:

class CombinedObjectQuerySet(models.query.QuerySet):
    def for_tenant(self, tenant):
        return self.filter(tenant=tenant)

    def as_model_instances(self):
        return [x.instance for x in self]


class CombinedObject(models.Model):
    model = models.TextField()
    tenant = models.ForeignKey('tenant.Tenant')
    data = JSONField()

    objects = CombinedObjectQuerySet.as_manager()

    class Meta:
        managed = False
        db_table = 'combined_objects'

    def __str__(self):
        return '%s wrapper (%s)'.format(self.model, self.instance)

    def __eq__(self, other):
        return self.instance == other.instance

    @property
    def model_class(self):
        return apps.get_model(*self.model.split('.'))

    @cached_property
    def instance(self):
        return self.model_class(**self.data)

This works great, as long as you don’t apply additional annotations, typecast to python values, or want to deal with related objects. That is where it starts to get a bit tricky.

We can handle annotations and typecasting:

@cached_property
def instance(self):
    data = self.data
    model_class = self.model_class
    field_data = {
        field.name: field.to_python(data[field.name])
        for field in model_class
        if field.name in data
    }
    instance = model_class(**field_data)
    for attr, value in data.items():
        if attr not in field_data:
            setattr(instance, attr, value)
    return instance

There’s still the issue of foreign keys in the target models: in this case I know the code is not going to traverse these and trigger extra database hits. We could look at omitting those fields to prevent that being possible, but this is working well enough for now.

Postgres VIEW from Django QuerySet

2020-03-03T12:29:46+00:00

It’s already possible, given an existing Postgres (or other database) VIEW, to stick a Django Model in front of it, and have it fetch data from that instead of a table.

Creating the views can currently be done using raw SQL (and a RunSQL migration operation), or using some helpers to store the SQL in files for easy versioning.

It would be excellent if it was possible to use Django’s ORM to actually generate the VIEW, and even better if you could make the migration autodetector generate migrations.

But why would this be necessary? Surely, if you were able to create a QuerySet instance that contains the items in your view, that should be good enough?

Not quite, because currently using the ORM it is not possible to perform the following type of query:

SELECT foo.a,
       foo.b,
       bar.d
  FROM foo
  INNER JOIN (
    SELECT baz.a,
           ARRAY_AGG(baz.c) AS d
      FROM baz
     GROUP BY baz.a) bar ON (foo.a = bar.a)

That is, generating a join to a subquery is not possible in the ORM. In this case, you could probably get away with a correlated Subquery, however that would probably not perform as well as using a join in this case. This is because a subquery in a SELECT is evaluated once for each row, whereas a subquery join will be evaluated once.

So, we could use a VIEW for the subquery component:

CREATE OR REPLACE VIEW bar AS

SELECT baz.a,
       ARRAY_AGG(baz.c) AS d
  FROM baz
 GROUP BY baz.a;

And then stick a model in front of that, and join accordingly:

SELECT foo.a,
       foo.b,
       bar.d
  FROM foo
 INNER JOIN bar ON (foo.a = bar.a)

The Django model for the view would look something like:

class Bar(models.Model):
    a = models.OneToOneField(
        'foo.Foo',
        on_delete=models.DO_NOTHING,
        primary_key=True,
        related_name='bar'
    )
    d = django.contrib.postgres.fields.ArrayField(
        base_field=models.TextField()
    )

    class Meta:
        managed = False

The on_delete=models.DO_NOTHING is important: without it, a delete of a Foo instance would trigger an attempted delete of a Bar instance - which would cause a database error, because it’s coming from a VIEW instead of a TABLE.

Then, we’d be able to use:

queryset = Foo.objects.select_related('bar')

So, that’s the logic behind needing to be able to do a subquery, and it becomes even more compelling if you need that subquery/view to filter the objects, or perform some other expression/operation. So, how can we make Django emit code that will enable us to handle that?

There are two problems:

Turn a queryset into a VIEW.
Get the migration autodetector to trigger VIEW creation.

The other day I came across Create Table As Select in Django, and it made me realise that we can use basically the same logic for creating a view. So, we can create a migration operation that will perform this for us:

class CreateOrReplaceView(Operation):
    def __init__(self, view_name, queryset):
        self.view_name = view_name
        self.queryset = queryset

    def database_forwards(self, app_label, schema_editor, from_state, to_state):
        queryset = self.queryset
        compiler = queryset.query.get_compiler(using=schema_editor.connection.alias)
        sql, params = compiler.as_sql()
        sql = 'CREATE OR REPLACE VIEW {view} AS {sql}'.format(
            view=schema_editor.connection.ops.quote_name(self.view_name),
            sql=sql
        )
        schema_editor.execute(sql, params)

    def state_forwards(self, app_label, state):
        pass

We can then have this operation (which needs to be passed a queryset).

This doesn’t really solve how to define the queryset for the view, and have some mechanism for resolving changes made to that queryset (so we can generate a new migration if necessary). It also means we have a queryset written in our migration operation. We won’t be able to leave it like that: due to loading issues, you won’t be able to import model classes during the migration setup - and even if you could, you shouldn’t be accessing them during a migration anyway - you should use models from the ProjectState which is tied to where in the migration graph you currently are.

What would be excellent is if we could write something like:

class Bar(models.Model):
    a = models.OneToOneField(
        'foo.Foo',
        on_delete=models.DO_NOTHING,
        primary_key=True,
        related_name='bar',
    )
    d = django.contrib.postgres.fields.ArrayField(
        base_field=models.TextField()
    )

    class Meta:
        managed = False

    @property
    def view_queryset(self):
        return Baz.objects.values('a').annotate(d=ArrayAgg('c'))

And then, if we change our view definition:

@property
def view_queryset(self):
  return Baz.objects.values('a').filter(
      c__startswith='qux',
  ).annotate(
      d=ArrayAgg('c')
  )

… we would want a migration operation generated that includes the new queryset, or at least be able to know that it has changed. Ideally, we’d want to have a queryset attribute inside the Meta class in our model, which could itself be a property. However, that’s not possible without making changes to django itself.

In the meantime, we can borrow the pattern used by RunPython to have a callable that is passed some parameters during application of the migration, which returns the queryset. We can then have a migration file that looks somewhat like:

def view_queryset(apps, schema_editor):
    Baz = apps.get_model('foo', 'Baz')

    return Baz.objects.values('a').filter(
        c__startswith='qux'
    ).annotate(
        d=ArrayAgg('c')
    )


class Migration(migrations.Migration):
    dependencies = [
        ('foo', '0001_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='Bar',
            fields=[
                ('a', models.OneToOneField(...)),
                ('d', ArrayField(base_field=models.TextField(), ...)),
            ],
            options={
                'managed': False,
            }
        ),
        CreateOrReplaceView('Bar', view_queryset),
    ]

We still need to have the CreateModel statement so Django knows about our model, but the important bit in this file is the CreateOrReplaceView, which references the callable.

Now for the actual migration operation.

class CreateOrReplaceView(migrations.Operation):
    def __init__(self, model, queryset_factory):
        self.model = model
        self.queryset_factory = queryset_factory

    def database_forwards(self, app_label, schema_editor, from_state, to_state):
        model = from_state.apps.get_model(app_label, self.model)
        queryset = self.queryset_factory(from_state.apps, schema_editor)
        compiler = queryset.query.get_compiler(using=schema_editor.connection.alias)
        sql, params = compiler.as_sql()
        sql = 'CREATE OR REPLACE VIEW {view_name} AS {query}'.format(
            view_name=model._meta.db_table,
            query=sql,
        )
        schema_editor.execute(sql, params)

The backwards migration is not quite a solved problem yet: I do have a working solution that steps up the stack to determine what the current migration name is, and then finds the previous migration that contains one of these operations for this model, but that’s a bit nasty.

There’s no (clean) way to inject ourself into the migration autodetector and “notice” when we need to generate a new version of the view, however we can leverage the checks framework to notify the user when our view queryset is out of date compared to the latest migration.

from django.apps import apps
from django.core.checks import register

@register()
def check_view_definitions(app_configs, **kwargs):
    errors = []

    if app_configs is None:
        app_configs = apps.app_configs.values()

    for app_config in app_configs:
        errors.extend(_check_view_definitions(app_config))

    return errors

And then we need to implement _check_view_definitions:

def get_out_of_date_views(app_config):
    app_name = app_config.name

    view_models = [
        model
        # We need the real app_config, not the migration one.
        for model in apps.get_app_config(app_name.split('.')[-1]).get_models()
        if not model._meta.managed and hasattr(model, 'get_view_queryset')
    ]

    for model in view_models:
        latest = get_latest_queryset(model)
        current = model.get_view_queryset()

        if latest is None or current.query.sql_with_params() != latest.query.sql_with_params():
            yield MissingViewMigration(
                model,
                current,
                latest,
                Warning(W003.format(app_name=app_name, model_name=model._meta.model_name), id='sql_helpers.W003'),
            )


def _check_view_definitions(app_config):
    return [x.warning for x in get_out_of_date_views(app_config)]

The last puzzle piece there is get_latest_queryset, which is a bit more complicated:

def get_latest_queryset(model, before=None):
    from django.db.migrations.loader import MigrationLoader
    from django.db import connection

    migration_loader = MigrationLoader(None)
    migrations = dict(migration_loader.disk_migrations)
    migration_loader.build_graph()
    state = migration_loader.project_state()
    app_label = model._meta.app_label
    root_node = dict(migration_loader.graph.root_nodes())[app_label]
    # We want to skip any migrations in our reverse list until we have
    # hit a specific node: however, if that is not supplied, it means
    # we don't skip any.
    if not before:
        seen_before = True
    for node in migration_loader.graph.backwards_plan((app_label, root_node)):
        if node == before:
            seen_before = True
            continue
        if not seen_before:
            continue
        migration = migrations[node]
        for operation in migration.operations:
            if (
                isinstance(operation, CreateOrReplaceView) and
                operation.model.lower() == model._meta.model_name.lower()
            ):
                return operation.queryset_factory(state.apps, connection.schema_editor())

This also has code to allow us to pass in a node (before), which limits the search to migrations that occur before that node in the forwards migration plan.

Since we already have the bits in place, we could also have a management command that creates a stub migration (without the queryset factory, that’s a problem I haven’t yet solved). I’ve built this into my related “load SQL from files” app.

This is still a bit of a work in progress, but writing it down helped me clarify some concepts.

Smart Lights or Smart Switches, or my philosophy of home automation

2020-02-22T20:00:36+00:00

I really like home automation. Being able to see when things are turned on, and being able to turn them off remotely, or build automations around events in the home is fun. I’ve built a garage door opener, a bunch of temperature, humidity and air pressure sensors, and played around with various light bulbs and switches.

I haven’t invested deeply in any platform: for a couple of reasons. I have some of the Sonoff devices, and like that it’s possible to flash my own firmware. This allows me to lock down the device so it can only connect to the one MQTT broker in my network: indeed, my IoT devices are all on a seperate network that has very strict limits.

But that’s not what I want to talk about today. I want to talk about smart lights and smart switches, and why switches are superior.

I don’t live by myself. I have a family, none of whom share my level of excitement about smart devices, but all of whom need to still turn lights on and off. They don’t have an Apple Watch (or iPhone, in most cases), and should not have to use one to turn the lights on.

In fact, I should not have to use my watch or phone to toggle the lights either. We have managed as a society to come up with a really simple system for controlling lights: you flick the switch, and the light toggles.

Direct control of lights using the existing switches is the number one priority of a smart light system. Having a smart bulb that you can turn off using Siri, but then means you need to turn the switch-off-and-then-back-on to get it to turn on is not acceptable.

Likewise, requiring someone to use a smart device to turn a light on is unacceptable.

Having a switch that prevents the smart light from being smart (ie, when it’s off, you have to use the switch to turn it on) is also unacceptable.

This makes existing smart bulbs a non-event for me. Even if you have a duplicate “smart” switch next to an existing switch, there’s still the chance someone will turn off the switch that needs to stay on.

What is a much better solution is to have the switch itself be smart. In most cases, that means the switch will no longer be mechanical, although it could be a press button. There are a bunch of smart switches that perform this way: they work manually, but also still allow an automated or remote toggle of the state.

These switches will not work in my home.

Pretty much all of these (except one exception from Sonoff, see this video for a really nice description of how this works) require a neutral wire at the switch. It seems that my house is wired up with the neutral at the light only, and then only a pair of wires (live, and switched live) that travel down to the switch. Thus, the switch is wired in series with the live wire, and the neutral is only connected directly to the fitting. Jonathon does a really good job in the above-linked video of describing the problem.

There is an alternative solution. Sonoff also make another device, the Sonoff Mini. This one is pretty small (much smaller than the Sonoff Basic), and can be wired up and programmed to behave just like a traditional hallway switch: toggling either the manual or smart switch will result in the lights toggling. The nice thing about these is that they have the new DIY mode (just pop them open, add a jumper, and you can flash them without having to connect header pins). In fact, I even wrote a script to automate the process. It’s not quite perfect, but it’s been good for flashing the five of these that I currently own.

You can then have the mini connected up to the light (at least, in other countries you can do this yourself: in Australia this is illegal unless you are an electrician, so be aware of that), and have the switch connected just to the switch pins on the mini. Bingo, smart switches the right way. When the smart fails, the switch still works the same as they have for generations.

(Unless the microcontroller itself dies, but that’s a problem I have not as yet solved).

As an aside, I wanted to comment on the setup from Superhouse. It’s quite nifty: all of the switches are low voltage, and control relays back in the switchboard. However, it relies on the logic running in a computer (and running JavaScript!) to connect which switch to which light.

This to me feels wrong. It’s better than having those connections over WiFi, but it still means there is a single point of failure. I think the architecture I have used - smart switches that are independent, and if they fail to connect to WiFi continue to work just the same as an old dumb switch - is superior. It’s more like the philosophy I have with web pages: the pages should still work when all of the JavaScript breaks, but when it doesn’t break, they can be nicer (and have some cool affordances).

To that end, I’ve tried to make each little module of my smart home somewhat independent. For instance, having an IR sensor connected to the same microcontroller that controls the lights that sensor means that even if WiFi breaks, the lights still behave exactly the same way.

Adding a PIR sensor to an Arlec Smart LED Strip

2020-02-22T19:27:30+00:00

I built a shed on Australia Day.

Actually, I built two of them over the Australia Day Long Weekend. I bought them from easyshed.com.au, and they were pretty nifty. Specifically, I bought a couple of the off the wall sheds, to attach them to the wall of my garage. There’s already a concrete path there that I was able to use as the floor.

One of the sheds is longer than the others, and is used for storage of tools. Mostly I need to get stuff during the day, but sometimes at night I may need to access something from in there. So, I bought a 2m Grid Connect LED Strip Light. It’s colour changing, but I’m not really that fussed about the colours: this one has a Warm White LED too, so that’s better than trying to twiddle the colours to get the right temperature.

Being Grid Connect, it’s really Tuya, which means it’s flashable. So I did. I’ve got a nice esphome firmware on there, which hooks it up to HomeKit via my MQTT bridge.

I wasn’t really able to get the colour parts working correctly, so I just disabled those pins, and use the light in monochromatic mode.

However, being the tinkerer I am, I opened it up to have a look inside. It’s got one of the standard Tuya mini-boards, in fact a TYWE3S.

This exposes a bunch of pins, most of which were in use:

GPIO 0: (Used during boot)
GPIO 2: UART0 (TX)
GPIO 4: Red LED channel
GPIO 5: Warm White LED channel
GPIO 12: Green LED channel
GPIO 13: unused
GPIO 14: Blue LED channel
GPIO 15: (Used during boot)
GPIO 16: unused

Because the LED strip runs at 12V, I was able to use this to power a PIR sensor, which I then hooked up to GPIO 13. I also tried to connect a DS18B20 temperature sensor to GPIO 16, but was not able to get it to be recognised. From the page linked above, perhaps I needed a pull-up resistor, however, I didn’t really need a temperature sensor.

Having a PIR sensor on the LED strip is worthwhile, however. Otherwise, you’d need to manually turn the lights on when going into the shed.

Having a sensor light is fantastic, but at times you might want the light to stay on, even when it does not detect motion. To achieve this, I have a global variable manual_override, that is set when the light is turned on “manually” (using HomeKit, as there is no physical switch). When this variable is set, the light will not turn off when motion is no longer detected.

I also found it was simpler to have the motion detector (binary_sensor in esphome) have a “delayed off” filter for the period I wanted the light to stay on for, rather than try to manage that in conjunction with the manual override.

The full YAML follows:

esphome:
  name: $device_name
  platform: ESP8266
  board: esp01_1m
  on_loop:
    then:
      - if:
          condition:
            not:
              mqtt.connected:
          then:
            - globals.set:
                id: has_connected_to_mqtt
                value: 'false'

globals:
  - id: has_connected_to_mqtt
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: manual_override
    type: bool
    restore_value: no
    initial_value: 'false'

light:
  - platform: monochromatic
    name: "White"
    id: white
    output: white_channel
    restore_mode: ALWAYS_OFF
    default_transition_length: 0.25s

output:
  - platform: esp8266_pwm
    pin:
      number: GPIO5
    id: white_channel
    inverted: False

sensor:
  - platform: wifi_signal
    name: "WiFi signal sensor"
    update_interval: 5min

binary_sensor:
  - platform: gpio
    pin: GPIO13
    name: "PIR Sensor"
    device_class: motion
    id: pir
    filters:
      - delayed_off: 15s
    on_press:
      then:
        - mqtt.publish:
            topic: HomeKit/${device_name}/MotionSensor/MotionDetected
            payload: "1"
        - if:
            condition:
              light.is_off: white
            then:
              - light.turn_on: white
              - mqtt.publish:
                  topic: HomeKit/${device_name}/Lightbulb/On
                  payload: "1"
    on_release:
      - mqtt.publish:
          topic: HomeKit/${device_name}/MotionSensor/MotionDetected
          payload: "0"
      - if:
          condition:
            lambda: 'return id(manual_override);'
          then:
            - logger.log: "Manual override prevents auto off."
          else:
            - logger.log: "Turning off after motion delay."
            - if:
                condition:
                  - light.is_on: white
                then:
                  - light.turn_off: white
                  - mqtt.publish:
                      topic: HomeKit/${device_name}/Lightbulb/On
                      payload: "0"

ota:

logger:

mqtt:
  broker: "mqtt.lan"
  discovery: false
  topic_prefix: esphome/${device_name}
  on_message:
    - topic: HomeKit/${device_name}/Lightbulb/Brightness
      then:
        - logger.log: "Brightness message"
        - globals.set:
            id: manual_override
            value: 'true'
        - light.turn_on:
            id: white
            brightness: !lambda "return atof(x.c_str()) / 100;"
        - globals.set:
            id: has_connected_to_mqtt
            value: 'true'

    - topic: HomeKit/${device_name}/Lightbulb/On
      payload: "1"
      then:
        - logger.log: "Turn on message"
        - if:
            condition:
              and:
                - binary_sensor.is_off: pir
                - lambda: "return id(has_connected_to_mqtt);"
            then:
              - globals.set:
                  id: manual_override
                  value: 'true'
              - logger.log: "Manual override enabled"
            else:
              - globals.set:
                  id: manual_override
                  value: 'false'
              - logger.log: "Manual override disabled"
        - light.turn_on:
            id: white
        - globals.set:
            id: has_connected_to_mqtt
            value: 'true'
    - topic: HomeKit/${device_name}/Lightbulb/On
      payload: "0"
      then:
        - logger.log: "Turn off message"
        - if:
            condition:
              lambda: 'return id(has_connected_to_mqtt);'
            then:
              - light.turn_off:
                  id: white
              - globals.set:
                  id: manual_override
                  value: 'false'
              - logger.log: "Manual override disabled"
        - globals.set:
            id: has_connected_to_mqtt
            value: 'true'

  birth_message:
    topic: HomeKit/${device_name}/Lightbulb/On
    payload: "1"
  will_message:
    topic: HomeKit/${device_name}/Lightbulb/On
    payload: "0"

There’s also some logic in there to prevent the light turning off and then back on when it first connects to the MQTT broker and is already turned on.

Heat Pump Hot Water

2020-01-17T20:57:41+00:00

Heat pumps seem like magic. You use a certain amount of energy (let’s go with 1kWh), and you use this to extract heat from the ambient air, and this allows you to apply a multiple (let’s go with 3kWh) of your original energy input.

This is significantly better than an element heater: in that situation, you can only apply a smaller multiple, depending upon the efficiency of the element.

Similarly, burning gas to heat up water is less efficient - you don’t quite get to apply all of the energy from the amount of gas you burn to the water. I believe boilers and element heaters are around 85% efficient.

This is also how reverse cycle air conditioners work - basically a transfer fluid is used to extract the heat from the ambient air, and either push this cooled air, or use the energy extracted to heat up other air to push around.

But heat pumps, unlike a gas heater, can use renewable energy as the source of heating water. This interests me greatly.

Currently, I spend around $100/month on gas - most of this is probably on my gas hot water, as a gas stove would use a small fraction. For my calculations, I went with 80% hot water, 20% stove. This would not reduce my gas bill to $20/month though, because of daily tariffs (the bane of low-consumption systems). Instead, an 80% reduction in usage would drop my bills to between $35 and $40 per month. Indeed, it would save me almost exactly $2/day.

But the other factor to take into account is how much more of my solar generation I would be using to run the heat pump. To do this, I increased my consumption measurements from my previous calculations by 4kWh (800W for 5 hours, picking the peak solar generation time).

This resulted in a projected increase in my power bill of about $1.10 per day.

So, having a heat pump would probably save me less than $1 per day.

With a $4200 installation price (including STC credits), this looks like a payback time in excess of 12 years. Which is far larger than the warranty period of the system - this is looking marginal.

If I double my PV system size, and install a heat pump, this should result in my energy costs reducing by around $3.40 per day. This is about $1240 annually, but then there are two capital costs to cover: closer to $10k. This reduces the payback to around 8 years.

Interestingly, this is better than the payback of me just adding more solar panels - I used a cost of $6000 for the PV upgrade, and $4000 for the Heat Pump here, rather than the $5k I used in a previous blog post.

But probably still not good enough, right now.

Arlec Powerboard esphome

2020-01-13T20:31:36+00:00

I bought one of the Grid Connect powerboards from Bunnings last week, and flashed it with a custom firmware.

The model I bought was the PB89HA, which is the one with 5 sockets (one of which is not switchable).

The button is on GPIO3, and the LED is an inverted GPIO1.

The four relays are on GPIO5, GPIO4, GPIO13 and GPIO12, starting with the one closest to the button.

Since there is only one button, and four switchable outlets, I had to come up with a mechanism for controlling all of them. I ended up (after playing around with single-through-quadruple click) with a single click for toggling the first relay (nearest the button). Then, a medium (between 1 and 2 second) press turns all relays off, and a long (greater than 2 second) press turns them all on.

This is not really ideal, as there is no way to toggle just relay 2-4 without using some sort of external control - which goes against my ethos with respect to smart home.

Having said that, I’m not actually sure how I’m going to use this board…I don’t really have a bunch of things that are potentially close together that need individual control. I guess I could have it turn off everything in the entertainment unit except the PVR - that might be a way to save power overnight. I’d want the button more accessible than the powerboard that currently controls them though.

Anyway, the base YAML file follows - be aware that this does not include wifi, and would need to be included in an actual config file (with a device name defined).

esphome:
  name: $device_name
  platform: ESP8266
  board: esp01_1m
  on_boot:
    - light.turn_on:
        id: led
        brightness: 20%

binary_sensor:
  - platform: status
    name: "Status"

  - platform: gpio
    pin:
      number: GPIO3
      inverted: true
      mode: INPUT_PULLUP
    name: button
    on_multi_click:
      - timing:
        - ON for at least 1s
        - OFF for at least 0.2s
        then:
          - switch.turn_off: relay_0
          - switch.turn_off: relay_1
          - switch.turn_off: relay_2
          - switch.turn_off: relay_3

      - timing:
        - ON for at least 2s
        - OFF for at least 0.2s
        then:
          - switch.turn_on: relay_0
          - switch.turn_on: relay_1
          - switch.turn_on: relay_2
          - switch.turn_on: relay_3

      - timing:
        - ON for at most 0.5s
        - OFF for at least 0.2s
        then:
          - switch.toggle: relay_0

switch:
  - platform: gpio
    id: relay_0
    pin: GPIO5
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/0/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/0/On
         retain: ON
         payload: 0

  - platform: gpio
    id: relay_1
    pin: GPIO4
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/1/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/1/On
         retain: ON
         payload: 0

  - platform: gpio
    id: relay_2
    pin: GPIO13
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/2/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/2/On
         retain: ON
         payload: 0

  - platform: gpio
    id: relay_3
    pin: GPIO12
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/3/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/3/On
         retain: ON
         payload: 0

light:
  - platform: monochromatic
    output: led1
    id: led
    restore_mode: ALWAYS_ON

output:
  - platform: esp8266_pwm
    pin:
      number: GPIO1
    id: led1
    inverted: True

sensor:
  - platform: wifi_signal
    name: "WiFi signal sensor"
    update_interval: 5min

ota:

logger:

mqtt:
  broker: "mqtt.lan"
  discovery: false
  topic_prefix: esphome/${device_name}
  on_message:
    - topic: HomeKit/${device_name}/Outlet/0/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_0
    - topic: HomeKit/${device_name}/Outlet/0/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_0

    - topic: HomeKit/${device_name}/Outlet/1/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_1
    - topic: HomeKit/${device_name}/Outlet/1/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_1

    - topic: HomeKit/${device_name}/Outlet/2/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_2
    - topic: HomeKit/${device_name}/Outlet/2/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_2

    - topic: HomeKit/${device_name}/Outlet/3/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_3
    - topic: HomeKit/${device_name}/Outlet/3/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_3

Fronius Dashboard 0.3.0

2019-12-20T23:19:15+00:00

I released a new docker version of Fronius Dashboard a few days ago. It now allows you to specify your inverter power (in Watts), using an environment variable. The default value is 5000.

docker stop fronius-dashboard

docker rm fronius-dashboard

docker pull schinckel/fronius-dashboard

docker run \
  --publish 4000:4000 \
  --detach \
  --restart always \
  --name fronius-dashboard \
  --env INVERTER=<ip-address-or-hostname> \
  --env INVERTER_SIZE=<inverter-size-in-watts> \
  schinckel/fronius-dashboard:latest

If your inverter is 5kW, then you don’t need to apply this update.

DNS Redirection and badly behaved clients

2019-12-20T22:53:05+00:00

A little while ago I repurposed a spare Raspberry Pi Zero (the one without WiFi), plugging it directly into my router, and setting it up as a pi hole DNS server.

I also changed the firewall rules in my network so that clients may only send DNS requests to that IP address (and it may then send DNS requests to whomever it chooses). And this is fine.

In fact, I didn’t really stop other devices sending DNS requests elsewhere, I just make it that any other devices that actually send out a DNS request have those packets forwarded to the pi hole. Which is even better.

Except: DNS responses also appear to indicate the server that fulfilled the request. Depending upon the client, this will either have no side effect, be logged as a strangeness, or fail.

I’ve only had a couple of devices that apparently have a hard-coded DNS server in their firmware, and to be honest, I’m not really that fussed about those devices. That hasn’t caused an issue for our day-to-day use.

However, during some tweaking (I was trying to get the local IPv6 DNS server to be set correctly by the DHCP setup), I managed to make it so it was no longer setting the pi hole as the DNS server. And for most devices, this was still fine. They may have logged extra errors about the incorrect server returning results, but everything was good.

Or so I thought.

See, some devices still had their old DHCP lease, and it was only when this expired and was renewed that they began to pick up incorrect DNS settings.

So that was how, at about 10 AM today, my inverter stopped uploading data to PVOutput.

Of course, I only figured out many hours later that this was the cause.

I could see that it was still making the DNS request (and the pi hole was responding), but the “Push Service” configurations were all reporting errors uploading. But the connection to Solar.Web was fine: and I was able to use my Fronius Dashboard, so I knew there was nothing wrong with the WiFi connection.

It turns out that the IP address for the Solar.Web upload must be hard-coded, because that was never being asked for by the inverter. And it was following the third path of DNS response from a different server: fail.

To be fair, one could argue this is the correct behaviour: it could prevent DNS poisoning - although, surely a nefarious DNS server would just respond with whatever IP address it saw was in the incoming packets.

Anyway, after trying a bunch of things (restarting the DataManager, and playing around with network configuration), I managed to get uploads working again, and figured out why the DNS setting was not being propogated.

Do I Need More Panels?

2019-12-15T23:20:03+00:00

Last time, I showed how it’s possible to do some analysis of different retailers and pricing, based on actual consumption and generation: or more specifically, actual feed-in and feed-out. But this is all stuff that is possible to do using a spreadsheet: indeed, until I finally got around to putting this into some nice SQL views, I was using a spreadsheet, and manually updating the values every day to keep it up to date. Whilst this was a fun thing to do every morning, it really was something I needed to automate. It is not necessary to have more than daily totals for the purposes of calculating costs: indeed all retailers use daily totals (or monthly averages) to determine the tariffs that need to apply.

But, if you want to examine your solar generation, and determine if you should have more (or fewer) panels, then you need more granular information.

In my specific case, I have panels that are covering around 1/5 of my roof space. My roof is almost flat, so I can fairly safely state that if I had twice as many panels (or 5 times as many panels), then I’d have generation that increases by that factor. I understand that this is not the case for many homes: anyone who has a highly raked roof, and especially someone who has to deal with an East-West split of panels is not going to be able to make the same assumptions.

However, even in the case of my flat roof, there’s complications. I only have single phase power, and SA Power Networks limits the export per-phase to 5kW. That’s the size of my inverter, so, if I was generating full power, and not using any, then it would appear at first glance to be not worth having any extra panels.

But there are two assumptions in that paragraph that almost never hold true. Firstly, there is almost always someone home at my house during the day, so we use a reasonable amount of energy. This is actually beneficial: we can have the dishwasher and washing machine (and sometimes even the oven) on during the day, and my Fronius Dashboard makes it easy to see if we are generating surplus. So this means we can make better use of the power we are generating, rather than running those appliances at night or when we are not generating enough energy.

The other one is about maximum generation. Whilst today when I looked, we were actually generating 5kW, for much of the year, we don’t get that close to that value at all, let alone for the full day. Especially during winter (when at times we do not even generate enough to supply our heating needs during the daylight hours). So, we could have more panels to generate more energy during winter, and also during summer, when we may find we run the Air Conditioner during the peak time.

However, with more than 5kW of generation, it is possible at times we would be generating more energy than we were using, and have a surplus of more than 5kW. This would mean we would only be able to export 5kW, and the rest would be wasted.

It would be interesting, though, to see what having twice as many panels did to our annual import/export amounts.

The process of calculating this requires us to know the generation and consumption values: for the purposes of these calculations, the import and export values are irrelevant: we will be calculating new values. It is possible to ask the inverter for generation values, and depending upon your local hardware, you can get the import and export values: from this we can calculate the consumption. Alternatively, you could get the import and export values from your retailer or supplier, and perform the same calculations based on your generation:

consumption = import + generation - export

Once we have that value for every interval, we can look at something that gives us “what if?” capacity.

CREATE OR REPLACE VIEW consumption AS
SELECT timestamp,
       import.energy + generation.energy - export.energy AS energy
  FROM generation
 INNER JOIN import USING (timestamp)
 INNER JOIN export USING (timestamp);

Since we do not care about shorter-than-daily intervals for the purposes of cost calculations, my function will return per-day values: specifically they will use the fact that postgres can define functions that return a set of values of a specific type; that type can even be the implicit type that is created for each table or view. I have a view called daily_summary, that looks like (including the summary view for reference):

CREATE OR REPLACE VIEW summary AS (
    SELECT timestamp,
           generation.energy AS generation,
           import.energy AS import,
           export.energy AS export,
           import.energy + generation.energy - export.energy AS consumption
      FROM generation
     INNER JOIN import USING (timestamp)
     INNER JOIN export USING (timestamp)
);

CREATE OR REPLACE VIEW daily_summary AS (
    SELECT timestamp::DATE AS date,
           SUM(generation) AS generation,
           SUM(import) AS import,
           SUM(export) AS export,
           SUM(consumption) AS consumption
      FROM summary
     GROUP BY timestamp::DATE
);

As such, I can write my function so it returns a set of daily_summary:

CREATE OR REPLACE FUNCTION panel_factor(FLOAT)
RETURNS SETOF daily_summary AS $$

WITH raw AS (
    SELECT timestamp,
           generation * $1 AS generation,
           consumption
      FROM summary
),
limited AS (
    SELECT timestamp::DATE AS date,
           generation,
           GREATEST(0, consumption - generation)::INTEGER AS import,
           CASE WHEN generation <= consumption THEN 0
                WHEN generation - consumption < 5000 / 12  THEN generation - consumption
                ELSE 5000 / 12 END::INTEGER AS export,
           consumption
      FROM raw
)
SELECT date, SUM(generation), SUM(import), SUM(export), SUM(consumption)
FROM limited
GROUP BY date
;

$$ LANGUAGE SQL;

This takes an argument that is a floating point value (I may want to increase my PV size by 50%, for instance), and multiplies the generation for each interval by this factor. It then works out what the import and export values would likely be for each interval. This is probably the least accurate part, because it’s possible that the initial data had both export and import during a given interval. However, I’ve ignored this, and just assumped that the generation and consumption for each interval are approximately stable.

Then we truncate our export to 5000 / 12, because my data is 5 minute intervals, so 5kW for an hour would be 5kWh, so 5kW for 5 minutes would be 5000 / 12 = 416.67kWh. This would necessarily change if you had different export limits, or a different time interval. Keep in mind that the larger the time interval, the less safe the assumption about consumption and generation stability across the period is.

From there, we add these values up for each day.

We can run this function and see what it does for our import/export amounts for some days. In my case, I can see that having 2x panels makes a fairly small difference during December, however having 2.5 times as many panels in June would mean I was exporting about as much as I would import: compared to last June when I imported almost 5x as much energy as I exported.

Interesting.

However, we can’t just look at import/export amounts: I only get paid 50% for a kWh I export compared to how much I import for, and then there are daily tariffs too. So, we want a function that compares costs: and to make it easier to look at, we will just use one that calculates costs over the whole period:

CREATE OR REPLACE FUNCTION compare(factor FLOAT)
RETURNS TABLE (name TEXT, cost INTEGER, difference INTEGER) AS $$

WITH old_costs AS (
    SELECT name,
           SUM(cost) AS cost
      FROM (
          SELECT retailer.name, cost(retailer, import::INTEGER, export::INTEGER)
            FROM daily_summary, retailer
      ) _
     GROUP BY name
), new_costs AS (
    SELECT name,
           SUM(cost) AS cost
      FROM (
          SELECT retailer.name, cost(retailer, import::INTEGER, export::INTEGER)
            FROM panel_factor(factor), retailer
      ) _
     GROUP BY name
)

SELECT name,
       new_costs.cost::INTEGER AS cost,
       (new_costs.cost - old_costs.cost)::INTEGER AS difference
  FROM old_costs INNER JOIN new_costs USING (name)


$$ LANGUAGE SQL STRICT;

And we can run this a few times with different values, to see what that does to our costs.

SELECT * FROM compare(2) ORDER BY cost;

name	cost	difference
AGL Essentials	210	-688
AGL Solar Savers	234	-763
Origin Energy	276	-695
AGL Residential Essentials	302	-615
Click Banksia Solar	303	-726
Lumo Basic	323	-643
SA Simply RAA VPP	334	-645
Power Direct Residential Saver	427	-554
Energy Australia Total Plan Home	437	-664
Powerband Home Flat + Solar FIT	447	-510
Red Energy Living Energy Saver	547	-547
PowerShop Shopper Market	650	-418
Kogan	702	-377
Alinta Energy No Fuss (Single Rate)	714	-466
Origin Energy Econnex	728	-431
Diamond Energy Everyday Renewable Saver	732	-573
Blue NRG DMO - General Usage	1476	-460

Interesting: AGL is still cheapest. What about if we had 3x the panels:

name	cost	difference
AGL Solar Savers	-120	-1117
AGL Essentials	-109	-1007
Origin Energy	-45	-1017
Click Banksia Solar	-33	-1062
AGL Residential Essentials	18	-899
Lumo Basic	26	-940
SA Simply RAA VPP	36	-944
Energy Australia Total Plan Home	132	-970
Power Direct Residential Saver	173	-808
Powerband Home Flat + Solar FIT	212	-745
Red Energy Living Energy Saver	297	-798
PowerShop Shopper Market	460	-608
Diamond Energy Everyday Renewable Saver	472	-834
Alinta Energy No Fuss (Single Rate)	503	-677
Kogan	533	-547
Origin Energy Econnex	534	-625
Blue NRG DMO - General Usage	1277	-659

Oh, AGL is still cheapest: however it’s now a different plan that wins out.

And if I had 5x the panels?

name	cost	difference
AGL Solar Savers	-505	-1502
AGL Essentials	-457	-1354
Click Banksia Solar	-398	-1427
Origin Energy	-395	-1366
Lumo Basic	-297	-1264
AGL Residential Essentials	-290	-1207
SA Simply RAA VPP	-288	-1268
Energy Australia Total Plan Home	-199	-1300
Power Direct Residential Saver	-101	-1082
Powerband Home Flat + Solar FIT	-41	-998
Red Energy Living Energy Saver	27	-1067
Diamond Energy Everyday Renewable Saver	194	-1112
PowerShop Shopper Market	258	-810
Alinta Energy No Fuss (Single Rate)	279	-901
Origin Energy Econnex	329	-830
Kogan	353	-727
Blue NRG DMO - General Usage	1074	-862

Neat. Still AGL Solar Savers. That does have the highest feed-in tariff, so that makes sense.

Now, that gives me the amount I would save on that plan, but how much would it save me compared to my current plan? Well, that’s around $890 so far, so that’s about $2400.

So, we can aggregate this into a table showing us how much we would save by having different increases in panel amounts:

SELECT factor,
       MIN(cost) AS cost,
       890 - MIN(cost) AS saving
  FROM generate_series(1, 5, 0.5) factor, compare(factor)
 GROUP BY factor
 ORDER BY factor;

factor	cost	saving
1	890	0
1.5	474	416
2.0	210	680
2.5	28	862
3.0	-120	1010
3.5	-243	1133
4.0	-345	1235
4.5	-431	1321
5.0	-505	1395

We can plot this:

That certainly shows the diminishing returns!

But what about looking at how many years a given system of that size would take to pay off given this level of saving?

Using a fairly conservative value of $5000 for doubling my panel/inverter amount (6.6kW/5kW inverter), this gives me a payback table that looks something like:

Panel factor	Payback years
1.5	6
2.0	7
2.5	9
3.0	10
3.5	11
4.0	12
4.5	13
5.0	14

That is interesting. It is worth noting that this is based on costs since March, so it doesn’t take into account the peak solar generation time (summer) here in Australia. But it does mean that doubling my panels could take twice as long to pay back as I believe my initial investment is going to take.

So, should I get more panels?

Probably.

I’m going to wait until I have a full year of proper data, and then re-evaluate. I always planned to get more at some point - it seems to me that the over time FITs will begin to drop (I mean, they have already: I have a friend who is grandfathered in to a $0.40c FIT, but she can’t make her system larger without losing that tariff), and the prices of batteries will drop even further.

Having a battery added into the mix could reduce even further the import costs, but at an obvious reduction of export energy too: and the big concerns about batteries right now are that they are marginal in reducing power bills by enough to break even: their payback time currently exceeds their warranty period; and more worryingly, battery chemistry suggests their performance will degrade significantly over that period too.

Oh, and the other thing I can do is run a simulation of what my power bill would look like without any PV generation. Pretty much every retailer would have been well over $2000, which means I really have saved a bunch of money (even though, as already mentioned, I haven’t had the system for a whole summer). Perhaps the most interesting thing is that my power bills from last year were still several hundred dollars more than what these calculations show - that means that even having the panels means as a household we have been far more aware of how much power we are using (even during the day).

PV Solar panel analysis in postgres

2019-12-13T23:10:55+00:00

Australian (and especially South Australian) electricity is expensive. That is probably part of the reason that SA is leading the world in residential rooftop solar.

But it is hard to negotiate the electricity retail market. Knowing your daily usage and feed-in enables you to determine which retailer you should choose, but only if you have the tools available to you to compare what your costs would have been with each retail plan.

And this does not even begin to address the question of if it is cost-effective to increase the size of your PV generation system: most supply companies (who, as it turns out, are different entities to the retailers) have per-phase limits of 5kW: that is, you may never feed back into the grid more than 5kW at any point in time.

Since I had my Fronius Smart Meter installed, I have been able to run reports and get my daily import and export values: these are also available through my retailer, and even through SA Power Networks, the distributor.

I have my data stored in three tables:

REATE TABLE generation (
  "timestamp" TIMESTAMP UNIQUE PRIMARY KEY,
  "energy" FLOAT
);

CREATE TABLE import_cumulative (
  "timestamp" TIMESTAMP UNIQUE PRIMARY KEY,
  "energy" INTEGER
);

CREATE TABLE export_cumulative (
  "timestamp" TIMESTAMP UNIQUE PRIMARY KEY,
  "energy" INTEGER
);

This is really only necessary because I need to take the cumulative values in the import and export tables, and turn them into absolute values for that period:

CREATE OR REPLACE VIEW import AS (
  SELECT timestamp,
         energy - LAG(energy, 1) OVER (ORDER BY timestamp) AS energy
    FROM import_cumulative
   ORDER BY timestamp
);

CREATE OR REPLACE VIEW export AS (
  SELECT timestamp,
         energy - LAG(energy, 1) OVER (ORDER BY timestamp) AS energy
    FROM export_cumulative
   ORDER BY timestamp
);

Now, from these views, it is possible to do some modelling of what different retailers would cost, but they generally need stuff per-day, so we can create a view that combines them. Since I will be doing stuff later that also needs consumption and generation, I will also include them:

CREATE OR REPLACE VIEW consumption AS
SELECT timestamp, 
       import.energy + generation.energy - export.energy AS energy
  FROM generation
 INNER JOIN import USING (timestamp)
 INNER JOIN export USING (timestamp);

And our summary (for each interval):

CREATE OR REPLACE VIEW summary AS (
    SELECT timestamp,
           generation.energy AS generation,
           import.energy AS import,
           export.energy AS export,
           import.energy + generation.energy - export.energy AS consumption
      FROM generation
     INNER JOIN import USING (timestamp)
     INNER JOIN export USING (timestamp)    
);

Getting close to having some usable data there now. What about daily summaries?

CREATE OR REPLACE VIEW daily_summary AS (
    SELECT timestamp::DATE AS date,
           SUM(generation) AS generation,
           SUM(import) AS import,
           SUM(export) AS export,
           SUM(consumption) AS consumption
      FROM summary
     GROUP BY timestamp::DATE
);

That gives us per-day summaries of our generation, import and export.

Here are some values from my system:

SELECT * FROM daily_summary LIMIT 10;

date	generation	import	export	consumption
2019-03-07	29851.915	7587	24191	13247.915
2019-03-08	14033.2047222222	7578	9156	12455.2047222222
2019-03-09	28074.8316666667	5746	20961	12859.8316666667
2019-03-10	15239.6936111111	7669	9144	13764.6936111111
2019-03-11	21101.1441666667	6528	12657	14972.1441666667
2019-03-12	28231.4280555556	5955	22382	11804.4280555556
2019-03-13	22880.3077777778	6040	17411	11509.3077777778
2019-03-14	30440.8319444444	6658	24791	12307.8319444444
2019-03-15	30141.3888888889	6576	26227	10490.3888888889
2019-03-16	29286.4413888889	7336	22224	14398.4413888889

(10 rows)

Right. So now we want to be able to look at costs for different retailers. There is a magnificent resource for getting this data: Energy Made Easy. The rates differ for different postcodes (WAT): I’m not sure if they are the same across a whole state or not. Probably not.

Anyway, it turns out that there are some retailers who have three different rates, depending upon your usage during a day. It’s not clear to me if this value is averaged across the whole month when you are on a smart meter: I know it is when you are on one of the old spinning disk meters. I assume it would be calculated on each day, as I think that works out as more expensive for the customer. Diamond Energy appears to have per-month limits, and four different rates, so I just pro-rated them to a 30 day month. I did not bother modelling the fourth rate, as I established fairly early on that they are not going to be close to being the cheapest. I may implement that later, just for completeness.

So, to be able to calculate costs for all retailers, we need to store up to three rates, and the two kWh thresholds that trigger those limits. Note that the second

CREATE TABLE retailer (
    name TEXT PRIMARY KEY,
    supply_charge NUMERIC NOT NULL,
    kWh_rate_1 NUMERIC NOT NULL,
    kWh_limit_1 NUMERIC,
    kWh_rate_2 NUMERIC,
    kWh_limit_2 NUMERIC,
    kWh_rate_3 NUMERIC,
    pay_on_time_discount NUMERIC,
    feed_in_tariff NUMERIC NOT NULL
);

ALTER TABLE retailer 
    ADD CONSTRAINT rate_2_requires_limit_1 
        CHECK (kWh_rate_2 IS NULL OR kWh_limit_1 IS NOT NULL),
    ADD CONSTRAINT rate_3_requires_limit_2 
        CHECK (kWh_rate_3 IS NULL OR kWh_limit_2 IS NOT NULL),
    ADD CONSTRAINT rate_3_requires_rate_2 
        CHECK (kWh_rate_3 IS NULL OR kWh_rate_2 IS NOT NULL);

The three check constraints just ensure that we do not accidentally insert a rate where there is no lower tier rate, or no limit in place to apply it from.

Here is a query that inserts values for the retailers for my postcode:

INSERT INTO retailer (
  name, supply_charge, 
  kWh_rate_1, kWh_limit_1, 
  kWh_rate_2, kWh_limit_2, 
  kWh_rate_3, 
  feed_in_tariff, 
  pay_on_time_discount)
VALUES 
  ('AGL Essentials', 0.8855, 0.3575, NULL, NULL, NULL, NULL, 0.1630, NULL),
  ('Origin Energy', 0.9033, 0.4002, 4, 0.4272, NULL, NULL, 0.1800, 0.10),
  ('AGL Solar Savers', 0.8910, 0.4038, NULL, NULL, NULL, NULL, 0.1800, NULL),
  ('Click Banksia Solar', 0.9966, 0.3941, NULL, NULL, NULL, NULL, 0.1700, NULL),
  ('Power Direct Residential Saver', 0.8910, 0.4038, NULL, NULL, NULL, NULL, 0.1420, 0.13),
  ('SA Simply RAA VPP', 0.9863, 0.3600, NULL, NULL, NULL, NULL, 0.1500, NULL),
  ('Kogan', 0.9566, 0.3154, NULL, NULL, NULL, NULL, 0.0748, NULL),
  ('PowerShop Shopper Market', 1.1912, 0.3760, NULL, NULL, NULL, NULL, 0.1020, 0.15),
  ('AGL Residential Essentials', 0.7752, 0.3512, NULL, NULL, NULL, NULL, 0.1420, NULL),
  ('Diamond Energy Everyday Renewable Saver', 0.9565, 0.3659, 3.33, 0.3948, 11.1, 0.4285, 0.12, NULL),
  ('Red Energy Living Energy Saver', 0.9790, 0.3641, NULL, NULL, NULL, NULL, 0.1200, NULL),
  ('Blue NRG DMO - General Usage', 0.7810, 0.3736, 10.96, 0.5775, NULL, NULL, 0.0680, NULL),
  ('Energy Australia Total Plan Home', 0.8580, 0.4070, NULL, NULL, NULL, NULL, 0.15, NULL),
  ('Alinta Energy No Fuss (Single Rate)', 0.9130, 0.3317, 3.29, 0.3695, NULL, NULL, 0.0950, NULL),
  ('Lumo Basic', 1.0120, 0.3539, NULL, NULL, NULL, NULL, 0.15, NULL),
  ('Powerband Home Flat + Solar FIT', 1.0322, 0.3066, 10.95, 0.3139, NULL, NULL, 0.1150, NULL),
  ('Origin Energy Econnex', 0.9081, 0.4024, 10.96, 0.4294, NULL, NULL, 0.1000, 0.15)
  ON CONFLICT DO NOTHING
;

To calculate the cost for a day, we need to add the daily supply charge, and the amount of import that applies at each rate multiplied by that rate, minus the FIT (feed-in tariff) for the export amount. Because some retailers have a “pay on time discount” (which is really just a late-payment penalty, but that is another blog post waiting to happen), it’s important to note that this discount applies to the whole bill - meaning it needs to be applied after the feed-in tariff has been subtracted.

Here is a function that calculates a daily cost, given a retailer record and import and export values:

CREATE OR REPLACE FUNCTION cost(retailer RETAILER, import INTEGER, export INTEGER)
RETURNS NUMERIC AS $$

SELECT COALESCE(1 - retailer.pay_on_time_discount, 1.0) * (retailer.supply_charge +
       -- rate 3
       CASE WHEN retailer.kWh_limit_2 IS NULL THEN 0
            WHEN import <= retailer.kWh_limit_2 THEN 0
            ELSE import - retailer.kWh_limit_2
       END / 1000.0 * COALESCE(retailer.kWh_rate_3, 0) +
       -- rate 2: hours between limit 1 and limit 2
       CASE WHEN retailer.kWh_limit_1 IS NULL THEN 0
            WHEN retailer.kWh_limit_2 IS NULL THEN GREATEST(import - retailer.kWh_limit_1, 0)
            WHEN import < retailer.kWh_limit_1 THEN 0
            ELSE LEAST(import, retailer.kWh_limit_2) - retailer.kWh_limit_1
       END / 1000.0 * COALESCE(retailer.kWh_rate_2, 0) +
       -- rate 1: up to limit 1
       CASE WHEN retailer.kWh_limit_1 IS NULL THEN import
            ELSE LEAST(import, retailer.kWh_limit_1)
       END / 1000.0 * retailer.kWh_rate_1 -
       retailer.feed_in_tariff * export / 1000.0);

$$ LANGUAGE SQL STRICT;

It is a little hard to follow, and if the daily amount was negative, it decreases that value (instead of ignoring the discount), but it will do for now. I would prefer not to choose a retailer that hides a penalty behind a so-called discount anyway, so they would have to be significantly cheaper to attract my custom.

Using this function, we can now calculate the total costs for all retailers, and compare them:

SELECT retailer.name,
       '$ ' || lpad(
         TRUNC(SUM(cost(retailer, import::INTEGER, export::INTEGER)), 2)::TEXT,
         9
       ) AS cost
  FROM daily_summary, retailer
 GROUP BY retailer.name
 ORDER BY cost ASC;

There is a bunch of code in there to make the output prettier: some of that is lost by the HTML output I have used to generate this table. In an SQL view, those decimal points would all line up…

name	cost
AGL Essentials	$ 897.87
AGL Residential Essentials	$ 916.85
Powerband Home Flat + Solar FIT	$ 955.59
Lumo Basic	$ 965.56
Origin Energy	$ 971.44
SA Simply RAA VPP	$ 978.81
Power Direct Residential Saver	$ 979.84
AGL Solar Savers	$ 997.31
Click Banksia Solar	$ 1028.43
PowerShop Shopper Market	$ 1065.24
Kogan	$ 1075.74
Red Energy Living Energy Saver	$ 1092.32
Energy Australia Total Plan Home	$ 1100.55
Origin Energy Econnex	$ 1155.60
Alinta Energy No Fuss (Single Rate)	$ 1176.65
Diamond Energy Everyday Renewable Saver	$ 1302.24
Blue NRG DMO - General Usage	$ 1929.36

(17 rows)

Which brings me to my conclusion: my current retailer is the cheapest for my current generation and consumption patterns.

But we can go further. We can always go further.

We can look at different months, and see if it’s always the case that a given retailer is the best option. Perhaps it is worthwhile shopping around, and switching between retailers at different times of the year.

CREATE OR REPLACE VIEW monthly_cost_comparisons AS

SELECT retailer.name, 
       to_char(date, 'YYYY-MM') AS month,
       SUM(cost(retailer, import::INTEGER, export::INTEGER)) AS cost
  FROM daily_summary, retailer
  GROUP BY retailer.name, to_char(date, 'YYYY-MM');

That by itself is not that useful, but we can use the crosstab feature to view it in a more readable manner.

CREATE EXTENSION IF NOT EXISTS crosstab;

SELECT * 
  FROM crosstab(
    $$SELECT name, month, '$ ' || lpad(cost::INTEGER::TEXT, 4)
       FROM monthly_cost_comparisons ORDER BY name$$, 
    'SELECT DISTINCT month FROM monthly_summary ORDER BY month'
) AS (
    "retailer" TEXT, 
    "2019-03" TEXT,
    "2019-04" TEXT,
    "2019-05" TEXT,
    "2019-06" TEXT,
    "2019-07" TEXT,
    "2019-08" TEXT,
    "2019-09" TEXT,
    "2019-10" TEXT,
    "2019-11" TEXT,
    "2019-12" TEXT
);

Frustratingly, this function requires enumerating the columns.

retailer	2019-03	2019-04	2019-05	2019-06	2019-07	2019-08	2019-09	2019-10	2019-11	2019-12
AGL Essentials	$ 14	$ 41	$ 128	$ 200	$ 209	$ 223	$ 90	$ 1	$ 1	$ -10
AGL Residential Essentials	$ 19	$ 45	$ 127	$ 196	$ 204	$ 219	$ 93	$ 10	$ 9	$ -5
AGL Solar Savers	$ 15	$ 45	$ 143	$ 223	$ 233	$ 249	$ 100	$ 1	$ 0	$ -11
Alinta Energy No Fuss (Single Rate)	$ 46	$ 72	$ 147	$ 216	$ 225	$ 242	$ 122	$ 47	$ 46	$ 13
Blue NRG DMO - General Usage	$ 91	$ 125	$ 226	$ 329	$ 341	$ 371	$ 203	$ 103	$ 102	$ 38
Click Banksia Solar	$ 20	$ 50	$ 144	$ 222	$ 232	$ 248	$ 104	$ 8	$ 8	$ -7
Diamond Energy Everyday Renewable Saver	$ 47	$ 76	$ 166	$ 246	$ 256	$ 276	$ 135	$ 45	$ 45	$ 11
Energy Australia Total Plan Home	$ 28	$ 57	$ 149	$ 228	$ 237	$ 255	$ 112	$ 19	$ 19	$ -2
Kogan	$ 47	$ 69	$ 132	$ 191	$ 198	$ 213	$ 112	$ 49	$ 49	$ 16
Lumo Basic	$ 22	$ 49	$ 134	$ 203	$ 212	$ 227	$ 98	$ 12	$ 12	$ -4
Origin Energy Econnex	$ 46	$ 71	$ 143	$ 211	$ 219	$ 236	$ 120	$ 48	$ 47	$ 14
Origin Energy	$ 17	$ 45	$ 136	$ 213	$ 222	$ 238	$ 98	$ 5	$ 5	$ -8
Power Direct Residential Saver	$ 27	$ 52	$ 131	$ 198	$ 207	$ 222	$ 100	$ 21	$ 21	$ 1
PowerShop Shopper Market	$ 44	$ 67	$ 133	$ 193	$ 201	$ 215	$ 111	$ 44	$ 44	$ 13
Powerband Home Flat + Solar FIT	$ 31	$ 55	$ 126	$ 187	$ 195	$ 208	$ 98	$ 26	$ 26	$ 4
Red Energy Living Energy Saver	$ 36	$ 63	$ 142	$ 212	$ 221	$ 237	$ 112	$ 32	$ 32	$ 6
SA Simply RAA VPP	$ 23	$ 50	$ 135	$ 206	$ 215	$ 230	$ 99	$ 13	$ 13	$ -4

From this, I can see that nothing is really going to save me more than a few bucks a month: and this does not take into account the $200 “bonus” I get from AGL if I stick with them for a year (or something like that).

I’d also like to look at actual billing periods, because then I can see that everything matches up to what my actual bills are (and then I can add in the bonuses at the times they applied).

My billing periods went to monthly on a specific date, so from then on they are regular one-month periods, but prior to that they were a little all over the place:

CREATE OR REPLACE VIEW billing_periods AS 

SELECT 'AGL Essentials'::TEXT AS retailer,
       DATERANGE(start::DATE, (start + INTERVAL '1 month')::DATE) as period
  FROM generate_series('2019-06-24'::DATE, now()::DATE, INTERVAL '1 month') start
  
 UNION ALL
 
 VALUES ('AGL Essentials'::TEXT, '[2019-03-07,2019-06-01)'::DATERANGE),
        ('AGL Essentials', '[2019-06-01,2019-06-24)')
  
  ORDER BY period DESC;

We can model the retailer bonuses in a way that only applies them when they apply, and if we are calculating the costs for that retailer:

CREATE TABLE retailer_bonus (
  retailer TEXT, 
  billing_period DATERANGE, 
  bonus NUMERIC
);

INSERT INTO retailer_bonus(retailer, billing_period, bonus)
     VALUES ('AGL Essentials', '[2019-08-24,2019-09-24)', 25);

And now some views for calculating the actual costs for each day, and then aggregating within the billing periods (and a pretty version).

CREATE OR REPLACE VIEW actual_costs AS

SELECT retailer,
       billing_periods.period,
       daily_summary.date,
       cost(_ret, import::INTEGER, export::INTEGER),
       generation
  FROM billing_periods
 INNER JOIN retailer _ret ON (_ret.name = billing_periods.retailer)
 INNER JOIN daily_summary ON (daily_summary.date <@ billing_periods.period);
 
 
CREATE OR REPLACE VIEW billing_period_costs AS
 
SELECT retailer, 
       period, 
       SUM(cost) - COALESCE((SELECT SUM(bonus) 
                               FROM retailer_bonus 
                              WHERE billing_period = period
                                AND retailer_bonus.retailer = retailer.name), 0) AS cost, 
       SUM(generation) AS generation
  FROM actual_costs
 GROUP BY retailer, period
 ORDER BY period DESC;
 
 
CREATE OR REPLACE VIEW billing_period_costs_pretty AS 
SELECT retailer, 
       period,
       '$ ' || LPAD(TRUNC(cost, 2)::TEXT, 7, ' ') AS cost,
       LPAD(TRUNC(generation::NUMERIC / 1000, 1)::TEXT, 6, ' ') || ' kWh' AS generation
  FROM billing_period_costs
 ORDER BY period DESC;

And now, we can look at the actual bills, with generation for that period:

 SELECT * FROM billing_period_costs_pretty;

retailer	period	cost	generation
AGL Essentials	[2019-11-24,2019-12-24)	$ -9.50	595.4 kWh
AGL Essentials	[2019-10-24,2019-11-24)	$ -1.20	879.0 kWh
AGL Essentials	[2019-09-24,2019-10-24)	$ 12.80	757.3 kWh
AGL Essentials	[2019-08-24,2019-09-24)	$ 104.30	610.6 kWh
AGL Essentials	[2019-07-24,2019-08-24)	$ 224.27	374.2 kWh
AGL Essentials	[2019-06-24,2019-07-24)	$ 210.24	302.8 kWh
AGL Essentials	[2019-06-01,2019-06-24)	$ 148.31	244.8 kWh
AGL Essentials	[2019-03-07,2019-06-01)	$ 183.65	1495.9 kWh

Note that the first row is the current month: it’s only partway through, so I will end up with even more of a credit than that, with any luck.

These values are all within a dollar or two of the amounts I was actually billed, so that is nice.

But wait, there is more!

We can use this, with just a couple more functions, to make fairly good estimates of what our outputs and costs would be if we had more panels.

But that one will have to wait for another day.

asyncpg and upserting bulk data

2019-12-13T20:47:52+00:00

I’ve been doing some work on analysing the output of my PV system, and needed a mechanism for inserting large numbers of rows of data pulled from the Fronius API. Since I mostly work in python, I decided to use a python script to get the data from the inverter, but I needed a clean way to insert potentially lots of rows (every 5 minute period from every day: batched in 16 day blocks).

I’d normally just use psycopg2, but since I was using Python 3, I thought I’d try out asyncpg.

The API is quite nice:

import asyncio
import asyncpg

async def write():
    conn = await asyncpg.connect('postgres://:@:/database')
    await conn.execute('query', [params])
    await conn.close()

asyncio.get_event_loop().run_until_complete(write())

So, that part was fine. There are more hoops to jump through because async, but meh.

But I had an interesting case: I wanted to do a bulk insert, and I didn’t know how many records I was going to be inserting. It seemed like it was going to be a bunch of work to build up placeholder strings based on the number of rows, and the number of columns in each row.

But no, there is a function that works perfectly (and turned out to be even better, because it takes an argument as to the name of the table):

    conn = await asyncpg.connect(...)
    await conn.copy_records_to_table(table, records=values)
    await conn.close()

That was really easy. It’s a requirement that each row of values is in the same order as the table definition, but my tables in this case were simple.

But there’s one catch: this works great for inserting data, but what about when some of that data is already present. We want to insert some rows, but update others: upsert. Postgres supports this using the ON CONFLICT ... DO ... syntax. So, how can this work with the bulk insert (that uses the Postgres COPY command under the hood)?

    # Our table just contains two columns: timestamp and value.
    await conn.execute('''CREATE TEMPORARY TABLE _data(
        timestamp TIMESTAMP, value NUMERIC
    )''')
    await conn.copy_records_to_table('_data', records=values)
    await conn.execute('''
        INSERT INTO {table}(timestamp, value)
        SELECT * FROM _data
        ON CONFLICT (timestamp)
        DO UPDATE SET value=EXCLUDED.value
    '''.format(table=table))

Sweet.

We can use the string formatting tools here because the table name is controlled by us, so there is not an SQL injection vector to worry about. You should not do this if that value could be coming from a user though.

But there is one problem. We are updating rows, even if the value has not changed. If you don’t think too hard, that’s probably fine. But the way postgres works is that it rewrites every row that has “changed”, even if it hasn’t really changed. It marks the old row as superseded, and that space will not be reclaimed until the table is vacuumed.

Instead, we should be able to just ignore those rows that have the same value (which, in my case, should be all rows except the last one, as that is the only one that should have different data).

    await conn.execute('''
        INSERT INTO {table}(timestamp, value)
        SELECT * FROM _data
        ON CONFLICT (timestamp)
        DO UPDATE SET value=EXCLUDED.value
        WHERE {table}.value <> EXCLUDED.value
    '''.format(table=table))

Oh nice: Postgres also supports a WHERE clause on the DO UPDATE clause.

Fronius Dashboard

2019-12-09T21:06:36+00:00

Last January, I had some solar panels installed. The inverter I chose was a Fronius Primo, which is a relatively well thought of piece of equipment.

It comes with access to Solar.Web, which is Fronius’ online portal. Using this, you can get realtime access to your power generation, and hourly updated energy generation for the current day. They also have an Apple Watch app (and iPhone, naturally).

Additionally, I configured PVOutput, which gives me generation stats with a five minute resolution.

Shortly after, I decided to have a Fronius Smart Meter installed, which gives me realtime access to feed-in and feed-out data. This matches up fairly well with the SA Power Networks smart meter data: which is only available a day or so after the fact. I wish I’d had this installed from day one.

One of the key things I wanted to be able to do was have an easily readable display that shows how much power we are generating (and how much energy we have generated today), but also the current consumption, and the feed in or out right now.

After a bit of work, I came up with a (what I think is) really nice UI. I wrote it using Elixir: mainly because I wanted to write something using Phoenix LiveView. It’s actually a bit too much like Rails for me to want to use it frequently, but I do wish there was something a bit like it for Django.

Anyway, enough of the technology choices, let’s look at the dashboard:

The data is updated every second; and at night time, it goes into a dark mode:

Like most of my side projects, this is freely available. You can install this now on any device that runs Docker, as long as it’s amd64 or arm. I’ve built it on a Raspberry Pi 3B+, but I think it should also work on a Raspberry Pi Zero.

$ docker run -it -p 4000:4000 \
    --restart unless-stopped \
    --name fronius-dashboard \
    --env INVERTER=<ip-address-or-hostname> \
    schinckel/fronius-dashboard:latest

This should download the latest version, and start it up. You will need to supply your own inverter’s IP address or hostname: but if you have fronius.lan as the hostname, then you can omit that line.

To make it available outside of your network, you’ll need to forward something to port 4000 of the device that is running this service. I have the aforementioned rPi3B+ running this, an MQTT broker and Node-RED, and my MQTT2HomeKit bridge, and expose only the Fronius Dashboard service to the internet.

It’s also designed to work well as a home screen app on iOS, and possibly Android devices.

There is no warranty, implied or otherwise as to the fitness of this to your purposes, but if you are nice, I’ll try to fix any bugs. You can view the source code at Source Hut.

I’m not sure how it will handle not having a Smart Meter installed.

I’ll also be more than happy to try to make this work with other inverters.

I’ve also been playing around with using nerves-project to build a standalone image that can be burned to a RaspberryPi Zero, but I’m still working through some kinks related to being able to configure that. I’m hoping to have that as a turn-key solution that I can install into some friends who have/are purchasing Fronius inverters.

Basically, I need a mechanism for storing the WiFi credentials on the nerves board, and allow configuring the IP address/hostname of the inverter. I got partway through an admin UI, but it’s not really complete yet.

NodeRED Coffee Grinder Redux

2019-12-02T22:12:28+00:00

I built a Node-RED flow for my Coffee Grinder. There was a very subtle bug.

There are four ways that you can start the grinder:

Trigger an HTTP Request for a Single
Trigger an HTTP Request for a Double
Start the grinder manually using the button on the socket
Start the grinder using HomeKit or MQTT

This is all fine, and 18 seconds after a manual trigger, the grinder will turn off.

Except, if you manually stop the grinder, and then restart it before the initial 18 seconds has elapsed, it will still turn of at that 18 second point. Which, whilst probably okay, is still frustrating.

What I wanted to do instead is “cancel” a flow, if the switch is turned off (in any way).

It turns out there is a way to use the “Trigger” node to get this behaviour:

This allows me to simplify the flow, because the values I need to turn things on and off (1 and 0) are the same ones we want to use to trigger and reset.

I found I needed to respond to the HTTP request from Siri Shortcuts immediately, because the timeout appears to be shorter than my grind time. It would have been nice to be able to respond when the grind had finished: perhaps I could do something where it returns bytes once per second until it’s done, but for now, this works great.

It actually got me thinking: I wonder how maintainable a really complex project would be in Node-RED.

Cascading Selects: Leaf Node Only

2019-12-02T21:56:37+00:00

I’ve written quite a lot of stuff about trees in Postgres, but not quite so much about how they might be used. One situation where they could be used is in categories: where each category may have sub-categories, and those may or may not have further sub-categories. When editing an object that has a single category, it would be useful to only show the values at each level, and then making a selection results in the display of another select element, that only has those values that apply at that level, and with that chain of parents selected.

Changing a selection should remove all subsequent items, and redisplay the “next” select with the new available options.

As you select different options from the “Category” select, you will see that only the leaf node is shown in the box on the right: but also (to show that it’s not just rendering the last value), we have a second “category” key/value pair from the text input.

This is a bit of a proof of concept: it doesn’t actually do an AJAX request: although there is a commented out fetch() request that shows where this would occur, but it doesn’t include the current selection.

The responses from the server would need to look something like:

[
  {"value": 8, "label": "Eight"},
  {"value": 9, "label": "Nine"},
  {"value": 10, "label": "Ten"},
  {"value": 11, "label": "Eleven"}
]