MQTT HomeKit Bridge

Writing HomeKit devices is possible (and even simple) using tools like HAP-python. However, devices like the esp8622 are slow to do the handshake stuff, and having to keep them awake to read temperature or other data on demand means you can’t use the deep sleep features.

These IoT devices can, however, quite easily handle publishing to an MQTT topic.

I’ve read most of the HomeKit Accessory Protocol spec (at least, the non-commercial one, but you’ll still need credentials to view that link), and I think I have a pretty good handle on it. And it occurred to me that it should be possible to bridge, in both directions, an MQTT broker and HomeKit.

Basically, you can then have a single bridge device (that you only need to register in HomeKit once), and have this connect to your MQTT broker. It can then perform two actions:

  • Listen for MQTT messages that meet certain criteria, and pass these through to HomeKit
  • Listen for HomeKit messages, and convert these into MQTT messages.

There’s a bit more to it than that: it keeps track of what devices are known, and will automatically add new devices when it detects one (via a matching MQTT topic). It could also remove devices that have not been seen for some time (or when a specific message indicates that device is no longer available).

I’ve chosen to make this as simple as possible - at this stage of my prototype there is no authentication in the MQTT broker, but that will have to change before I hook up anything other than temperature sensors. My Garage Door opener is still a standalone HomeKit device!

So, down to the nuts and bolts.

A message that matches the following pattern will be processed:

HomeKit/<device_id>/<service_type>/<characteristic_name>

For instance, I can currently see some messages that look like:

HomeKit/esp8266_12345678/TemperatureSensor/CurrentTemperature 20
HomeKit/esp8266_12345678/HumiditySensor/CurrentRelativeHumidity 58
HomeKit/123456789ABCDEF/TemperatureSensor/CurrentTemperature 20.125
HomeKit/TEST/Switch/On 1

The thing you might notice is that two of those messages have the same device id - the bridge knows this, and will add a second service to the accessory.

To be honest, this solution seems too simple, but it has been working really well for me for some time now. I have configured the sensors to send retain (persistent) messages, but I think I’m going to turn that off, except in the case of things like the switch device.

The other thing I haven’t totally nutted out yet is the authentication/authorisation stuff for MQTT. I have had some thoughts at this point though:

  • A device will generate a password when it first boots (and stores this).
  • This password will be used with the device id to authenticate with the broker.
  • When the client attempts to connect, a check will be made to see if the user exists - if so, the password must match. If not, the user will be created.
  • Any user created in this manner will be able to read and write topics that match HomeKit/<user_id>/#
  • A special user (the HomeKit bridge user) must be able to read and write all HomeKit/# topics.

Now that I’ve gotten OTA working with these devices, I need some mechanism for triggering this via MQTT.

OTA firmware updates with MicroPython/ESP8266

It’s been a hard couple of days, but I’ve finally managed to get OTA (over-the-air) firmware updates working on an ESP8266 running MicroPython.

Building and flashing OTA firmware

This assumes you have a working ESP toolchain (maybe see this comment if you don’t yet), and have been able to build and flash custom micropython firmware.

The bootloader/OTA software is available at yaota, but the instructions for how to actually build and flash the initial OTA server and regular app are not complete. Here’s what I ended up having to do to get it to work.

  • Before you build, you need to generate keys:
      $ cd ota-client
      $ . gen_keys.sh
      $ python -c "import rsa_sign; rsa_sign.dump_c(rsa_sign.load_key())"
  • The last step outputs some data from the key - specifically you need the mod value, and this needs to be used in a config.h file at the root of the project. Duplicate the config.h.example file, and replace the MODULUS value with your public key’s modulus.

  • I needed to make some other changes too: until they are merged, you can see them here. They are seperate pull requests, because whilst they all need to be applied to get a build to complete, they are all independent.

  • Once these patches have been applied, run make at the root of the project.

  • Flash the generated yaota8266.bin file to location 0x0 of your device.

  • Flash the micropython OTA build to location 0x3c000 of your device.

You should now be able to apply OTA updates. I also included in my ota-server the patch from ulno that allows for triggering OTA from within micropython code. I added into my micropython firmware a file ota.py, that contains:

import machine

def start():
    machine.RTC().memory('yaotaota')
    machine.reset()

This means I can trigger an OTA state by doing:

import ota
ota.start()

Performing an OTA update.

In order to do this, you must have built and flashed your own yaota8266, with your own keys. Otherwise you won’t be able to sign the updates, and the ota-server binary will reject them.

Trigger an OTA state (either using the process above, or by hitting a button in the 3 seconds after the bootloader is complete).

Then, in the ota-client directory, execute the following:

$ ota_client.py live-ota <../../micropython/ports/esp8266/build/firmware-ota.bin> -a <10.9.8.160>

This will send off the 2274-ish packets to the device, and after this you should have your new firmware installed. Note you’ll need to supply your firmware file and IP address.


Notes that I came across during this process.

  • If you are able to flash both images correctly, but get errors from the micropython app, check that you aren’t flashing a non-ota build. In my case, the boot sequence was performing as expected, but when no OTA state was detected, and the micropython app was due to start, it was crashing with an error:
     Fatal exception (0):
     epc1=0x4020a8ac, epc2=0x00000000, e...
  • I was unable to get an OTA update to work on my ESP8266-01 device. This could be because it doesn’t have enough flash, or some other reason.

  • The sonoff device(s) I have require flashing using -fm dout mode: with other modes a write_flash will appear to succeed, but the binary will not be viable, and only garbage will be seen on the screen.

  • Some USB-serial converters do not provide enough current to power an ESP8266 running in anything other than flash-from-UART mode. That is, you will be able to flash them, but not boot into regular apps.

  • Having a USB-serial converter that has RTS/DTR means you can have esptool.py automatically trigger boot mode. This is nice. It’s a shame that the sonoff devices don’t expose enough pins to hook this up though.


As an aside, my micropython builds contain the mqtt client library, and automatically set up the WiFi credentials for the network they are installed in. ESP8266 devices retain their WiFi credentials and attempt to restore their connection state after restart, so that can simplify code somewhat, and you can just edit the configuration in the modules/inisetup.py module.

Now, I just need to come up with some mechanism for triggering the OTA state in a device using MQTT, and for devices that are not continually awake (I have several sensors that use deep-sleep for some time, and then wake up, publish an MQTT message and then go back to sleep).

Arlec Powerboard esphome

I bought one of the Grid Connect powerboards from Bunnings last week, and flashed it with a custom firmware.

The model I bought was the PB89HA, which is the one with 5 sockets (one of which is not switchable).

The button is on GPIO3, and the LED is an inverted GPIO1.

The four relays are on GPIO5, GPIO4, GPIO13 and GPIO12, starting with the one closest to the button.

Since there is only one button, and four switchable outlets, I had to come up with a mechanism for controlling all of them. I ended up (after playing around with single-through-quadruple click) with a single click for toggling the first relay (nearest the button). Then, a medium (between 1 and 2 second) press turns all relays off, and a long (greater than 2 second) press turns them all on.

This is not really ideal, as there is no way to toggle just relay 2-4 without using some sort of external control - which goes against my ethos with respect to smart home.

Having said that, I’m not actually sure how I’m going to use this board…I don’t really have a bunch of things that are potentially close together that need individual control. I guess I could have it turn off everything in the entertainment unit except the PVR - that might be a way to save power overnight. I’d want the button more accessible than the powerboard that currently controls them though.

Anyway, the base YAML file follows - be aware that this does not include wifi, and would need to be included in an actual config file (with a device name defined).

esphome:
  name: $device_name
  platform: ESP8266
  board: esp01_1m
  on_boot:
    - light.turn_on:
        id: led
        brightness: 20%

binary_sensor:
  - platform: status
    name: "Status"

  - platform: gpio
    pin:
      number: GPIO3
      inverted: true
      mode: INPUT_PULLUP
    name: button
    on_multi_click:
      - timing:
        - ON for at least 1s
        - OFF for at least 0.2s
        then:
          - switch.turn_off: relay_0
          - switch.turn_off: relay_1
          - switch.turn_off: relay_2
          - switch.turn_off: relay_3

      - timing:
        - ON for at least 2s
        - OFF for at least 0.2s
        then:
          - switch.turn_on: relay_0
          - switch.turn_on: relay_1
          - switch.turn_on: relay_2
          - switch.turn_on: relay_3

      - timing:
        - ON for at most 0.5s
        - OFF for at least 0.2s
        then:
          - switch.toggle: relay_0

switch:
  - platform: gpio
    id: relay_0
    pin: GPIO5
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/0/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/0/On
         retain: ON
         payload: 0

  - platform: gpio
    id: relay_1
    pin: GPIO4
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/1/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/1/On
         retain: ON
         payload: 0

  - platform: gpio
    id: relay_2
    pin: GPIO13
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/2/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/2/On
         retain: ON
         payload: 0

  - platform: gpio
    id: relay_3
    pin: GPIO12
    on_turn_on:
      - mqtt.publish:
          topic: HomeKit/${device_name}/Outlet/3/On
          retain: ON
          payload: 1
    on_turn_off:
      - mqtt.publish:
         topic: HomeKit/${device_name}/Outlet/3/On
         retain: ON
         payload: 0

light:
  - platform: monochromatic
    output: led1
    id: led
    restore_mode: ALWAYS_ON

output:
  - platform: esp8266_pwm
    pin:
      number: GPIO1
    id: led1
    inverted: True

sensor:
  - platform: wifi_signal
    name: "WiFi signal sensor"
    update_interval: 5min

ota:

logger:

mqtt:
  broker: "mqtt.lan"
  discovery: false
  topic_prefix: esphome/${device_name}
  on_message:
    - topic: HomeKit/${device_name}/Outlet/0/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_0
    - topic: HomeKit/${device_name}/Outlet/0/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_0

    - topic: HomeKit/${device_name}/Outlet/1/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_1
    - topic: HomeKit/${device_name}/Outlet/1/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_1

    - topic: HomeKit/${device_name}/Outlet/2/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_2
    - topic: HomeKit/${device_name}/Outlet/2/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_2

    - topic: HomeKit/${device_name}/Outlet/3/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay_3
    - topic: HomeKit/${device_name}/Outlet/3/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay_3

Hacking Arlec's 'Smart' sensor light

Quite some time ago, I purchased one of the Arlec Smart Security Lights from Bunnings. The big draw for me was that these devices have an ESP8266, and run a Tuya firmware, which can be trivially flashed without opening up the unit.

In this case, though, the device was not really that capable. Whilst there is a PIR sensor (and an LDR for tuning if the light should turn on yet or not), the status of the PIR is not exposed at all. Instead, the firmware allows toggling between three states: “ON”, “OFF”, and “SENSOR”.

That’s not actually that useful. For one, it makes using it in conjunction with a physical switch really inconvenient. The behaviour I would prefer is:

  • Light ON/OFF state can be toggled by network request.
  • Light ON/OFF state can be toggled by physical switch. It must not matter which state the switch is in, toggling it must toggle the light.
  • PIR ON turns ON the light.
  • PIR OFF turns OFF the light, but only if it was turned ON by PIR.

As the last point indicates, the only time the PIR turning off should turn the light off is if the light was turned on by the PIR. That is, either physical switch actuation or a network request should turn the light into manual override ON.

There is no manual override OFF.

Most of this was already present in a firmware I wrote for adding a PIR to a light strip. However, the ability to also toggle the state using a physical switch is important to me: not in the least because there is a switch inside the door where I want to mount this, and I’m very likely to accidentally turn it on when I go outside: it’s also a much better solution to a manual override than just having to use HomeKit. I’ll possibly add that feature back into the aforementioned project.

Like all of the other Grid Connect hardware I’ve used so far, it was easy to flash using tuya convert. But I did run into a bunch of problems from that point onwards. The base contains the ESP8266, one of the TYWE2S units, and it’s possible to see without opening up the sensor unit that this has three GPIO pins connected: GPIO4, GPIO5 and GPIO12.

With a custom firmware, it was possible to see that GPIO5 is connected to the green LED near the PIR sensor, but the other two appear to be connected to another IC on the PCB. I thought perhaps this could be accessed using the TuyaMCU protocol, but had no luck.

As it turns out, I’d prefer not to have to use that. There are two more wires, it would be great if I could connect one of them to the relay, and the other to the PIR.

Indeed, with limited rewiring (I did have to cut some tracks and run wires elsewhere), I was able to connect GPIO12 to the point on the PCB that the output from the other IC that triggered the relay, and GPIO4 to the input of the IC that was sensing the PIR output.

I also ran an extra pair of wires from GPIO14 and GND, to use to connect to the physical switch. These will only transmit low voltage.

Unfortunately, I forgot to take photos before putting it all back together and having it mounted on the wall.

Then we just need the firmware:

esphome:
  name: $device_name
  platform: ESP8266
  board: esp01_1m

globals:
  - id: manual_override
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: mqtt_triggered
    type: bool
    restore_value: no
    initial_value: 'false'

sensor:
  - platform: wifi_signal
    name: "WiFi signal sensor"
    update_interval: 60s

binary_sensor:
  - platform: gpio
    pin: GPIO4
    id: pir
    device_class: motion
    filters:
      - delayed_off: 15s
    on_press:
      - light.turn_on: green_led
      - mqtt.publish:
          topic: HomeKit/${device_name}/MotionSensor/MotionDetected
          payload: "1"
      - switch.turn_on: relay
    on_release:
      - light.turn_off: green_led
      - mqtt.publish:
          topic: HomeKit/${device_name}/MotionSensor/MotionDetected
          payload: "0"
      - if:
          condition:
            lambda: 'return id(manual_override);'
          then:
            logger.log: "Manual override prevents auto off."
          else:
            switch.turn_off: relay

  - platform: gpio
    pin:
      number: GPIO14
      mode: INPUT_PULLUP
    name: "Toggle switch"
    filters:
      - delayed_on_off: 100ms
    on_state:
      - switch.toggle: relay
      - globals.set:
          id: manual_override
          value: !lambda "return id(relay).state;"

ota:

logger:

output:
  - platform: esp8266_pwm
    id: gpio5
    pin:
      number: GPIO5
    inverted: False

switch:
  - platform: gpio
    id: relay
    pin:
      number: GPIO12
      # inverted: True
      # mode: INPUT_PULLDOWN_16
    on_turn_on:
      - if:
          condition:
            lambda: 'return id(mqtt_triggered);'
          then:
            logger.log: "No MQTT message sent"
          else:
            mqtt.publish:
              topic: HomeKit/${device_name}/Lightbulb/On
              retain: ON
              payload: "1"
    on_turn_off:
      - if:
          condition:
            lambda: 'return id(mqtt_triggered);'
          then:
            logger.log: "No MQTT message sent"
          else:
            mqtt.publish:
              topic: HomeKit/${device_name}/Lightbulb/On
              retain: ON
              payload: "0"


light:
  - platform: monochromatic
    id: green_led
    output: gpio5
    restore_mode: ALWAYS_OFF
    default_transition_length: 100ms

mqtt:
  broker: "mqtt.lan"
  discovery: false
  topic_prefix: esphome/${device_name}
  on_message:
    - topic: HomeKit/${device_name}/Lightbulb/On
      payload: "1"
      then:
        - globals.set:
            id: mqtt_triggered
            value: 'true'
        - switch.turn_on: relay
        - globals.set:
            id: mqtt_triggered
            value: 'false'
        - globals.set:
            id: manual_override
            value: !lambda "return !id(pir).state;"
    - topic:  HomeKit/${device_name}/Lightbulb/On
      payload: "0"
      then:
        - globals.set:
            id: mqtt_triggered
            value: 'true'
        - switch.turn_off: relay
        - globals.set:
            id: manual_override
            value: 'false'
        - globals.set:
            id: mqtt_triggered
            value: 'false'

I’ve also implemented a filter on sending the state to MQTT: basically, we don’t want to send a message to MQTT if we received the same message. There is a race condition that can occur where this results in fast toggling of the relay as each toggle sends a message, but then receives a message with the alternate state. I’ve had this on my Sonoff Mini firmware too.

Smart Devices Aren't (or why connected devices suck)

I love tinkering with gadgets. I’ve put a bunch of sensors around my house, so I can see the temperature in various places, and have a couple of smart light and power devices too. At the moment, they are limited to my laundry (where the hard-wired switch is in the wrong place, due to the moving of a door), my workbench (because the overhead lights there run from a power point, so it was trivial to put in a smart switch), and the lounge room (where I had room in the light fitting to put a Sonoff Mini).

In all of these cases, with the exception of the Laundry, since the switch is not really accessible, I have taken great care to ensure that the physical switches still toggle the light. In that case I have an Ikea bulb connected to an Ikea dimmer.

In my study, I have a desk lamp that has a smart (dimmable) bulb in it, and it irks me no end that I have to use a smart device or computer to turn it on or off. I will be getting some more of the Ikea dimmers to alleviate this, but in the mean time, it’s a pain to turn on or off.

Having said that, I love the option of being able to automate power and lighting, or turn things off from a distance. I just don’t like that being the only way.

I installed Home Assistant on the weekend. But, in order to fit that onto my Raspberry Pi, I needed to use a bigger Micro SD card.

Which meant I needed to clone the old one.

Which took several hours.

I’d already installed Home Assistant before running out of space, and had converted a couple of my esphome devices to use the API instead of just MQTT for connection, including the lounge room light.

Now, it turns out, by default there is an “auto reboot if API not found in 15 minutes” setting, which meant that during the four or five hours it took to create an image of the Micro SD, verify this, copy to a new SD card, and then verify that, my lights (and a powerboard in my office) would flick off every 15 minutes. Likewise, if they cannot connect to a WiFi access point they will power cycle. I believe this second one can be resolved using a Captive AP setting that will mean if they can’t connect to a network, they will create their own.

Which really got me thinking. Smart devices should continue to work in every way possible when they don’t have access to the local network, or the internet. In my case, my smart devices do not have access to the internet anyway, because they don’t need to. However, the point is the same.

In situations where a network connection, or even worse, a working connection to a server that you don’t control, is no longer available, you dont’ want your lights or god forbid, your coffee machine to not be able to perform their simple task.

This worries me somewhat about the current trends in smart homes. At some point, companies will stop supporting their devices (this has already happened), and they will become less useful than their dumb counterparts. And add further to our global waste problems.

But having a significant system outage (even an intentional one, like in my case), made me think about other aspects of my home automation as well.

I’ve been using NodeRED for a couple of automation tasks. One of them was to have different grind lengths for my coffee grinder, and making this available to Siri.

However, with the device running NodeRED not operating, I was no longer able to rely on this.

I was heading this way philosophically before, but (OMG NO COFFEE) this just cemented something else in my mind. Automations, where they don’t rely on interaction between multiple devices, should live on the local device where possible. Further to this, where the interaction between devices is required for the automation (like the PIR sensor in the laundry I have that turns on the Ikea lightbulb), the devices should connect directly to one another, without requiring some other mechanism to trigger the automation.

In my case, I have a physical button that I press to trigger a long grind. But, the grind only stops if the NodeRED server tells it to. And, I had no way to (when NodeRED was not running), trigger a short grind.

I was able to fix this: I now have a short press triggering a long grind, and a long press triggering a short grind. That seems backwards, but since I mostly do a long grind in the morning before I’ve had time to properly wake up, I want that the easiest one to trigger…


Having to program this in my esphome firmware instead of NodeRED made for an interesting exercise. Because we need to turn off the device after a period of time, but need to be aware of other events that have happened in the meantime, we need to use scripts.

script:
  - id: short_grind
    then:
      - switch.turn_on: relay
      - delay: 13s
      - switch.turn_off: relay
  - id: long_grind
    then:
      - switch.turn_on: relay
      - delay: 17s
      - switch.turn_off: relay

Whenever our relay turns on, we want to start our long grind script, so that even if the relay was triggered some other way than through the script, it will turn off after 17s if not before. Whenever it turns off, we want to stop any instances of our scripts running. We can also use Template Switches to have logical devices we can use to trigger the different scripts, either from Home Assistant, or from button presses:

switch:
  - platform: gpio
    id: relay
    pin: GPIO2
    restore_mode: ALWAYS_OFF
    on_turn_on:
      - script.execute: long_grind
    on_turn_off:
      - script.stop: short_grind
      - script.stop: long_grind
  - platform: template
    name: "Grind a Single"
    optimistic: true
    id: grind_a_single
    icon: mdi:coffee-outline
    turn_on_action:
      - script.execute: short_grind
      - script.wait: short_grind
      - switch.template.publish:
          id: grind_a_single
          state: OFF
    turn_off_action:
      - switch.turn_off: relay
  - platform: template
    name: "Grind a Double"
    optimistic: true
    id: grind_a_double
    icon: mdi:coffee
    turn_on_action:
      - script.execute: long_grind
      - script.wait: long_grind
      - switch.template.publish:
          id: grind_a_double
          state: OFF
    turn_off_action:
      - switch.turn_off: relay

Both of these template switches will also turn off the grinder when toggled off if they are currently on.

There’s only one more bit of logic that’s required, and that’s the handling of the physical button. I wanted this to trigger either length based on the amount of time that the button is held down for, but I also want a UX affordance of knowing when you have held it down long enough to trigger the alternate action. Finally, if it’s on, any type of press should turn it off, and not trigger a new grind.

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO14
      inverted: true
      mode: INPUT_PULLUP
    on_press:
      - light.turn_on:
          id: led
          transition_length: 0s
      - delay: 500ms
      - light.turn_off:
          id: led
          transition_length: 0s
    on_click:
      - max_length: 350ms
        then:
          - if:
              condition:
                - switch.is_on: relay
              then:
                - switch.is_off: relay
              else:
                - script.execute: long_grind
      - min_length: 500ms
        max_length: 2s
        then:
          - if:
              condition:
                - switch.is_on: relay
              then:
                - switch.is_off: relay
              else:
                - script.execute: short_grind

Remember that the turning off of the relay will stop any running scripts.

So, now, when you hold down the button, when the light turns off, you can release it and it will trigger a short grind. If you just tap the switch and release it immediately, it will trigger a long grind. Any button press when the grinder is already running will turn it off.

ESPHome variables from Home Assistant

One of my key tenets of Home Automation is that as much stuff should be done locally as possible. Whilst with HomeAssistant, that means “in the local network”, I feel that “in the device” is even better. However, sometimes values should still be configurable (from within Home Assistant), but the device should still work even if there is not (and has never been) a connection to Home Assistant since boot.

For instance, I have a sensor light. The device contains a PIR sensor, with configurable delay, sensitivity and lux settings. This in turn drives a relay connected to a pair of PAR bulbs, and additionally has a green LED. This device usually only has the ESP8266 controlling if the thing should ever turn on at all, but I hacked it so that it exposes the PIR sensor, and allows triggering the relay.

But the delay between the motion no longer being detected and the light turning off should be configurable. I don’t want to have to re-flash the device if I decide to change this value, so I wanted to make it that the value will be fetched from Home Assistant (if it is set).

This turned out to work really well. There are a few parts that need to be set up in the YAML (this is not the complete file):

esphome:
  name: ${device_name}
  platform: ESP8266
  board: esp01_1m
  on_boot:
    - logger.log:
        level: DEBUG
        format: 'Light ON time is set to %d'
        args: ['id(light_on_time)']

globals:
  - id: light_on_time
    type: int
    restore_value: true
    initial_value: '30'

sensor:
  - platform: homeassistant
    id: on_time
    entity_id: input_number.${device_name}
    on_value:
      then:
        - globals.set:
            id: light_on_time
            value: !lambda 'return int(x);'

It really is that simple. The first boot will set the light_on_time variable to 30. Then, when it connects to Home Assistant, it will look for an input_number.<device_name> (which matches the device name). If it finds one (or is ever told about this value changing), then it will commit that new value to the flash, and this will be be restored after a reboot.

There is one other thing we could do here, to make it so that we don’t write the value to the flash if it has not changed (and prevent wear to that, since it is limited to a number of writes):

sensor:
  - platform: homeassistant
    id: on_time
    entity_id: input_number.${device_name}
    on_value:
      then:
        if:
          condition:
            lambda: 'return id(light_on_time) != int(x);'
          then:
            - logger.log:
                level: DEBUG
                format: 'Light ON time changed from %d to %d seconds'
                args: ['id(light_on_time)', 'int(x)']
            - globals.set:
                id: light_on_time
                value: !lambda 'return int(x);'

With regards to a similar problem, detecting if it is dark enough to turn the light on should be something like “a bit before sunset to a bit after sunrise”. I could set the lux threshold on the device, but it would be nice to have motion detection work during the day too.

Here, we can have a global variable sun_is_down that defaults to being true, and is only set to false when the sun comes up. However, we would also want this to trigger when first connecting to Home Assistant and the sun is already up.

We can use a wait_until in our on_boot handler to trigger this:

globals:
  - id: sun_is_down
    type: bool
    restore_value: false
    initial_value: 'true'

esphome:
  ...
  on_boot:
    - wait_until:
        api.connected:
    - if:
        condition:
          sun.is_above_horizon:
        then:
          - logger.log:
              level: DEBUG
              format: 'Sun is up, light will not turn on.'
          - globals.set:
              id: sun_is_down
              value: 'false'
        else:
          - logger.log:
              level: DEBUG
              format: 'Sun is down, light will turn on.'

sun:
  on_sunrise:
    - elevation: 10°
      then:
        - logger.log:
            format: 'The sun is now down, light will turn on.'
        - globals.set:
            id: sun_is_down
            value: 'false'
  on_sunset:
    - elevation: 10°
      then:
        - logger.log:
            format: 'The sun is now up, light will not turn on.'
        - globals.set:
            id: sun_is_down
            value: 'true'

I’ve used a 10° elevation to trigger the light to turn on a bit before sunset and still turn on a bit after sunrise when motion is detected. I haven’t figured out a way to get the same check to apply in the sun.is_above_horizon: check.

Hacking the DETA rewireable plug

There are a bunch of inexpensive IoT devices coming onto the market. One of these ranges are the GridConnect devices available from Bunnings. I’ve been eyeing these off, because they include some fairly reasonable looking switches and plugs, and, most importantly, are legal to have installed in your home in Australia.

I’m not okay with sticking a bunch of devices into my home network that communicate with the outside world. I like to use Apple’s HomeKit, which means the communication is funnelled through their systems, and everything I have configured as a HomeKit accessory either has no cloud component, and is where possible isolated onto a seperate wireless network.

In fact, most of my devices are home built, and have limited accesibility: they are permitted to connect to an MQTT server and nothing else.

Most of the commercially available items in Australia fail this test: they all have their own little cloud service. I dislike this for a couple of reasons: the first is security and privacy. I don’t trust that these providers will protect my data.

The second is a little more subtle: if an IoT provider goes out of business (or decides to end-of-life some products), then you are no longer able to access them in the way you might have liked.

So, I’ve been creating my own devices, or re-purposing commercial devices where possible. I grabbed a couple of the Mirabella globes, and was able to flash them with an ESPHome firmware, that I believe hardens the device somewhat. Having controllable light globes is neat, but realistically, they are of limited use. Having to use an App to turn on and off your lights is unacceptable, but also so is having an always listening microphone. And, again, what happens when you want to turn your lights on and your internet is down?

So, I’m more interested in smart switches.

I was not able to find much information about the Deta line of products that support GridConnect, so I went and bought the cheapest one, that I thought I was going to be realistically able to reverse engineer.

Opening up the “user accessible” region, we see the three screws for attaching the cable, and some lovely 2.0mm triangular screws.

IMG 2119

After spending more money on a set of screwdriver bits that had this size and shape, we are able to see the top of the PCB.

IMG 2120

Note that there are two relays: this is safer than things like the Sonoff Basic, that only switch the active wire, rather than both.

The underneath shows clearly the separation between the different AC lines.

IMG 2121

Finally, we can see the brains behind this switch:

IMG 2122

Note the code on the microcontroller: TYWE2S. Plugging that into a search engine yielded some interesting results.

It turns out that not only are these using an ESP8285 (which is just an ESP8266 with onboard flash, and in a smaller package), but they are running the Tuya firmware. So, it wasn’t even necessary to have opened up the casing.

Since I had a Raspberry Pi Zero W configured to run tuya-convert, I built up a simple firmware that would enable me to run OTA updates after flashing, and settled down to working out which GPIO pins are attached to which parts: the button, LED and relay.

(Oh, and this was interesting: both the active and neutral lines are toggled by relays, as opposed to something like the Sonoff, that only toggles the active line).

At some point, possibly because I was tired, I flashed a firmware that accessed GPIO pins 6 and 7. This locks the device up, and prevents booting. So, I then soldered some wires onto the device (after opening it back up again: so it turned out I did need that fancy screwdriver bit) to get access to the UART.

Interestingly, the ESPHome firmware detects that it’s been unable to boot, and boots into a safe mode. So, again, in hindsight, I still didn’t need to open it up.

Through trial and error, I was able to determine which GPIO was the button. That’s the easiest (and was helpful for me to be able to determine the LED and relay).

Surprisingly, it was GPIO1. Those familiar with the ESP8266 may know this is normally the TX pin for the UART. Which explains why, when I had the serial port connected, I was seeing a bunch of weird things when I pushed the button.

Once I had that, it was a matter of trying each GPIO in turn to see which was the relay and which was the LED. I did them in pairs, and happened to choose GPIO13 and GPIO14 in the same test, so for a while I thought maybe the relay and the LED were hardwired together.

Anyway, I now have a working firmware for this device:

substitutions:
  device_name: deta_plug

wifi:
  # Hah!

esphome:
  name: $device_name
  platform: ESP8266
  board: esp01_1m

binary_sensor:
  - platform: status
    name: "Status"

  - platform: gpio
    pin:
      number: GPIO1
      inverted: true
      mode: INPUT_PULLUP
    name: GPIO1
    on_press:
      - switch.toggle: relay

switch:
  - platform: gpio
    id: led
    pin:
      number: GPIO13
      inverted: true
  - platform: gpio
    id: relay
    pin: GPIO14
    on_turn_on:
      - switch.turn_on: led
      - mqtt.publish:
          topic: HomeKit/${device_name}/Switch/On
          retain: ON
          payload: 1
    on_turn_off:
      - switch.turn_off: led
      - mqtt.publish:
         topic: HomeKit/${device_name}/Switch/On
         retain: ON
         payload: 0


sensor:
  - platform: wifi_signal
    name: "WiFi signal sensor"
    update_interval: 60s

ota:

logger:

mqtt:
  broker: "mqtt.lan"
  discovery: false
  topic_prefix: esphome/${device_name}
  on_message:
    - topic: HomeKit/${device_name}/Switch/On
      payload: "1"
      then:
        - switch.turn_on:
            id: relay
    - topic: HomeKit/${device_name}/Switch/On
      payload: "0"
      then:
        - switch.turn_off:
            id: relay

I’m using my MQTT ⟺ HomeKit bridge, since that works great, but you could easily change the MQTT topics, or do whatever else you want to do.

Update: turns out, I just needed to search for the code: this page has everything I would have needed.

EspHome and HomeKit

HomeKit is Apple’s home automation software, and it’s pretty nice. It is local by default, although it can be configured to allow access from outside your home network, it does so in a way that does not store all of your data on an external server. Instead, the hub (an AppleTV, iPad or HomePod that lives in your house) can be contacted by your iOS or macOS device, and interact that way.

I’ve been playing around with the ESP8266 microcontroller, and whilst you could put a complete HomeKit Automation Protocol client on there, it’s probably going to run too slow to be useful. Instead, I’ve written an MQTT to HomeKit bridge, and just have simple devices that send data to the local MQTT broker.

I’ve even set up a different wireless network, just for those devices. They cannot connect to the outside internet, only to the MQTT broker (which is then in turn able to connect to the HomeKit hub). That can comfortably run on a Raspberry Pi Zero W.

The story for writing custom firmware for the IoT devices is a bit more complex. I started out with MicroPython, and then (after running into problems with devices stopping and requiring a reset), moved to an Arduino approach. Following that (and after meeting the author of MicroPython), I went back to that. But, as I thought more about it, there were a bunch of features that I wanted to implement, and my deployment processes were somewhat awkward.

I’d looked at the various packaged firmware: Espurna, and Tasmota in particular. All of these have a bunch of people using them (which is good, because then I don’t have to reimplement everything). But all of them were not quite able to do what I wanted, which was just send messages to a specific MQTT topic (and perhaps listen on that one too).

Then I came across ESPHome. This enables you to use YAML to define a device and it’s services, and is far more flexible than the prepackaged firmware. You can run the build tool in docker, which compartmentalises it and means you don’t have to worry about maintaining an installation of the tool. And, it can do OTA updates, as well as provide access to logs over Wifi!

So, now, I can have a really nice workflow for building and deploying firmware.

esphome:
  name: temperature_01
  platform: ESP8266
  board: nodemcuv2

wifi:
  ssid: "SSID"
  password: "PASSWORD"

# Enable logging
logger:

ota:

i2c:
  - id: bus_a
    sda: D2
    scl: D1
    scan: True

sensor:
  - platform: bmp280
    i2c_id: bus_a
    address: 0x76
    temperature:
      name: "Temperature 1"
      state_topic: HomeKit/temperature_01/TemperatureSensor/CurrentTemperature
    pressure:
      name: "Inverter pressure 1"
  - platform: wifi_signal
    name: "WiFi signal sensor"
    update_interval: 60s

mqtt:
  broker: "mqtt.lan"
  discovery: false
  topic_prefix: esphome/temperature_01

In addition to this, I came across the tuya-convert package, and Mirabella Wifi Bulbs, which are $15 at K-mart, so now I have a couple of those. Although, I am still not happy with not being able to have the smarts at the switch, but since I don’t have neutral wires in the switch, that’s a problem.