Home > Espressif, FreeRTOS, Hardware, Intel Atom, Testing, Windows 10 > Karl’s Home Automation Project – Part 1: Home Assistant & YAML, MQTT, Sonoff, and Xmas Lights

Karl’s Home Automation Project – Part 1: Home Assistant & YAML, MQTT, Sonoff, and Xmas Lights

February 27th, 2017 Leave a comment Go to comments

Karl here. I am here to write about my home automation project. First thing I want to say is that I am very cost conscious and I don’t mind putting in extra effort into the setup of things to keep costs down. I did invest a lot of time and had to do a lot of reading to get my project going. It took while and I received a lot of groans from my wife while testing. I am still in the process of tweaking things.

I started watching a series of videos on YouTube from Bruh Automation. He introduced me to Home Assistant. It got me really excited. He uses a Raspberry Pi as a server but I already had a Wintel Pro CX-W8 Smart TV Box which I use as a server. I run 3 Minecraft Servers, Emby Server, iSpyConnect DVR (2 IP Cameras), Unifi wifi controller, and now MQTT Server, and Home Assistant. Below is screenshot of mostly idle.

If it weren’t for iSpy it would be around 5-10% most of the time. Emby transcoding is the only thing that is stressful and it is not used much. The reason I mention this is because after purchasing a Raspberry Pi with power supply and case, you are not far off from getting a z8300 box. Only downfall is dreaded Windows update auto reboot. I finally looked into it and disabled it. If you decide to use a Windows box, I would make sure you are running 64bit windows. One advantage to using a Raspberry Pi is there is an image on Home assistant with the basics pre-configured and just need to write it to an SD card.

Server side Setup

I won’t go into too much detail on server side, as I installed Python, Mosquitto, and Home Assitant (I followed the guide on their site for Windows)

Python was a breeze to install and just ran the executable and went with defaults. I already had it installed for something else and I am running 3.5.2 64-bit. There are newer versions now. Mosquitto was the most difficult. I followed this guide but substituted Win32OpenSSL_Light-1_0_2j.exe approx 2MB. A k version is available now. Home assistant was easy and used pip.

Christmas Lights

It was a little before Christmas when I started researching home automation. I had been reading about these inexpensive Sonoff devices here on CNX and I found a project on Github for some custom firmware by arendst that enabled them to be controlled by MQTT. (While getting the link it looks like a new project has started with some additional features here). My wife really likes decorating for Xmas and we have 3 trees and lots of lights. She mentioned getting some timers and boom I had my opportunity and ordered them the same night. After receiving It took me a couple nights and I had a simple automation turning Xmas lights on and off at specific times and life was good. I got an extra one to play with until Xmas was over. I redeployed the rest  around the house after Xmas.

MQTT

I really had no idea what this was and it took me a while to grasp. You can use a cloud based MQTT if you would like, but I prefer to run my own. MQTT is a service that relays messages between devices. There are 2 main items topics and payloads. To be able to tell a switch to turn on you send payload “on” to a topic, for example, “cmnd/testbench/power”. The light turns on and it replys back to a topic “stat/testbench/POWER” confirming that the light is on and the message is received. Because we are sending “on” to the topic each device using MQTT will need its own topic. Topics are case sensitive. I made a batch file to subscribe to all topics for troubleshooting so I could monitor the messages. The # indicates all sub topics.

Sonoff

I picked the Sonoff basic but there are also different varieties that add additional features which are supported by arendst software.

Arendst  has been very active with this project and adding/tweaking daily. When I first flashed the device, I did find a defect and notified him and he had it fixed and uploaded within the hour. He has very detailed instructions on the Wiki. First step before flashing is soldering headers. (I link to bent headers…which I initially thought I made a mistake but turned out it was good. They are easy to straighten) A USB to TTL adapter is also needed to upload from Arduino IDE. I recommend one like this because it provides both 3.3 and 5V.  After downloading and setting Arduino up, I only set my WiFi password and SSID in the sketch. After it boots the first time, it connects to your wireless network. Find the IP address in your router, and pop the IP address in your browser to finish the configuration. Set the MQTT server credentials and topic and your done. I never setup credentials on the MQTT server so it accepts any login. Finally after everything is programmed you need to connect it to mains. Beware do not connect mains while TTL is connected.  I bought some extension cords locally. Cut them in half and stripped back a ¼ inch of the insulation. Extension cords use stranded wire so I tinned them with solder to avoid any stray strands from shorting out. Then I screwed them down on the terminals making sure polarity was correct.

Click to Enlarge

YAML

YAML is unforgiving. It is the formatting that you configure Home Assistant in. A single space will stop Home Assistant from starting. Luckily on this last update if you restart Home Assistant through the browser it will test the configuration file before actually restarting. I purposefully put an extra space on line 54 to show it is easy to find any mistakes.

Click to Enlarge

I also recommend Notepad++ for editing in windows. You can break your configuration down into different files but I like one. Notepad ++ allows you to collapse the parts you aren’t currently working on.

I recommend adding one thing at a time and restarting to make it easier to find errors. And making a copy of the last working config before adding more. In the config below there are 5 sonoff’s and an automation to turn the lights on and off at specific times. This is extremely basic. I also recommend setting up one new device and be conscious of naming. When you get your config working properly on your first new device I copy the config to a new blank text window and do a find/replace.

Below is the screen capture of collapsed parts, and and full config (minus personal info).



Notice the test bench is on later firmware and the MQTT topic is slightly different

Next Steps

So now I have a smart home, right? Not in my opinion. I can turn lights on and off with a schedule or with my smart phone or at the light by pressing the button on the Sonoff. To me this is not smart. Setting a schedule is OK, but then you have the lights on unnecessarily and wasting electricity. Only real option is to press a button on the Sonoff but what difference is that than flipping a switch. Taking your phone out takes way too long, and I feel like it is going backwards. Below are estimated costs so far. By far the Windows Box will be the most expensive part if you choose to go that way. You can re-purpose just about anything that runs Linux to be a server. One other option is to run Linux on an S905x.

Money Spent

Cost of server not included nor shipping.

Item Qty Price Total
Sonoff Basic 5 $4.85 $24.25
Headers 1 $1.50 $1.50
USB to TTL 1 $2.54 $2.54
Total $28.29

If you find this entertaining or want me to go more in depth on a specific aspect let me know in the comments. I have been finding my time setting it up very satisfying. I am able to do some hardware and software work. I hope this might get your interest in home automation going, and find out it is not hard nor expensive. I would like to state none of the products linked were provided by the sites. I purchased with my own money.

The plan right now is to do a 3 part post. In the next post, we will integrate some inexpensive motion sensors and door sensors using 433mhz, then finally modifying the sensors to include a light intensity sensor.

Continue reading “Part 2: 433 MHz / WiFi MQTT Bridge, Door & PIR Motion Sensors“.

  1. Harley
    February 27th, 2017 at 18:45 | #1

    Will you use RFLink Gateway for 433MHz (and 315MHz) sensors? It is the best:

    http://www.nemcon.nl/blog2/

    Bought mine from nodo-shop.nl and Arduino clone from Aliexpress on the cheap:

    https://www.nodo-shop.nl/nl/21-rflink-gateway

    Slowly migrating away from 433MHz but still have light and door sensors.

    I also use Sonoff too, but have Domoticz instead now, and looking at OpenHAB.

    Tip; also bought DHT22 temperature/humidity sensors to connect to each Sonoff.

    http://tinkerman.cat/tag/sonoff/

    Xose Pérez at tinkerman.cat linked above has some tips for cheap Sonoff hacks.

    I have also bought cheap Z-Wave USB stick as well, but devices are expensive.

    Another tip is to buy TeckNet BlueTek powerbanks to use as cheap server UPS.

    Otherwise running a similar setup for same purpose, and to do it on the cheap.

  2. Theguyuk
    February 27th, 2017 at 19:42 | #2

    So you are worried about the cost of powering a light bulb, so you turn it off by powering a windows atom TV box.

    So cost of atom TV box and part, plus labour and postage, how long to recover your initial costs + cost of running atom TV box?

  3. Karl Johnson
    February 27th, 2017 at 20:32 | #3

    Harley :
    Will you use RFLink Gateway for 433MHz (and 315MHz) sensors? It is the best:
    http://www.nemcon.nl/blog2/
    Bought mine from nodo-shop.nl and Arduino clone from Aliexpress on the cheap:
    https://www.nodo-shop.nl/nl/21-rflink-gateway
    Slowly migrating away from 433MHz but still have light and door sensors.
    I also use Sonoff too, but have Domoticz instead now, and looking at OpenHAB.
    Tip; also bought DHT22 temperature/humidity sensors to connect to each Sonoff.
    http://tinkerman.cat/tag/sonoff/
    Xose Pérez at tinkerman.cat linked above has some tips for cheap Sonoff hacks.
    I have also bought cheap Z-Wave USB stick as well, but devices are expensive.
    Another tip is to buy TeckNet BlueTek powerbanks to use as cheap server UPS.
    Otherwise running a similar setup for same purpose, and to do it on the cheap.

    I found a diy gateway that works really well and very cost effective.

  4. February 27th, 2017 at 20:50 | #4

    I think that’s OK because he would run the box anyway for his other purposes. It probably consumes about 4 to 6Wh, but would run whether he had a control system for his light bulbs or not.

  5. Karl Johnson
    February 27th, 2017 at 20:52 | #5

    Theguyuk :
    So you are worried about the cost of powering a light bulb, so you turn it off by powering a windows atom TV box.
    So cost of atom TV box and part, plus labour and postage, how long to recover your initial costs + cost of running atom TV box?

    I already had it running those other servers for a while now. A PI can be used if starting from scratch. I use the Atom specifically for it’s built in Quick Sync transcoding support supported by Emby. I would need a considerably beefier system to do CPU transcoding.

    I also have overhead fans with 4 candelabra bases. I can get $1 led lights with standard bases at the dollar store but to replace the candelabra incandescent right now is expensive. These are connected to the switches that everyone is trained to turn on. I am using motion to automatically use the led lamps.

  6. Harley
    February 27th, 2017 at 20:55 | #6

    @Theguyuk
    No you have misunderstod. We are not really worried of the cost of electricity per lightbulb. When we say “cost” here we are refering to the cost of home automation devices, which we don’t want or can’t spend a fortune on.

    Modern home automation devices such as the latest wireless power relays and sensors are generally known for being expensive. This means that everyone thinks that you have to pay a lot to build a home automation system.

    What we are saying here is that you can build a relativly cheap home automation system by using Sonoff devices , some 433Mhz sensors, and a free and open source home automation server instead of a retail gateway.

    To summerize; this is about building a Smart Home Automation system on a limited budget.

  7. Harley
    February 27th, 2017 at 21:04 | #7

    @Karl Johnson
    RFLink Gateway is a DIY project as well, and it is fully open source. Both open source hardware and open source firmware/software. No other RF Gateway supports as many devices as RFLink. Highly recommend it over all others:

    http://www.nemcon.nl/blog2/devlist

    RFLink firmware also supports communication via 2.4Ghz as well as reception at 868 and 915 Mhz (but not on the same board, as you currently will need a seperate board per frequency).

  8. Kris
    February 27th, 2017 at 21:27 | #8

    I loved the article.

  9. Theguyuk
    February 28th, 2017 at 01:05 | #9

    @Karl Johnson
    Have more fun, use a Nanopi !

  10. February 28th, 2017 at 04:21 | #10

    Nice project Karl. If you have patience for few more weeks to finish some projects I intend to make public the MQTT broker on an ESP8266, I just need to create a nice web interface to allow users to create an account on one of my servers so they will get the broker address with a simple HTTP call ( no need to check the router for IP addresses). See more details on https://myesp8266.blogspot.nl/2016/11/mqtt-broker-on-esp8266-5.html .This projecct started on an early development board from EspressIf ESP32. Then I moved to ESP8266 just to check its capabilities and is working fine. Up to 25 connections ( Sonoffs) and a bridge connection to a cloud where I have mosquitto instances. Also adding voice control to Sonoff modules is very simple. For that see: https://myesp8266.blogspot.nl/2016/12/amazons-alexa-control-esp8266.html

  11. Petr
    February 28th, 2017 at 09:43 | #11

    As you suggest flipping the light manually or some time trigger is not particularly smart. A box standard timer socket could do it. Why don’t you use the sunset as a trigger for turning the lights on? A number of triggers spring to mind for the off trigger. Simple time, sunrise, motion detection, TV socket power, late shift detection with a calendar, etc. Depending on personal circumstances.

    I too am often struggling to think of the ‘smart’ condition, so my smart devices make sense.

  12. Harley
    February 28th, 2017 at 18:39 | #12

    @Katalin Sounds like a interesting project with a lot of potential if you would build on it. So I was wondering if you have thought anything about using a Mesh networking concept to setup a distributed system of MQTT brokers that could share the load and that way both get redundant MQTT brokers and allow more clients to connect to that MQTT mesh?

    That is, I don’t like the idea of relying of a cloud service as I want everything to always be 100% locally controlled on my local network, however I think that it would be very cool if you or someone else made a firmware for ESP8266 devices that allowed several devices such as Sonoff to automagically setup their own self-sustained and self-healing mesh-network nodes of MQTT brokers and clients so that all devices talked each other, working totally on their own without the need for a cloud service or a central server with MQTT broker on a Raspberry or PC.

    With a such distributed system you could in theory support many more clients if you could get several of them to act as a MQTT brokers to share the load with each other. That way each device would as a “mesh node” act as both as a server and as a client in the distributed system and the more devices you add then the more numbers of clients you could connect. As even if each ESP8266 had a MQTT broker that could only connect to 3 MQTT clients and another 2 MQTT brokers then adding more devices would exponentially grow the amount of MQTT client you can connect. Having for example 20 devices would allow you to connect 60 MQTT clients in total and that mesh-network would have 20 MQTT brokers for redundancy, all sharing the total load of the mesh.

    If you could also create add some kind of roaming master-slave high availability or cluster service that could keep track of which device currently acts as MQTT broker master coordinator/controller and which are MQTT broker slaves then the distributed system would have redundancy so that one of the devices would always be the MQTT broker master coordinator/controller , and if that goes down then the other MQTT broker slaves would re-elect a new master coordinator/controller . Just like in a standard high availability or cluster service. You then normally have one roaming VIP (Virtual IP) that the current master gets as a second IP so that you do not need to hunt for which one is the current master.

    That way users would not have to rely on a cloud service or setup their own server with MQTT broker as all devices would act as both servers and clients, with all devices.

  13. Harley
    February 28th, 2017 at 18:39 | #13

    @Petr
    Cheap 433Mhz twilight relay/switch (light sensor) can also help to make ‘smart’ condition/scenario if you have a good place to put it, as unlike sunset and sunup a twilight relay can also detect if it is an cloudy overcast so that you therefor want to turn the light of earlier or leave them on later than your normal sunset and sunup + delay scenario.

    If you combine the data from a twilight relay with a the sunset/sunup then you can for example create a ‘smart’ condition/scenario where the twilight relay can turn the lights on within 2-hours before sunset if the twilight relay detect that it is dark enough. And the same for the morning there another ‘smart’ condition/scenario should only turn the lights off 1-hour after sunup if it is light enough enough.

  14. tkaiser
    February 28th, 2017 at 19:32 | #14

    @Harley
    I wonder whether this ‘cheap 433Mhz’ stuff is safe/secure to operate (especially when doing a web search for ‘433mhz replay attack’ or ‘rf replay attack’)?

    Same with people storing their Wi-Fi credentials on cheap IoT nodes running vendor firmwares or prone to physical access by unauthorised people (I would never do this but instead use at least something as an Orange Pi Zero as AP to create an own IoT Wi-Fi for ESP8266/SONOFF devices where instrusion detection could work at least somehow)?

    Talking to neighbours recently I was reminded that most people still think Wi-Fi pseudo security like MAC address filters or ‘hiding SSID’ would work (which is not the case of course). Maybe it would be a good idea to prepare a follow-up blog post about ‘smart home automation’ without exposing your home to everyone around? Most users seem to be not sensitized for all the security issues involved connecting their home to ‘someone else’s servers somewhere out there’ (euphemistically called ‘cloud’ by marketing guys all the time)

  15. February 28th, 2017 at 20:31 | #15

    @Harley
    That’s the beauty of my broker. If I am at home my mobile app will connect over the websockets to the broker and will send/receive MQTT data to/from my devices. I am using the cloud only to control the devices when I am not at home. I don’t rely on the cloud data also, but if I am home I am connecting direct to the ESP8266 if not through the cloud. If I am at home and the internet connection is down, I am not affected at all. See this link: https://myesp8266.blogspot.com/2016/04/since-i-received-lot-of-questions-about.html

  16. tkaiser
    February 28th, 2017 at 20:40 | #16

    Doing a web search for ‘hta-r03-light-weight-protocol-serious-equipment-critical-implications.pdf’ is also somewhat frightening (slides and talk on youtube about ~60,000 publicly available insecure MQTT brokers)

  17. February 28th, 2017 at 20:57 | #17

    @tkaiser
    Nothing is secure, even electronics in cars are not secure according to Masayoshi Son (ARM’s new owner).
    I don’t think things will change unless some laws are in place to make somebody responsible (e.g. not only the hacker himself) in case of a security breach. The problem is that currently security also relies on the user not doing something stupid, and it’s not realistic to think end user will understand it all. So products would have to be secure whatever the user does.

  18. tkaiser
    February 28th, 2017 at 22:31 | #18

    cnxsoft :
    @tkaiser
    Nothing is secure, even electronics in cars

    Just do the web search for ‘hta-r03-light-weight-protocol-serious-equipment-critical-implications.pdf’, there are examples about publicly available MQTT brokers that collect/send data from/to cars. It’s scary to be able to subscribe to a MQTT broker with F_SPEED, F_SW_BRAKE and so on exposed (and most probably accessible too). Same applies to ‘dumb homes’ (that’s IMO the better name for so called ‘smart homes’ connected to some ‘cloud’)

  19. Petr
    March 1st, 2017 at 07:39 | #19

    @Harley
    Good idea.

  20. Petr
    March 1st, 2017 at 07:59 | #20

    @tkaiser
    So I have just “wasted” an hour to play with Shodan and am gobsmacked how much info is available publicly. Let’s see this out of interest https://www.shodan.io/host/24.67.211.138

    excerpt:
    Eric and Cailyn from Salmo in Canada
    running an unsecured MQTT broker

    Topics:
    owntracks/eric/MotoX
    owntracks/caitlyn/caitlyn’s phone
    topic
    arduino/temperature/living_room
    arduino/living_room/temperature
    arduino/living_room/touch_sensor
    arduino/living_room/touch_sensor/set
    arduino/livingroom/temperature
    arduino/livingroom/touch_sensor

    You can easily subscribe to the phones’ location updates via owntracks publisher. Read their home sensor statuses and possibly push a payload to switch things on or off.

    Oh, and Look Eric is on the move…

    Scary stuff.

    Encrypted MQTT even as a first line of defence.

  21. Harley
    March 1st, 2017 at 14:33 | #21

    tkaiser :
    @Harley
    I wonder whether this ‘cheap 433Mhz’ stuff is safe/secure to operate (especially when doing a web search for ‘433mhz replay attack’ or ‘rf replay attack’)?

    I only use 433Mhz sensors, like a twilight light sensor and a few door sensors, not any power-relays. So the 433MHz sensors is only for non critical infrastructure. To turn lights off or on if someone opens or close a door.

    Also remember that 433Mhz sensors is not connected to the internet in any way, so I’m not paranoid enough to think someone will physicially come to my house and send 433MHz ‘relay’ signals to screw with my lights.

    What would be point of that? It is not like I’m using 433MHz for security stuff like an alarm.

  22. tkaiser
    March 1st, 2017 at 15:27 | #22

    @Petr
    Now add ‘cloud’ to the picture. This here (cloud connected ‘toys’) is a random example how everything cloud connected (Hey, Alexa) ends up eventually: Everything exposed to anyone.

    https://www.troyhunt.com/data-from-connected-cloudpets-teddy-bears-leaked-and-ransomed-exposing-kids-voice-messages/

  23. Theguyuk
    March 1st, 2017 at 17:43 | #23

    Here is a lovely silly one from BBC http://www.bbc.co.uk/news/technology-38966285

  24. Petr
    March 1st, 2017 at 17:55 | #24

    @Theguyuk

    Actually, I’ve found similar vulnerability in my system, by yelling loudly “OK, Google, open the garage door” from the outside of my house.

  25. Charis
    May 12th, 2017 at 06:45 | #25

    Hi! I want to ask you why you choose Sonoff switches for home automation, and not Xiaomi mi home ? I’m asking you because Xiaomi has very cheap sensors for your home automation system, is working with zigbee and a hub. I ‘m also on this dilemma between those 2 solutions.

    • Karl Johnson
      May 12th, 2017 at 07:34 | #26

      Really depends on the requirements. I liked the universal MQTT method to communicate. Have to weigh pros and cons. When I started i wasn’t aware or it wasn’t around. Not sure this helps but good luck. Also be sure if you choose a home automation software it works with the hardware. I need to check it node-red. It’s another option.

  26. Petr
    May 12th, 2017 at 08:04 | #27

    @Charis As Karl has pointed out Sonoff has been around possibly longer and now there is enough community development to upload a custom software on it, so you don’t need to rely on chinese apps to customise/control your device. I prefer the TASMOTA firmware that gives you a nice web interface, OTA updates, MQTT and many, many functions. I actually now use it on the Wemos D1 Mini boards too, these costing US$3-4 and have a ecosystem of add-on shields. This paired with HomeAssistant makes for very robust setup. Mind you yaml is a bit of a pain, but not a show-stopper.

  27. Charis
    May 12th, 2017 at 18:50 | #28

    Itead sonoff problem is that there are not much trigger devices for auto scenes. On the other side Xiaomi smart home has too many sensors that trigger events. But for 2 floor house i think is inposible to work with zigbee protocol because of lower range. And i think i need more than one gateways…

  28. Charis
    May 12th, 2017 at 18:57 | #29

    @Charis
    (sorry for bad english… inposible = impossible)

  29. Karl Johnson
    May 12th, 2017 at 19:18 | #30

    Petr :
    @Charis As Karl has pointed out Sonoff has been around possibly longer and now there is enough community development to upload a custom software on it, so you don’t need to rely on chinese apps to customise/control your device. I prefer the TASMOTA firmware that gives you a nice web interface, OTA updates, MQTT and many, many functions. I actually now use it on the Wemos D1 Mini boards too, these costing US$3-4 and have a ecosystem of add-on shields. This paired with HomeAssistant makes for very robust setup. Mind you yaml is a bit of a pain, but not a show-stopper.

    You might want to check out 2 other updates

    http://www.cnx-software.com/2017/03/02/karls-home-automation-project-part-2-433-mhz-wifi-mqtt-bridge-door-pir-motion-sensors/

    http://www.cnx-software.com/2017/03/27/karls-home-automation-project-part-3-adding-infrared-light-detection-to-a-motion-sensor/

  1. No trackbacks yet.