Archive

Posts Tagged ‘mqtt’

Karl’s Home Automation Project – Part 4: MQTT Bridge Updated to Use YS-IRTM IR Receiver & Transmitter with NodeMCU

April 20th, 2017 1 comment

In a previous article, I wrote about an MQTT bridge by 1technophile. I added a DHT temperature and humidity sensor as well as a light sensor. Previously it included a software decoder to decode the IR signal. I never did test the IR transmitter on the gateway, as I didn’t have the parts. But thanks to IC Station, who sent me over a small YS-IRTM hardware based decoder and NodeMCU that I am writing about today. I have replaced the software based version with the YS-IRTM module in the latest update.

Click to Enlarge

Click to Enlarge

I found this project challenging. I admit I am a little weak in my programming skills. It was difficult to find documentation but I found a forum talking about this device and basics of how it works. When an IR code is recognized it sends 3 hex codes via serial connection on the transmit pin. To transmit, it expects 5 hex codes: A1,F1,xx,xx,xx. A1,F1 tells it to send the following codes. You can also set the baud rate but I left default 9600.

It is simple wiring wise. It only takes 4 dupont wires. It took a bit of coding to get it working but I finally got it to communicate via software serial. I started on a Arduino Uno with the code and then migrated it over to the ESP8266 board. I did have a little trouble when I first moved to the ESP board. I initially thought I might need a level shifter but that didn’t help. I am a little surprised I didn’t need a level shifter because the ESP needs only 3.3 volts. I was getting some weird responses and finally figured out I had to put in a slight delay. Maybe the ESP’s speed comes into play.

The way to use this is fill out your SSID and password and your MQTT server with credentials. Flash the device. You will need to add the necessary libraries. 1technophile has good documentation in his wiki.

Once flashed and ready to find your IR codes you will need to subscribe to the topics with the Windows command below. Give the gateway a moment to connect and point your IR remote at the sensor and press a button to find out code.

In your window, you will get something like this “home/sensors/ir 4,fb,8,” which is my power button for my TV. To test the code:

With this code, the TV will toggle on and off.

Click to Enlarge

After this you can use your favorite home automation project and control your IR devices with automations. You can omit any sensors that you don’t need. You will get some erroneous MQTT data if not all sensors are used. Below are the bits of Arduino code added for the IR module, and here’s the link to the github code:

I plan on 3D printing an enclosure with CR-10 I am reviewing, and I will remove the IR LED, and move it to a more suitable position, as both facing the same way isn’t ideal for my setup.

I would like to thank IC Station for sending the NodeMCU ($5.81 shipped) and IR transmitter and receiver ($3.39 shipped) for review. You can get 15% discount with coupon Karics. I finally have a complete gateway.

$12 AI Light ESP8266 based WiFi RGB Light Bulb Supports MQTT via ESPurna Open Source Firmware

March 3rd, 2017 20 comments

AI-Thinker is famous in the maker world for their ESP8266 modules, but they’ve also recently launched a WiFi RGB light bulb that sells for about $12.5 and up on Ebay and Aliexpress (here and there). Some people noticed, and bought samples online, including Xose Pérez (aka Tinkerman), ESPurna open source firmware developer, who could confirm ESP8266 was used in the light bulb, did some investigations, and eventually added the light bulb into ESPurna, which means it can be managed using MQTT or a web interface.

AI Light looks very similar to Philips Hue, but comes with WiFi instead of Zigbee. AI Light “M1636” key features:

  • RGBW LED E27 bulb with 16.7M colors
  • Connectivity – 802.11 b/g/n WiFi
  • Encryption – AES
  • Voltage Range – 110-240V
  • LED Power – 5 watts
  • WiFi Power Consumption – ≤0.3W
  • Temperature Range – -5~45degree
  • Humidity – ≤80%
  • Certifications – FCC, CE, ROHS

If you’re going to use the stock firmware, you can control the LED with Tuyasmart Android app. You’ll find the user’s manual and more photos on the FCC page for the light bulb. But there are already plenty of Wifi light bulbs on the market,  and what makes this light bulb interesting is that it’s based on ESP8266, and you can have full control over it using open source firmware.

The bulb cap is allegedly very easy to pop out, as it’s not glued to the board.A close up on the board itself reveals it’s indeed powered by Espressif ESP8266EX WiSoC connected to a 1MB Winbond 25Q80BVSIG flash, and MY-Semi MY9291QD LED driver.

Click to Enlarge

If you look from the bottom left to middle left of the inner circle, you’ll see 3V3, GND, RX, TX and IOO pads, which we can use after soldering some wires, and connect a USB to TTL board in order to flash the firmware. Note that IOO must be connected the GND to enter flash mode, you can remove the wire after flashing, in order to check the serial output during a normal boot.

After further investigation, Xose found out that there’s already some software implementation for MY9291 driver in Noduino OpenLight project, made by the developers who designed Noduino ESP8266/ ESP32 boards, and are likely the developers of AI Light. All needed source code can be found in Noduino-SDK released under a GPLv3 license, and includes a driver written in C language for MY9291 LED driver chip. Xose wrote a wrapper to make the driver work with Arduino ESP8266, and released the code on Github.

The code sample below shows how to set the LEDs to RED color at 100% duty cycle:

Ai-Thinker Ai Light / Noduino OpenLight have now been added to ESPurna 1.6.8 firmware, and you can turn the light on and off, select the color from the web interface, and/or control it via MQTT.

ESPurna installed on AI Thinker Light Bulb – Click to Enlarge

Karl’s Home Automation Project – Part 2: 433 MHz / WiFi MQTT Bridge, Door & PIR Motion Sensors

March 2nd, 2017 11 comments

Karl here again for part 2 of my home automation project. We will be looking at how to automate your lights based on time of day and motion. In the first part we setup Home Assistant and uploaded firmware to basic Sonoff Wifi switches. Today we will setup a 433 MHz to MQTT bridge and some sensors.

433 MHz

Depending on your country 433 MHz is an open frequency to use to communicate with. There are hundreds of different types of devices that use 433 MHz to communicate information. We will be focusing on 2 today from Gearbest: WMS07 motion sensor (left) and WDS07 door/window sensor (2 parts, right).

I am not taking the door/window sensor apart, since it is super basic, but I’ve included some photos of the PIR motion detector.

Click to Enlarge

Click to Enlarge

433 MHz Bridge

While contemplating how to get presence on a per room basis I ran across this project. It monitors 433 MHz signals and publishes it to the MQTT server. It is a really an easy project. It also has an IR to MQTT feature. I did have an IR receiver and tested it but have not implemented it. He has some good instructions on his page so I won’t go over too much. You can do the bare bones version and just leave off the unused sensors. I also went an extra step and added a light intensity sensor and DHT sensor to the project. It can be found here. I am not going to add those to this write up because trying to keep costs down.

What you will need is

  1. NodeMCU $5.69
  2. H3V4F Receiver Module $1.21
  3. Prototyping board $2.88

That is all that is needed. For about $10 you have an inexpensive 433 MHz bridge. You can put in a box if you want and hide it in a central location away from interference. I would suggest soldering headers to your board just in case something goes bad. I didn’t at first and made my life a pain. There are a ton of 433 MHz receivers. I purchased all the ones on Gearbest and this is by far the best. I did upgrade to a superheterodyne but I am not sure it is any better. I upgraded because I wanted to put the door sensor on my mailbox and get a notification when the mail was delivered. It is about 200’ away and is a little spotty even with a new 433 MHz receiver. I used this antenna design (see picture on right), as it seemed to work the best

Coverage is the biggest concern.  I have a brick single story ranch style home about 2000 square feet and it covers the inside with ease and a lot of the area around the house. If you have a multi-story house or would need multiple receivers you would need to change the MQTT topics to avoid getting duplicates. Below is the final project. To be honest temperature is really the only thing that is useful to me, but wanted to see what could be done. I purchase the DHT11 and the readings are not good. If you want to do this go with the DHT22. Below is a mostly loaded bridge. I don’t have an infrared transmitter yet. I have a different one coming that does the encoding/decoding on a chip and will follow up when I receive it. I am hoping it will be easier/better than using the Arduino library.

Motion Sensor

The motion sensor itself is really easy to setup with jumpers. I suggest turning the LED off, and the time to 5 min after finished setting up with the jumpers. If you notice there is a micro switch in the top left of this picture. It is meant to be a tamper switch but I use it as a toggle switch to quickly turn off the lights. The motion sensor is meant to be used for a security system but I just have them sitting on night stands and corner tables. It works really well to override or turn a light on when Home Assistant ignores the motion. A little squeeze of the box and the light will toggle states on or off.

After your bridge is set up and connected take the motion sensor out and put some batteries in it. Run your batch file to see what code is being sent. For this one we need 2: motion and tamper. Write these codes down.

Home Assistant

Below is the YAML code that I am using with Home Assistant. I made it find and replace friendly. If you copy and find the 4 items below it should work. I think it is relatively easy to follow. It is the typical timed lamp on motion that is on Home Assistant website with some slight modifications. I had to add the turn off motion script because the motion sensors only sends when it senses motion. I also had to add the tamper toggle switch. When you are adding multiple sensors you can only have one “binary_sensor:” group and one “automation:” group etc.

Find/Replace Explanation
generic use livingroom or masterbedroom etc no spaces
5555555 use the motion number you found earlier
8888888 make up a number around your tamper/motion number
9999999 use the tamper number you found earlier.

 

Door Sensor

For the mailbox sensor here is an example. Same thing on this one run the batch file and find the open and closed codes. I have it send me a notification via pushbullet.

Almost there

We are almost there. Lights are turning on and off magically. Life is good. But there is one situation where it’s not so good. The gloomy day. With the automations above we cannot determine if the blinds are pulled or it is gloomy. We still need the lights to come on under those circumstances to make it really cool. In the next installment we are going to take the motion sensors above and add a light intensity sensor to them. We will be able to do this cheap. We still have a pretty good budget. With the bridge above you open yourself to a bunch of battery operated sensors. You can also control devices, as well, with a transmitter. Any of the transmitters should work on GearBest. You can get the one linked and throw away the receiver. It’s only $1.25.  If you have any questions or concerns feel free to leave a comment.

Item Qty Price Total
Initial Setup Sonoff Basic 5 $4.85 $24.25
Headers 1 $1.50 $1.50
USB to TTL 1 $2.54 $2.54
$28.29
Motion Sensors NodeMCU 1 $5.69 $5.69
H3V4F Receiver 1 $1.21 $1.21
Prototyping board 1 $2.88 $2.88
Motion Sensor 4 $7.03 $28.12
$37.90
Grand Total $66.19

Continue reading “Part 3: Adding Light Detection to a Motion Sensor“.

Karl’s Home Automation Project – Part 1: Home Assistant & YAML, MQTT, Sonoff, and Xmas Lights

February 27th, 2017 24 comments

Karl here. I am here to write about my home automation project. First thing I want to say is that I am very cost conscious and I don’t mind putting in extra effort into the setup of things to keep costs down. I did invest a lot of time and had to do a lot of reading to get my project going. It took while and I received a lot of groans from my wife while testing. I am still in the process of tweaking things.

I started watching a series of videos on YouTube from Bruh Automation. He introduced me to Home Assistant. It got me really excited. He uses a Raspberry Pi as a server but I already had a Wintel Pro CX-W8 Smart TV Box which I use as a server. I run 3 Minecraft Servers, Emby Server, iSpyConnect DVR (2 IP Cameras), Unifi wifi controller, and now MQTT Server, and Home Assistant. Below is screenshot of mostly idle.

If it weren’t for iSpy it would be around 5-10% most of the time. Emby transcoding is the only thing that is stressful and it is not used much. The reason I mention this is because after purchasing a Raspberry Pi with power supply and case, you are not far off from getting a z8300 box. Only downfall is dreaded Windows update auto reboot. I finally looked into it and disabled it. If you decide to use a Windows box, I would make sure you are running 64bit windows. One advantage to using a Raspberry Pi is there is an image on Home assistant with the basics pre-configured and just need to write it to an SD card.

Server side Setup

I won’t go into too much detail on server side, as I installed Python, Mosquitto, and Home Assitant (I followed the guide on their site for Windows)

Python was a breeze to install and just ran the executable and went with defaults. I already had it installed for something else and I am running 3.5.2 64-bit. There are newer versions now. Mosquitto was the most difficult. I followed this guide but substituted Win32OpenSSL_Light-1_0_2j.exe approx 2MB. A k version is available now. Home assistant was easy and used pip.

Christmas Lights

It was a little before Christmas when I started researching home automation. I had been reading about these inexpensive Sonoff devices here on CNX and I found a project on Github for some custom firmware by arendst that enabled them to be controlled by MQTT. (While getting the link it looks like a new project has started with some additional features here). My wife really likes decorating for Xmas and we have 3 trees and lots of lights. She mentioned getting some timers and boom I had my opportunity and ordered them the same night. After receiving It took me a couple nights and I had a simple automation turning Xmas lights on and off at specific times and life was good. I got an extra one to play with until Xmas was over. I redeployed the rest  around the house after Xmas.

MQTT

I really had no idea what this was and it took me a while to grasp. You can use a cloud based MQTT if you would like, but I prefer to run my own. MQTT is a service that relays messages between devices. There are 2 main items topics and payloads. To be able to tell a switch to turn on you send payload “on” to a topic, for example, “cmnd/testbench/power”. The light turns on and it replys back to a topic “stat/testbench/POWER” confirming that the light is on and the message is received. Because we are sending “on” to the topic each device using MQTT will need its own topic. Topics are case sensitive. I made a batch file to subscribe to all topics for troubleshooting so I could monitor the messages. The # indicates all sub topics.

Sonoff

I picked the Sonoff basic but there are also different varieties that add additional features which are supported by arendst software.

Arendst  has been very active with this project and adding/tweaking daily. When I first flashed the device, I did find a defect and notified him and he had it fixed and uploaded within the hour. He has very detailed instructions on the Wiki. First step before flashing is soldering headers. (I link to bent headers…which I initially thought I made a mistake but turned out it was good. They are easy to straighten) A USB to TTL adapter is also needed to upload from Arduino IDE. I recommend one like this because it provides both 3.3 and 5V.  After downloading and setting Arduino up, I only set my WiFi password and SSID in the sketch. After it boots the first time, it connects to your wireless network. Find the IP address in your router, and pop the IP address in your browser to finish the configuration. Set the MQTT server credentials and topic and your done. I never setup credentials on the MQTT server so it accepts any login. Finally after everything is programmed you need to connect it to mains. Beware do not connect mains while TTL is connected.  I bought some extension cords locally. Cut them in half and stripped back a ¼ inch of the insulation. Extension cords use stranded wire so I tinned them with solder to avoid any stray strands from shorting out. Then I screwed them down on the terminals making sure polarity was correct.

Click to Enlarge

YAML

YAML is unforgiving. It is the formatting that you configure Home Assistant in. A single space will stop Home Assistant from starting. Luckily on this last update if you restart Home Assistant through the browser it will test the configuration file before actually restarting. I purposefully put an extra space on line 54 to show it is easy to find any mistakes.

Click to Enlarge

I also recommend Notepad++ for editing in windows. You can break your configuration down into different files but I like one. Notepad ++ allows you to collapse the parts you aren’t currently working on.

I recommend adding one thing at a time and restarting to make it easier to find errors. And making a copy of the last working config before adding more. In the config below there are 5 sonoff’s and an automation to turn the lights on and off at specific times. This is extremely basic. I also recommend setting up one new device and be conscious of naming. When you get your config working properly on your first new device I copy the config to a new blank text window and do a find/replace.

Below is the screen capture of collapsed parts, and and full config (minus personal info).

Notice the test bench is on later firmware and the MQTT topic is slightly different

Next Steps

So now I have a smart home, right? Not in my opinion. I can turn lights on and off with a schedule or with my smart phone or at the light by pressing the button on the Sonoff. To me this is not smart. Setting a schedule is OK, but then you have the lights on unnecessarily and wasting electricity. Only real option is to press a button on the Sonoff but what difference is that than flipping a switch. Taking your phone out takes way too long, and I feel like it is going backwards. Below are estimated costs so far. By far the Windows Box will be the most expensive part if you choose to go that way. You can re-purpose just about anything that runs Linux to be a server. One other option is to run Linux on an S905x.

Money Spent

Cost of server not included nor shipping.

Item Qty Price Total
Sonoff Basic 5 $4.85 $24.25
Headers 1 $1.50 $1.50
USB to TTL 1 $2.54 $2.54
Total $28.29

If you find this entertaining or want me to go more in depth on a specific aspect let me know in the comments. I have been finding my time setting it up very satisfying. I am able to do some hardware and software work. I hope this might get your interest in home automation going, and find out it is not hard nor expensive. I would like to state none of the products linked were provided by the sites. I purchased with my own money.

The plan right now is to do a 3 part post. In the next post, we will integrate some inexpensive motion sensors and door sensors using 433mhz, then finally modifying the sensors to include a light intensity sensor.

Continue reading “Part 2: 433 MHz / WiFi MQTT Bridge, Door & PIR Motion Sensors“.

How to use Sonoff POW with ESPurna Firmware and Domoticz Home Automation System

January 21st, 2017 12 comments

Sonoff POW is an ESP8266 based wireless switch with a power meter that comes pre-loaded with a closed-source firmware that works with eWelink app for Android or iOS by default. But we’ve also seen Sonoff POW, and other Sonoff wireless switches from the same family, can be flash with open source firmware supporting MQTT (Message Queuing Telemetry Transport) lightweight messaging protocol such as ESPurna, and I initially sent data from Sonoff POW to ThingSpeak via an MQTT broker (mosquitto) to draw some pretty charts. I did that with the switch connected to a lightbulb, but I’ve since installed Sonoff POW in my office to measure the room’s power consumption minus the ceiling light and aircon as shown below.

Wall Mounted Sonoff POW WiFi Switch – Click to Enlarge

Sonoff cable mechanism is really a pain for hard copper wires, as they are hard to push inside the mechanism, and something come out. I finally managed by it took longer than expected to install. I had to cut the mains cable, and rewire the gang box too. The good thing is that I did not need to drill a hole in my wall, as the device is very light.

I could see the power value updated in ESPurna web interface, depending on the load on my computer, and whether I turned on or off other devices. That’s all good, but instead of using ThingSpeak, whose open source implementation is not updated, I decided to try Domoticz, and already wrote a short guide showing how install Domoticz in NanoPi NEO ARM Linux development board. I had not gone through the setup yet, as I had to study a little more, and upgrade Sonoff POW firmware first. I also planned to use vThings CO2 monitor with Domoticz, but canceled since it can’t be configured remotely, and a USB connection is needed.

ESPurna OTA Firmware Update

So I’ll focus only on Sonoff POW in this post, and first we need to update the firmware since Domoticz support is only recent. I’ll assume you have already followed the post entitled How to Build and Flash ESPurna Open Source Firmware to Sonoff POW Wireless Switch.

First we need to update platformio and ESP8266 development platform to the latest version otherwise we’ll get some build issues:

I updated the source code with git pull, but for whatever reasons the build failed, even after cleaning the code. So I did what any developer with enough experience would do in that case: start with a fresh check out ;), and rebuild the OTA firmware from there:

In order to update the firmware over the network, you’ll need to change sonoff-pow-debug-ota section in platformio.ini with your own IP address (upload_port) and password (in upload_flags) used in ESPurna web interface:

Once it’s done, you can upgrade the firmware, and then the file system as follows:

The Sonoff POW will reboot, and cut the power for about 2 seconds after both updates. My Sonoff POW is controlling my computer power, but that’s OK since I’m behind a UPS. Now I can access the web interface, and one of the improvement is that you’re being asked to setup a new password right after the update.

Click to Enlarge

I was then redirected to the Status page showing power, voltage (a bit low?), current, and power factor.

Click to Enlarge

I then jumped to MQTT menu to set the IP address to my NanoPi NEO board, and clicked Update.

Click to Enlarge

There’s a new DOMOTICZ menu which we’ll check out a little later.

Installing and Configuring MQTT in Domoticz

ESPurna communicates with Domoticz via MQTT, so the first task was to follow and adapt Domoticz MQTT wiki.

First login to your Domoticz server (NanoPi NEO) and access a terminal window to update the packages, install npm, node.js, Node RED, and mosquitto:

We then need to go to the Hardware page in Domoticz and configure a new “MQTT Client Gateway with LAN Interface” as shown in the screenshot below.

Click to Enlarge

We can test whether it works or not by creating a new Dummy device in the same Hardware section

Then click on Create Virtual Sensors, to add a new Temperature sensor which we’ll call Fictive Temp.

Now go to the list of Devices (Setup->Devices) to check the idx value (1 in our case), and a publish a MQTT message to update the temperature value of our virtual sensor:

The temperature switch from 0 to 25°C. Our installation is working. Great!

Using Sonoff POW with Domoticz

In theory, we should be able to get two type of data for Sonoff POW: relay status and power levels. However, after looking at ESPurna source code, domoticz.ino only seems to handle the relay status that can be changed from Domoticz web interface, but the power values are only send in pow.ino to the MQTT server, with data not directly compatible with Domoticz. Maybe I missed something as Tinkerman – ESPurna developer – can use Sonoff SC to send temperature data to Domoticz. Alternatively, it might be possible to convert that data somehow with Node RED, but that’s something I’ll try later. So today, I’ll only try to control the switch from Domoticz.

To do so, I created another Dummy device called Sonoff POW Switch, and from there, another Virtual Sensor of Switch type.

Click to Enlarge

Click to Enlarge

We’ve already configured MQTT in ESPurna web interface, and from the screenshot above,we can see that “Sonoff POW Switch” Idx is 3, a value we need to update in the DOMOTICZ section of ESPurna web interface.

Now I can go Domoticz interface in my phone, and not my computer since my office’s Ethernet switch will be turned off, click on the Switch tab, and turn on and off Sonoff POW by clicking on the lightbulb as shown below.

Click to Enlarge

It works fine, however note that the initial switch status was wrong (off instead on on), despite the switch sending regular updates to the MQTT server.

NanoPi NEO Power Adjustments and Installation

Normally, at this stage, it should be easy simply install NanoPi NEO outside the office close to my router in the living room. But I’ve come across a few issues doing so, which I’m going to report.

First I decided to make a very short Ethernet cable to connect NanoPi NEO directly to my router. I have done a couple of Ethernet cables in the past a few meters long, and they all work. I tested my ultra short straight Ethernet cable connections with a multimeter, and the 8 wires were properly connected, however, when I connected NanoPi NEO to the router with that cable it failed to get a link. Maybe there was aonther issue with the cable, so I made another one just as short… Another fail. It turns out very short Ethernet cables may cause issues, which are normally solved by twister pairs, but with such short cables the length of the twisted pairs is also extremely short, maybe 2 to 3 cm which may not be sufficient. So I ended up using a “normal” 1.5 meter cable, not as neat but it works.

The power strip close to my router was full, and since I did not want to add another, I decided to use the spare USB port on my modem router in order to power NanoPi NEO board. A USB 2.0 port can only deliver 2.5W max, so I was clearly looking for problems here. In order to avoid an issues, I made use of h3consumption script to adjust the behavior of CPU cores and disable unused peripherals.

Let’s check NanoPi NEO current settings in a terminal:

h3consumption allows us to change the following settings:

So I decided to disable USB, and use two CPU cores at most in order to limit the board’s power consumption, and avoid random reboots:

The changes were properly applied after a reboot.

I powered the board with my modem router, and could use it without issue. I’ll monitor NanoPi NEO’s uptime to check if this works.

How to Use Sonoff POW ESP8266 WiFi Power Switch with MQTT and ThingSpeak

December 11th, 2016 11 comments

ITEAD Studio’s Sonoff is a family of cheap home automation products based on ESP8266 WiSoC, and I’ve already tested Sonoff TH16 wireless switch with a humidity and temperature sensor using the stock firmware and eWelink app for Android or iOS. It works, but up to recently it required a registration to a cloud service (the company will now allow use from the local network), and the source code is closed. So for the second device under review, namely Sonoff POW wireless switch with a power consumption monitor, I decided to install ESPurna firmware working on ESP8266 Sonoff devices and NodeMCU, as it’s open source, supports Sonoff POW natively, includes a web interface to control the device from the LAN, and includes an MQTT client.

MQTT (Message Queuing Telemetry Transport) is a lightweight publish/subscribe messaging protocol used to control IoT sensors and devices, and it’s a popular method to gather data from client to a MQTT broker to push the data to the cloud or a local database.

iot-sensors-mqtt-cloud

So typically, you’d have a bunch of sensor nodes (like Sonoff devices) communicating over MQTT to an MQTT Broker in your local network, which could be an OpenWrt router or a Linux development board like a Raspberry Pi, which in turns gets the data the the cloud to services like AWS IoT, Xively, or ThingSpeak. It’s also possible to use Cloud services to control MQTT devices remotely through the MQTT broker.

I eventually plan to use NanoPi NEO board to run both MQTT and ThingSpeak locally (not connected to the cloud) in order to monitor the power consumption of my small office, but since I’m all new to this, I’ve started experimenting by connecting a 30W light to Sonoff POW, and use a desktop computer running Ubuntu 16.04 for MQTT and ThingSpeak.

sonoff-pow-connection

Click to Enlarge

Click to Enlarge

Since I’ve already installed ESPurna firmware to the device, I disconnected the USB to serial board (important since Sonoff POW board has a hot ground), and connected it to the mains (220V in my location). That means we already have an MQTT client which first I had to configure.

Click to Enlarge

Click to Enlarge

Since it was the first time I connected a load to the device, I went to ESPurna’s status menu to check power usage was reported, and my 30 Watts light bulb was drawing 27 Watts. Close enough. I changed the hostname to sonoff-office, and setup two SSID in order to connect Sonoff POW to my local network in client mode, instead of using it in Access Point mode by default. You’ll need to tap on Update each time you modify the settings. Since the SSID must be entered manually, please note that SSID are case sensitive, e.g. CNX-SOFTWARE is different from cnx-software.

Click to Enlarge

Click to Enlarge

I wanted to calibrate the power using the 30W light bulb, so I entered 30W in AC RMS Active Power field, and tapped on Update, but the web interface reported “no changes”. I’m not sure how to use that part. Finally the most important part for this tutorial is to set the MQTT settings with MQTT IP address, and leaving other fields unchanged. However, you can change MQTT Topic field for example replacing /test/switch/{identifier} by /myiotstuff/{identifier}.

Now that our MQTT client is configured, I need to install mosquitto MQTT broker in Ubuntu:

mosquitto-clients is not really needed, but I’ll use it to test the MQTT broker a little later. Once you installed it, the MQTT Broker should already run automatically.

The last line of the log above shows a client connection from Sonoff POW. Now, we need to check the topic, and since ESPurna documentation is still work in progress, you could either check out the source code, or IMHO more fun, capture MQTT packet with tcpdump or Wireshark as I’ve done below.

Wireshark MQTT Capture - Click to Enlarge

Wireshark MQTT Capture – Click to Enlarge

Here we can see that Sonoff POW will send a Publich Message with the power level using the topic “/test/switch/sonoff-office/power29”.  “/test/switch” is the string we’ve defined in the web interface, “sonoff-office” the hostname we’ve given to Sonoff Pow, and “power29” indicates 29 Watts of power is currently used.

We can also start a client in Ubuntu 16.04 terminal window to check more MQTT topics with # wildcard for sonoff-office host:

We can use MQTT to get the IP address, firmware and file system version, hearbeat message, power use, and relay status (on or off).

It’s all good, but now we need to do something to draw the data, and possibly analyze it. I selected ThingSpeak for this purpose since it can be installed in the local network, or through their service in the cloud. By the end of my testing, I’ve noticed ThingSpeak has a new MQTT API, meaning it should be possible to connect your MQTT broker directly to it, but for this guide I use mqspeak instead as a bridge between MQTT and ThingSpeak. It may still be useful, as the open source version of ThingSpeak is not updated anymore, and lacks the MQTT API.

You’ll need Python 3 and pip3 to install mqspeak:

Once it’s done, we’ll need to create a config files as explained on mqspeak’s github repo, and I created /etc/mqspeak.conf with the following content:

Brokers are used to configure MQTT broker IP address and port, as well as the topic(s) to subscribe to, while Channels take care of ThingSpeak configuration with the channel’s Id and write API key, update rate in seconds (15s minimum), update type (see github for details), and fields defined in your ThingSpeak’s channel(s), which will create later on. I wrote one broker for the power consumption topic, and other for the relay status. However, I eventually ignored the relay status, as it’s not updated often enough and cause ThingSpeak’s channel to only be updated when the relay changes status, even if there are power updates in the meantime. A workaround is to use two different channels for ThingSpeak.

mqspeak connects directly to api.thingspeak.com, so if you are using ThingSpeak cloud services, the next step is to register an account and setup one or more channels.

Extra Instructions for a local installation of ThingSpeak

However, if you’ve installed ThingSpeak in Ubuntu 16.04 or other Linux operating systems locally or on your own server, you’ll need to change the server in the source code, and reinstall mqspeak.

  1. Get the source code:
  2. Modify mqspeak/sending.py to replace api.thingspeak.com using HTTPS with localhost (or other IP address/URL where you’ve installed ThingSpeak) with HTTP:
  3. Install mqspeak

An improvement would be to install a signed SSL certificate, like the one offered by LetsEncrypt and configure the rails server to use https instead. I have not setup ThingSpeak server to start automatically yet, so I have to start it manually for now:

End of instructions specific to local installation.

The instructions specific for the local installation of ThingSpeak are now done, and all instructions below are valid for both the local installation and cloud service. Now open a web browser, go ThingSpeak (cloud or local), and click on “Get Started Now” in order to register an account.

Click to Enlarge

Click to Enlarge

Once it is done, login and click on “New Channel”.

Click to Enlarge

Click to Enlarge

Give it a name, a description, create fields as needed, for example power-consumption and power-status, and click on Save Channel.  Update /etc/mqspeak.conf accordingly with the fields’ name, and channel Id.

thingspeak-api-keyNow select API Keys tab to copy and paste the write API key into mqspeak.conf.

Now we can start mqspeak:

ESPurna firmware will send a power update every 60 seconds (this can be changed in code/src/pow.ino), so you’ll see a new message pop-up every 60 seconds with your channels Id and write API key. I’ve let it run for about one hour, and got the follow chart in ThingSpeak after turning on and off the lights from time to time.
thingspeak-power-consumptionThat’s pretty cool, so it only shows the current power in watt, and we’d probably want to get power consumption in kW/h per day, week and month at some time, and I have yet to study how to do that, Exporting the data to excel would be a workaround if this can not be handled in ThingSpeak. ThingSpeak.com (but not the open source version) offers some Matlab processing of the data, so that’d be another options.

The next steps would be to install MQTT and ThingSpeak in NanoPi NEO board, enable HTTPS in ThingSpeak, autostart rails server and mqspeak at boot time, make ESPurna firmware publish the “Power” topic more often than every 6 second, and find some way to generate useful kW/h consumption charts from the data stored in ThingSpeak within ThingSpeak, or but exporting the data.

How to Install ThingSpeak in Ubuntu 16.04

December 7th, 2016 11 comments

Last week-end I installed ESPurna open source firmware with MQTT server on Sonoff POW WiFi switch, and the next step is find a way to draw power consumption charts in some web based interface. We could do this in the IoT cloud with services like Xively or ThingSpeak, but since one of the goals of replacing the default firmware was not to rely on a proprietary cloud based solution, I decided to find a way to draw those chart in a local server, and it so happens that ThingSpeak is also open source with the code available on Github. Hardware platforms like NanoPi NEO / NEO Air, or Orange Pi Zero boards appear to be particularly well suited for the task of running an MQTT broker and Thingspeak, but at first I wanted to install ThingSpeak in my own Ubuntu 16.04 computer to have a try.

Click to Enlarge

Click to Enlarge

As you can see from the screenshot above I manage to do it, but it requires a bit more efforts than expected, as the project has not been updated since 2015, and does not work out of the box with the latest operating system.

I used various resources on the web including the instructions on Github, as well as this ThingSpeak script for Ubuntu 14.04, and a few other resources.

First we have to make sure Ubuntu 16.04 is fully upgraded:

Ubuntu 16.04 comes with Ruby 2.3, but we need the older Ruby 2.1.0 version for ThingSpeak, so let’s uninstall ruby to avoid conflicts:

Now we can install dependencies, Ruby 2.1.0, and Bundle:

Once this is done, we can get ThingSpeak source code and install it:

This looked successful so I moved on to database configuration:

It’s recommended to change the username and password in config/database.yml for test, development and production databases with your own for security purpose. Once it’s done, let’s try to create the databases:

Sadly it starts with an error:

So I checked mysql2 version and upgraded it to see if it would fix the issue:

The previous error is gone, but only to be replaced by a new one…

Finally, I found out (can’t find where anymore) that I had to edit Gemfile in ThingSpeak directory, and add an older version to mysql2:

Let’s update mysql2, and try to create the databases again:

Damn a permission error. I could not find a proper fix, so at this point the title of the post should possibly become “How NOT to install Thingspeak in Ubuntu 16.04”, as although it will work, the steps below makes the installation insecure since I simply give full databases’ access to thing user. But that will do since I’ll only use it in my LAN, and maybe somebody will point to a secure solution to the issue.

[Update: Thanks to Arthur, I’ve got a more secure solution . I’ve left both insecure and secure workaround for reference, but obviously you should use the secure one, especially it’s not hard]

Insecure (don’t use it, I just left it to show what you should not do):

Secure way (strongly recommended):

This time I can create the databases for Thingspeak:

So now we can go to the next step to load the database with some data required by Thingspeak to work:

Great! Yet another error:

After spending a while for a solution I eventually found it in Rails Github with the reason being that MySQL 5.7 used in Ubuntu 16.04 does ot allows for NULL key.

We’ll need to create config/initializers/abstract_mysql2_adapter.rb file with:

Then we need add the following line at the end of config/environment.db

and run the command again:

Success! Finally…

The final step is to start the server:

Now start your web browser and you can access your local Thingspeak installation @ http://localhost:3000.
I’ll now have to study a little more about Thingspeak, install MQTT, as well as one of the MQTT to Thingspeak bridges available on the web, and see if I can plot power consumption data there.

How to Build and Flash ESPurna Open Source Firmware to Sonoff POW Wireless Switch

December 3rd, 2016 36 comments

ITEAD Studio Sonoff family is comprised of various inexpensive ESP8266 WiFi power switch, and the company sent me two of their latest CE certified models with Sonoff TH16 + external temperature & humidity probe, and Sonoff POW to measure power consumption. I checked the hardware is the first part of the review, and used Sonoff TH16 to control a water pump with the stock firmware and Ewelink Android app in the second part. It works reasonably well, but it relies on the cloud, so if you lose your Internet connection or the service closed, you can’t control the relay manually anymore. Luckily, the UART pins are exposed on Sonoff switches so you can solder a 4-pin header and connect a USB to TTL to flash your own firmware.

Click to Enlarge

Click to Enlarge

Please don’t connect Sonoff devices to the mains when programming them, it’s very dangerous, instead the USB to TTL board will power the system, and will allow you to program the board safely. Later you’ll be able to update the firmware, if needed, over the network.

The next step is to select the firmware you want to use, and I’ve been advised two firmware for ESP8266, namely ESPurna specifically designed for Sonoff devices, and ESPEasy with a larger community of users. The latter may be usable to control the relay, but it has yet to support HLW8012 chip used to measure power consumption in Sonoff POW, so I decided to go with ESPurna.

That’s the description of the firmware from its bitbucket page:

ESPurna (“spark” in Catalan) is a custom C firmware for ESP8266 based smart switches. It was originally developed with the ITead Sonoff in mind. Features:

  • Asynchronous WebServer for configuration and simple relay toggle with basic authentication
  • Communication between webserver and webclient via websockets with secure ticket check
  • Flashing firmware Over-The-Air (OTA)
  • Up to 3 configurable WIFI networks, connects to the strongest signal
  • MQTT support with configurable host and topic
  • Manual switch ON/OFF with button (single click the button)
  • AP mode backup (double click the button)
  • Manual reeset the board (long click the button)
  • Visual status of the connection via the LED
  • Alexa integration (Amazon Echo or Dot) by emulating a Belkin WeMo switch
  • Support for automatic over-the-air updates through the NoFUSS Library
  • Support for DHT22 sensors
  • Support for the HLW8012 power sensor present in the Sonoff POW
  • Support for current monitoring through the EmonLiteESP Library using a non-intrusive current sensor (requires some hacking)
  • Command line configuration

I could not find firmware release for ESPurna, but Xose Pérez – the developer – has provided some basic instructions to build and flash the firmware to Sonoff. Those are not really detailed however, and it took me nearly a full day to successfully build and flash the firmware to Sonoff POW, mostly because I was not quite familiar with most of the tools used. So I’ve reproduced the step I went through in Ubuntu 16.04, and hopefully this can help people getting things done more quickly.

Let’s retrieve the source code, and enter the code directory first:

You can build the project with PlatformIO or the Arduino IDE. The instructions are for PlatformIO so that’s what I used. There are two ways to build the code with the project wither using Platform IDE for Atom and the command line, or simply using the command line. With insights, I ended up using the command line, but I’ll show both methods.

Setup PlatformIO IDE for Atom to build ESPurna

First, you’ll need to download PlatformIO IDE for Atom for your operating systems, and install it. For Ubuntu 16.04, I selected “Download .deb” for Linux and installed it through through Ubuntu Software program. Alternatively, after download, you can install it from the command line:

You can now start Atom program in Ubuntu dash, select Open Project, and browse for espurna/code directory to load the project we’ve just gotten from Bitbucket.

Click to Enlarge

Click to Enlarge

The tick button on the top left corner is to build the project, and the right button just under is to upload the firmware to the target board. But if we try to build the firmware now it will fail with an error about “espressif8266_stage”. That’s because we need to install Espressif 8266 (Stage) development platform. First we need to enable Developer mode in the IDE by going to the top menu to select PlatformIO->Settings->PlatformIO IDE, and checking “Use development version of PlatformIO“.

platformio_core_enable_development_version

Click to Enlarge

Now install PlatformIO shell commands from either a system Terminal, or PlatformIO IDE terminal (PlatformIO->Terminal->New Terminal):

and finally install ESP8266 development platform:

At this point you can click on the tick icon to build the default “node-debug” environment, a build output window will show in the IDE, and quickly disappear if the build is successful.

Setup PlatformIO via Command Line to Build ESPurna

If instead we want to use the command line we can install the latest pip version, the developer version of PlatformIO, and the staging version of ESP8266 development platform:

You can check the build environment is set properly by running the following command in espurna/code directory:

It will automatically download, build and install all dependencies and build for “node-debug” firmware for NodeMCU board. If it is successful, it will end as follows:

So I find the command line option much more easy and straightforward.

Build ESPurna for Sonoff POW

However, we are not using NodeMCU board here, but Sonoff POW, and there are two environments defined just for that:

  • sonoff-pow-debug – Build firmware to flash over serial
  • sonoff-pow-debug-ota – Build OTA firmware to upgrade the firmware over the network

The parameters for each environment are all defined in platformio.ini. First we need to build sonoff-pow-debug environment:

But it failed with an error message related to hlw8012 library:

I reported the issue on Bitbucket, but the main developer could not reproduce the issue. Eventually I found out that it could be a PlatformIO bug, as the system does not recursively checking for includes outside of main.ino. So I added <hlw8012.h> in the main.ino as follows:

and the build could complete:

Since we’ve already changed the code, you may also consider changing “#define ADMIN_PASS  fibonacci” in code/src/config/general.h to use a different default password. The password can also be changed in the web interface, but this makes sure you won’t have a device somewhere with the default password common to most users.

Flashing Firmware to Sonoff POW

Now that we’ve made sure the firmware could build, it’s time to flash it to the device.

First we need to setup some udev rules to allow flashing over serial:

Now connect the USB to TTL to a USB port on your computer, press the button (connected to GPIO0) on Sonoff POW for several seconds until both LEDs are off to make sure you are in bootloader mode, and start flashing with:

Success! Great. If you have your own firmware to flash it may be useful to know the actual command used to flash the firmware was:

Building and Flashing the ESPurna filesystem

Wait! We’ve just flashed the firmware, isn’t it all? Nope, as the webserver files are stored in another partitions, and compressed in a single index.html.gz file for better performance. The exact reasons why are further explained here.

We’ll need Node.js and gulp command line client:

Now inside espurna/code folder , we can check if building the file systems works with two commands:

Here my successful gulp attempt:

Finally, we can run the following (which also runs the two commands above) to flash the file system to the board, after entering bootloader mode by pressing the button:

Now that’s done.

A Quick look at ESPurna Web Interface

ESPurna firmware and filesystem has now been flashed to Sonoff POW. But does it work?

Click to Enlarge

Click to Enlarge

I can see a new SONOFF_POW_XXXXX access point, so that does look good. I can connect using the default password “fibonacci”, then go to my web browser to access http://192.168.4.1, and login again with admin/fibonacci credentials.

Click to Enlarge

Click to Enlarge

ESPURNA 1.03 interface goes to the status menu first, and there I can turn on and off the relay remotely, and check the power consumption in watts, which remains at 0 watt since I have not connected it to the mains yet. It’s also possible to turn the relay on and off with the button, and there an option to select whether to turn on or off the relay at boot time, which is great since I need it on at all times.

The web interface also allows you to change general parameters including the hostname and password, as well as enable or disable the HTTP API (disabled by default). The WiFi section is used to connect to up to 3 wireless routers, the MQTT section lets you configure an MQTT (Mosquito) broker, and the power section is used to calibrate the power monitoring device.

If you just intend to check the current power consumption and turn on and off the switch with your phone, you don’t have to do anything else. But I’d like to find some ways to draw daily, weekly, monthly charts of my office power consumption using either MQTT or the HTTP API. I’ll have to study how to do that, and that will hopefully be the topic of my next post about Sonoff POW.

Sonoff POW can be purchased on ITEAD Studio for $10.50 plus shipping, but is currently out of stock with the company manufacturing a third batch soon.