Archive

Posts Tagged ‘tutorial’

Getting Started with Onion Omega2+ LEDE WiFi IoT Board and Expansion Dock

January 16th, 2017 20 comments

Onion Omega2 LEDE (OpenWrt fork) WiFi board is powered by Mediatek MT7688 MIPS SoC, targets IoT projects, and sells for as low as $5. There are actually two versions: Omega2 with 64MB RAM, 16MB flash, and Omega2+ with 128MB RAM, 32MB flash and a micro SD slot. Onion sent me the latter for review, together with an expansion dock that allows powering up the board though USB , and adds a USB host port, an RGB LED, buttons, and access to GPIO via a female header. In this quick start guide, I’ll start by taking some unboxing pictures, and then report my experience following the documentation to configure the board, blink the RGB LED, and control a LED on a breadboard using a GPIO from the header.

Onion Omega2+ Unboxing

I received the two boards in their respective package, and which are both stored in anti-static bags.

Click to Enlarge

Click to Enlarge

Let’s check Onion Omega2+ board first. The top includes a chip antenna and an u.FL connector for an external antenna, as well as the main components covered by a shield with some info like FCC ID, and the MAC address with the last four digits (hexits?) in bold since they are used to access the board. The bottom of the board are two rows of headers, and a micro SD card slot. There’s also a footprint for another header or connector, but I could not find out the details.

Click to Enlarge

Click to Enlarge

Next up is the dock. We have a 2x 15-pin female header with clear marking for the pins that include power signals, GPIOs, I2C, UART, and USB.

Click to Enlarge

Click to Enlarge

The button on the top is for power, the one at 45 angle is the reset button, and we also have a micro USB port for power, a USB port for storage, an RGB LED, and the header for Onion Omega2 board.

Click to Enlarge

Click to Enlarge

Plugging Omega2 into the dock is very easy, and the only thing you have to check is that it is inserted correctly.

Onion-Omega2-vs-LinkIt-Smart-7688Onion Omega2+ is not my first Mediatek MT7688 board, as I’ve reviewed LinkIt Smart 7688 too, and took side-by-side picture of both boards for comparison. Omega2+ is smaller, but LinkIt Smart board already include a micro USB port for power.

Initial Setup for Onion Omega2 and Expansion Dock

I normally test the documentation as much as I test the board, and after a web search, I ended up on that Getting Started page. However, it was for Onion Omega, the first version of the board introduced in 2015, and while the instructions are similar, they are not quite the same. Finally, I found the actual Omega2 Wiki, and could successful complete the setup with some efforts.

I’ll be using a computer running Ubuntu 16.04 to access the board, but it also works with Windows with Bonjour Service, and Mac OS X.

The Zeroconf services is needed to play with the board unless you access the board directly with its IP, but it’s normally already installed in Linux distriutions, so we are good to go. First we need to connect a micro USB to USB cable to a power source like the USB port of your computer, and turn on the board with the power switch.

Click to Enlarge

Click to Enlarge

At first both the RGB LED on the dock and Omega2+ LED will turn on for a short time, after which the RGB LED will turn off, and Omega2 LED will blink for a few seconds, and once the LED stops blinking and remains solid the boot is done.

Omega-Onion2-Access-Point

You should then see an new “Omega-XXXX” access point in your WiFi networks, where XXXX is the last digits of your MAC address shown on bold on the board. We now need to connect to the access point using password: 12345678

Omega2 Web Configuration

One it’s done, open a web browser and go ti http://192.168.3.1 or http://omega-XXXX.local/ to access Omega2 Setup Wizard.

Omega-Onion2-Setup-Wizard

Click Start to login with the default credentials (username: root ; password:onioneer), and the next page will let you connect the board to your WiFi router.Omega-Onion2-WiFi-Configuration

Selection you ESSID, input you WiFi password. and clikc on “Configure WiFi“. Omega-Onion2-Cloud

The wizard offers you to register your board to the cloud, but this is completely optional, and you could simply select Skip Step to go to the next step (firmware update). But I tried to registered the device to the cloud for this review.Onion-Cloud-RegistrationYou’ll need to provide your name, an email address and a password to register an account first.Onion-Omega2-Cloud-NameYou’ll then be asked for a device name and a description to confirm the registration.Onion-Cloud-Connection-failedSadly this step failed and I got the window above. Clicking on the red cross button did nothing. If I login to the cloud service, I can see the board listed, but detected as offline. I’m not the only one to have this issue, and Onion developers are looking into it.

Onion-Omega2+-Firmware-Update-ConsoleSo instead I went to the next step to upgrade the firmware and install Console web-based virtual desktop.Onion-Omega2-Firmware-Download-StuckThis also failed as the progress bar did not move at all, and I waited for around 20 minutes. I could also see my router’s DHCP server gave an IP address to the board, so it should have been able to connect to the Internet.

Omega2 Command Line Configuration

So I used to backup configuration method, using the command line as explained in the documentation. You just need to SSH the board as root with the same password as in the web configuration (onioneer):

Note Ω-ware firmware version is 0.1.5 b130.

wifisetup allows you to scan the network, and connect the board to your router:

Good. Firmware update failed in the web setup wizard, but we can retry it with oupgrade command line:

The firmware could be downloaded, and it looked like the system rebooted as I lost access to SSH terminal. The LED was still on for a while after it happened, then the LED went off (forever), at least longer than the 15 seconds listed in the documentation, and in that case they explain you need to power cycle the board. I used the power switch on the expansion dock to do so.

The board LED blinked for a pretty long time (maybe 2 minutes), but eventually it stopped and remained solid, and I could login to the board:

The firmware was updated to version 0.1.7 b139, so all good even though the whole setup did not go 100% smoothly. In case something goes really wrong and you can’t access the board at all, you could try to do a Factory Restore by pressing and holding the reset button for 10 seconds then releasing it.

Omega2 LEDE System Info

Since we’re done with the configuration, let’s quickly check some system info:

So we have a relatively recent kernel (Linux 4.4), 24.4MB space available to the user, 125664 KB total memory, and a MIPS 24KEc processor…

Controlling Omega2’s Dock RGB LED (via PWM)

We can start playing with the GPIO on the board starting with the RGB LED on the dock  that should be connected to pin 15, 16 and 17. The documentation explains expled sample can be used for this and we can see the R, G, B hexadecimal values. I want to show red color only, and I set blue and green to zero:

Oops, segfault. Let’s try something else like a pinkish color:

It runs, but the RGB LED remains off. It’s not an hardware problem since the RGB LED turns on at boot time. expled is actually a bash script that can be found in /usr/bin/expled and calls “fast-gpio“program which access GPIOs directly without using sysfs. Maybe it’s another firmware issue.

Controlling Omega2 GPIOs – LED example

In order to play with the expansion header, I connected a 5V LED to a breadboard together with two resistors and a transistor (for 3.3 to 5V conversion), and connected it to pin 1 on the header.

Click to Enlarge

We’ve already seen fast-gpio tools in expled script, but I used another GPIO tools for the LED, namely gpioctl that relies on sysfs.

We first need to set the GPIO pin as an output pin using the dirout command (dirin would set it as an input):

We can now turn the LED on by setting GPIO 1 to HIGH with dirout-high option:

The get command above will check the value of the pin. The LED did turn on as it should, and we can turn it off with dirout-low option:

Success.

If you want to use multiplexed pin with I2C, SPI, UART, PWM, I2S… you’ll need to check out omega2-ctrl program. I have not tried it for this guide to keep it short.

Onion Omega2 and BreadBoards

Many similar small IoT board are designed to fit on a breadboard, but Onion Omega2 board’s header pins are using 2 mm pitch, not 2.5 mm pitch, so they can’t be used with a breadboard directly. Instead, you’d have run wires from the board to the breadboard or purchase a BreadBoard Dock as pictured below.

If you do not have a dock, or breadboard expansion board, you can still power the Omega2 module/board using a 3.3V power source for example with a regulator such as LD1117, or something like YwRobot MB102 breadboard power supply.

If you are interested in getting a board, you may have to wait as while Omega Expansion Dock sells for $14.99 on Onion store, Omega2 boards are not listed yet. For reference, Omega2 board went for $5, and Omega2+ board for $9 on Kickstarter.

Self-hosted OpenGL ES Development on Ubuntu Touch

January 15th, 2017 4 comments

Blu wrote BQ Aquaris M10 Ubuntu Edition review – from a developer’s perspective – last year, and now is back with a new post explaining how to develop and deploy OpenGL ES applications directly on the Ubuntu Touch tablet.

Ever since I started using a BQ M10 for console apps development on the go I’ve been wanting to get something, well, flashier going on that tablet. Since I’m a graphics developer by trade and by heart, GLES was the next step on the Ubuntu Touch for me. This article is about writing, building and deploying GLES code on Ubuntu Touch itself, sans a desktop PC. Keep that in mind if some procedure seems unrefined or straight primitive to you – for one, I’m a primitive person, but some tools available on the desktop are, in my opinion, impractical on the Touch itself. That means no QtCreator today, nor Qt, for that matter.

The display of any contemporary Ubuntu Touch device is powered by Mir – a modern compositor/surface manager taking care of all (rectangular-ish) things eventually appearing on screen. We won’t be delving much into Mir beyond obtaining an EGL context (EGL being the binding layer between GLES and the native windowing system). But enough ado – let’s get to work.

Preparations for doing GLES on a Ubuntu Touch box:

The above, as of the time of this writing, should provide you with gcc/g++-4.9, make and gdb-7.9, among other things. The last package and its dependencies provide you with up-to-date Mir headers. Git comes out of the box, IIRC, but if it’s missing just apt-get it.

We need a primer to step on, so here’s my adaptation of Don Bright’s Mir/GLES adaptation of Joe Groff’s OpenGL tutorials, using Daniel van Vugt’s Mir/EGL examples (yes, that’s a quite a chain-work):

I’ve taken the liberty to expand on the work of those gentlemen by bringing the Mir integration up to date, handling Touch’s novelty Desktop Mode and throwing in my own dusty GLES sample code, for good measure.

To build and install the primer, just do:

That will provide you with an original police-car flashing-lights primer. An alternative primer featuring tangential-space bump-mapping can be built by passing arg ‘guest’ to the build script:

Both versions of the primer use a fundamentally identical interface — a resource-initialization procedure and a frame-drawing procedure, so it’s not much of an effort to use the respective routines from either primers in the framework of the host app hello.cpp, and thus get a running render loop.

A few words about the peculiarities of the GLES development for Ubuntu Touch. It took me some time to show anything on screen, despite the fact I had a valid draw context and a render loop soon after the primer was building successfully. The reason is Unity8 on the Touch will not simply let you run a window-painting app from the terminal – you would get your Mir and EGL contexts alright, but the target surface will never be composited to the screen of the device upon eglSwapBuffers() unless you take certain actions. You have two alternatives here:

  • Produce a valid Click package from your app and subsequently install that to the Apps pane (what our build script does), where you can launch from an icon, or…
  • Use a launcher app to start your window app (info courtesy of Daniel van Vugt):

Unfortunately the second (much quicker and convenient) approach is not currently usable due to a bug, so we’ll stick with the first. Any command-line args we’d want to pass to the app will need to be written to the app’s .desktop file, which can be found at the official app location after installation:

In that file, set the desired args on the ‘Exec’ line, like this:

Another peculiarity was that in Desktop Mode the app window does a classical ‘zoom to full size’ animation at start. Nothing extraordinary in that, if not for the fact that the Mir surface itself resizes along with the window. Now, a default viewport in a GLES context spans the geometry of the target surface at the time of its creation, which, in our case, is the start of the window-zoom animation, with its tiny surface geometry. One needs to wait for the zoom animation to finish, and then set the viewport geometry to the final geometry of the Mir surface, or live with a post-stamp-sized output in the lower left corner of the window, if the viewport is left unchanged.

Once we get past those teething hurdles we actually get quite a nicely behaving full-screen app on our hands – it composites smoothly with all other Ubuntu Touch desktop elements like the Launcher tab at the desktop’s left edge and the pull-down Indicator pane on right (see screenshot). Our app even does live output to the Scopes selector screen (i.e. the task-switching screen) — behold the miracles of modern-day screen compositors! ; )

Click for Original Size (1920×1080)

But hey, don’t just take my word for it – try out GLES coding on a Ubuntu Touch device – you have the basics covered:

  • App’s rendering loop and the entirety of the flashing-screen primer are in hello.cpp
  • Mir context creation and subsequent EGL context binding are in eglapp.cpp
  • Bump-mapping primer is entirely in app_sphere.cpp
  • Various helpers are spread across util_* TUs and hello.cpp
  • All files necessary for the generation of the Click package are in resource folder.

In conclusion, self-sustained development on the Ubuntu Touch is a perfectly viable scenario (take that, iOS!). Moreover, the GPU in the BQ M10 turned out to have a very nice modern GLES3 (3.1) stack – see excerpts from the app logs below. Actually, this is my first portable device with a GLES 3.1 stack, so I haven’t started using it properly yet — the GLES2 primer above doesn’t make use of the new functionality.

If I have to complain about something from the development of this primer, it’d be that I couldn’t use my arm64 code on the primer, since there are only armhf (32-bit) EGL/GLES libraries available for the Touch. So 64-bit code on the Ubuntu Touch remains in console land for now.

Excerpts from the primer logs:

egl version, vendor, extensions:

1.4 Android META-EGL
Android
EGL_KHR_get_all_proc_addresses EGL_ANDROID_presentation_time EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_fence_sync EGL_KHR_create_context EGL_ANDROID_image_native_buffer EGL_KHR_wait_sync EGL_ANDROID_recordable EGL_HYBRIS_native_buffer2 EGL_HYBRIS_WL_acquire_native_buffer EGL_WL_bind_wayland_display

gl version, vendor, renderer, glsl version, extensions:

OpenGL ES 2.0 (OpenGL ES 3.1)
ARM
Mali-T720
OpenGL ES GLSL ES 3.10
GL_EXT_debug_marker GL_ARM_rgba8 GL_ARM_mali_shader_binary GL_OES_depth24 GL_OES_depth_texture GL_OES_depth_texture_cube_map GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_EXT_read_format_bgra GL_OES_compressed_paletted_texture GL_OES_compressed_ETC1_RGB8_texture GL_OES_standard_derivatives GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_sync GL_OES_texture_npot GL_OES_vertex_half_float GL_OES_required_internalformat GL_OES_vertex_array_object GL_OES_mapbuffer GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_type_2_10_10_10_REV GL_OES_fbo_render_mipmap GL_OES_element_index_uint GL_EXT_shadow_samplers GL_OES_texture_compression_astc GL_KHR_texture_compression_astc_ldr GL_KHR_texture_compression_astc_hdr GL_KHR_debug GL_EXT_occlusion_query_boolean GL_EXT_disjoint_timer_query GL_EXT_blend_minmax GL_EXT_discard_framebuffer GL_OES_get_program_binary GL_OES_texture_3D GL_EXT_texture_storage GL_EXT_multisampled_render_to_texture GL_OES_surfaceless_context GL_OES_texture_stencil8 GL_EXT_shader_pixel_local_storage GL_ARM_shader_framebuffer_fetch GL_ARM_shader_framebuffer_fetch_depth_stencil GL_ARM_mali_program_binary GL_EXT_sRGB GL_EXT_sRGB_write_control GL_EXT_texture_sRGB_decode GL_KHR_blend_equation_advanced GL_OES_texture_storage_multisample_2d_array GL_OES_shader_image_atomic

GL_MAX_TEXTURE_SIZE: 8192
GL_MAX_CUBE_MAP_TEXTURE_SIZE: 4096
GL_MAX_VIEWPORT_DIMS: 8192, 8192
GL_MAX_RENDERBUFFER_SIZE: 8192
GL_MAX_VERTEX_ATTRIBS: 16
GL_MAX_VERTEX_UNIFORM_VECTORS: 1024
GL_MAX_VARYING_VECTORS: 15
GL_MAX_FRAGMENT_UNIFORM_VECTORS: 1024
GL_MAX_COMBINED_TEXTURE_IMAGE_UNITS: 48
GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS: 16
GL_MAX_TEXTURE_IMAGE_UNITS: 16

vThings WiFi CO2 Monitor Quick Start Guide

December 28th, 2016 5 comments

I’ve already checked out vThings CO2 Monitor hardware and we’ve seen it’s based on ESPrino ESP8266 board, and my model includes CM1106 CO2 sensor and BMP180 temperature and pressure sensor. I’ve now installed it in my kitchen, about 3 to 4 meters from the gas stove, and getting data to ThingSpeak.

vair-monitor-co2-sensor

The door and window of my kitchen are open all day, and the wall have ventilation holes. That’s important for CM1106 sensor since it auto calibrates every 3 days in clear air. If you plan to use such sensor in a closed environment, you should buy Vthings with CM1102 CO2 sensor that costs more, but does not require calibration.

Since all WiFi systems I’ve just so far starting AP mode for configuration, I first looked for an access point, but… nothing… Then I decided to read the documentation (might be useful at times), and the monitor is actually configured via a Chrome (desktop only) add-on through USB. There are three types of devices made by vair-monitor, and I first used  vThings Configuration Utility add-on, but eventually found out I had to use vThings – Dual Beam Configuration Utility.

vthings-chrome-apps

vThings Configuration vs Things – Dual Beam Configuration Utility

I used Ubuntu (Linux), but if you are using a Windows or Mac computer, you’ll need to install drivers first. Once you’ve connected the monitor through USB and started “vThings- Device Configuration Tool” the following windows should be shown.

Click to Enlarge

Click to Enlarge

The fist thing to do is to connect the monitor to your WiFi router by entering its SSID and password, and click on Set WiFi.

Click to Enlarge

Click to Enlarge

It should connect to your router, and the first time updated the firmware automatically. Wait a couple of minutes for it to complete, and you can go to the next step to configure one or more of the following Public, Private or Generic services:

Public Private Generic
BeeBottle DomoticGa HTTP
Blynk.cc DomoticZ MQTT
dweet.io FHEM RF 433/315
EmonCMS Homeseer
ThingSpeak HomeAssistant
UbiDots JeeDom
OpenHAB
Pimatic

I decided to go with ThingSpeak since I got familiar with it while writing Sonoff POW tutorial.

Click to Enlarge

Click to Enlarge

Select the data provided by the sensors inside your vthings Co2 monitor, in my case CO2 levels, temperature, and pressure, and nothing else, or connection will fail, as I found out when I used 4 default fields including humidity, and ThingSpeak was not updated at all. You’ll also need ThingSpeak API write key, that you can get my create a channel on ThingSpeak.com as shown below.

thingspeak-co2-monitor-thingspeak-channel-configuration

Once the channels is create on ThingSpeak website, and you’ve added the API write key in vThings Device Configuration Tool, you could go to Generic Services->HTTP and notice an HTTP request has been created, so if you have installed ThingSpeak locally, you could change api.thingspeak.com to your own IP address.

Click to Enlarge

Click to Enlarge

By default the data will be updated every 120 seconds, but you can change that in Settings->Update Interval. Once configuration is done, you can unplug it from your PC, and connected to the location you want to monitor. vThings Device Configuration Tool requires a USB connection to find the device, it can not find it over WiFi, so if you want to change configuration, you’ll need to connect it back to your computer. There’s a function to (auto)start a webserver in vESPrino, but it did not seem to work for me.

After a few hours or minutes depending on your update internal you should get some nice charts on ThingSpeak with CO2 levels, temperature and pressure, or other data based on the sensors you’ve selected while purchasing the hardware.

Click to Enlarge

Click to Enlarge

The channel is public if you are interested/curious in seeing the data. ThingSpeak will show 60 samples (2 hours in my case) by default, but let’s see what happened over the last 12 hours with CO2 levels.

Click to Enlarge

Click to Enlarge

The CO2 levels started at about 500 to 600 ppm while I did the configuration in my office (windows closed), and dropped to around 404 ppm once I installed in the kitchen. That value correspond roughly to the current CO2 ppm value in the atmosphere (in Hawaii). Three times around 18h00 people warmed food and CO2 jumped to around 500 ppm. During the night, CO2 levels slowly increased to 480 ppm, likely because of the plants cycle (producing oxygen during the day, and carbon dioxide during the night). This morning CO2 levels spiked at around 900 ppm when cooking right after 6am and 8am.

That’s all fun, but is there a real benefit to measuring CO2 levels in your house? In the kitchen I could probably trigger an alert over 1,500 ppm in which case it may mean something is burning, but smoke detectors are much cheaper and better suited to the task. Vladimir Savchenko, vThings developer, found a study claiming that high CO2 levels may decrease creative thinking and lead to bad sleep, so he used vThings CO2 monitor in his bedroom and discovered CO2 levels reached close to 4,000 ppm, and that just open the door or window would greatly reduce the concentration of the gas.

sleepwithcloseddoortext-co2-levelsvThings CO2 monitor does not only monitor CO2 levels as we’ve seen above, as temperature, humidity, and/or pressure sensor can be included in the case, as well as a PM2.5 & PM10 laser dust sensor.

vThings CO2 Monitor v3 is sold for 60 Euros with CM1106 CO2 sensor, and more if you use a better CO2 sensor, or add extra environmental sensors. 135 Euros would get you a top of line monitor with a laser dust sensor, CDM7160 CO2 sensor, temperature and humidity sensor, and RF connectivity.

Something is “Eating” my Android TV Box Internal Storage!

December 21st, 2016 6 comments

No, it’s not a joke. I’ve been playing for a while with Eweat R9 Plus Android TV box after inserting a 1TB hard drive in the SATA bay, but while in most other reviews, the apps and files used for testing are just taking around 3 GB, the 9.31 GB storage in that device was completely full, and I have yet to install some apps, and copy some files part of my testing procedure…

android-storage-fullI did not immediately find out about the “storage full” issue, as I first I just noticed Kodi would not start anywhere, and the default video app would just reboot after adjusting the volume… But let’s check what takes all that space…

Click to Enlarge

Click to Enlarge

I had to uninstall 3Dmark in order to be able to take screenshots, which explains why I have a bit more space now, but the system reports all that space (and more???) is taken by apps. I clicked on Apps, to find out Es File Explorer was the worse offender but with only “285 MB” used. That did not add up, so I started to adb shell to try to find the exact files:

Media directory takes 3.1GB and data directory 4.5GB. After some more checking, I eventually the space mostly is taken by two large “external.db” databases for com.android.providers.media:

Reading from Android’s developer website:

Android Provider provides convenience classes to access the content providers supplied by Android.

Android ships with a number of content providers that store common data such as contact informations, calendar information, and media files. These classes provide simplified methods of adding or retrieving data from these content providers.

I have a lot of files (millions) on my SATA hard drive, many of them not media files, but the box is probably scanning the files and storing some metadata for faster access or/and search within Android. Earlier this year, I had a problem with Media Scanner in Zidoo X1 II TV box, as it slowed down the system greatly while scanning a USB hard drive and affected video playback. I solved the issue with media.Re.Scan: app which allowed me to stop the scanning process at the time. We don’t really need the app however, and since we don’t have space, you’d have to uninstall other apps to install the utility. Instead go to Settings->Apps, select “Show System” on the top right, and scroll down until you find “Media Storage”.

Click to Enlarge

Click to Enlarge

Now click on Media Storage, select Disable, and click on “Storage” in order to finally select “Clear Data” to free up (a lot of) space.

android-media-storage-clear-data

You can now have fun with your TV box with plenty of storage, and a lot less (apparent) bugs…android-storage-usbHappy to have gotten rid of Media Storage (aka The grinch) right before Christmas :).

Facebook Zstandard “zstd” & “pzstd” Data Compression Tools Deliver High Performance & Efficiency

December 19th, 2016 11 comments

Ubuntu 16.04 and – I assume – other recent operating systems are still using single-thread version of file & data compression utilities such as bzip2 or gzip by default, but I’ve recently learned that compatible multi-threaded compression tools such as lbzip2, pigz or pixz have been around for a while, and you can replace the default tools by them for much faster compression and decompression on multi-core systems. This post led to further discussion about Facebook’s Zstandard 1.0 promising both smaller and faster data compression speed. The implementation is open source, released under a BSD license, and offers both zstd single threaded tool, and pzstd multi-threaded tool. So we all started to do own little tests and were impressed by the results. Some concerns were raised about patents, and development is still work-in-progess with a few bugs here and there including pzstd segfaulting on ARM.

Zstd vs Zlib Compression Ratio vs Speed

Zstd vs Zlib Compression Ratio vs Speed

Zlib has 9 levels of compression, while Zstd has 19, so Facebook has tested all compression levels and their speed, and drawn the chart above comparing compression speed to compression ratio for all test points, and Zstd is clearly superior to zlib here.

They’ve also compared compression and decompression performance and aspect ratio for various other competing fast algorithms using lzbench to perform this from memory to prevent I/O bottleneck from storage devices.

Name Ratio C.speed D.speed
MB/s MB/s
zstd 1.0.0 -1 2.877 330 940
zlib 1.2.8 -1 2.730 95 360
brotli 0.4 -0 2.708 320 375
QuickLZ 1.5 2.237 510 605
LZO 2.09 2.106 610 870
LZ4 r131 2.101 620 3100
Snappy 1.1.3 2.091 480 1600
LZF 3.6 2.077 375 790

Again everything is a comprise, but Zstd is faster than algorithms with similar compression ratio, and has a higher compression ratio than faster algorithm.

But let’s not just trust Facebook, and instead try ourselves. The latest release is version 1.1.2, so that’s what I tried in my Ubuntu 16.04 machine:

This will install the latest stable release of zstd to your system, but the multi-thread is not build by default:

There are quite a lot of options for zstd:

Since we are going to compare results to other, I’ll also flush the file cache before each compression and decompression using:

I’ll use the default settings to compress Linux mainline directory stored in a hard drive with tar + zstd (single thread):

and pzstd (multiple threads):

Bear in mind that some time is lost due to I/O on the hard drive, but I wanted to test a real use case here, and if you want to specifically compare the raw performance of compressor you should use lzbench. Now let’s decompress the Zstandard tarballs:

My machine is based on an AMD FX8350 octa-core processor, and we can clearly see that by comparing real and user time, the test is mostly I/O bound. I’ve repeated those test with other multi-threaded tools as shown in the summary table below.

Compression Decompression File Size (bytes) Compression Ratio
Tools Time (s) “User” Time (s) Time (s) “User” Time (s)
ztsd 130.056 91.608 45.124 21.26 1,881,020,744 1.48
pzstd 58.929 86.56 38.175 23.39 1,883,697,296 1.48
lbzip2 84.216 353.84 37.109 167.416 1,855,837,345 1.50
pigz 61.121 121.332 34.36 15.26 1,903,915,372 1.47
pixz 177.596 1233.88 36.24 78.116 1,782,756,524 1.57
pzstd -19 275.361 1939.536 26.85 21.832 1,794,035,552 1.56

I’ve included both “real time” and “user time”, as the latter shows how much CPU time the task has spent on all the cores of the system. If user time is large that means the task required lots of CPU power, and if a task completes in about the same amount of “real time”, but a lower “user time”, it means it was likely more efficient, and consumes less power. pigz is the multi-threaded version of xz algorithm relying on lzma compression which delivers a high compression ratio, at the expense of longer compression time, so I also run pzstd with level 19 compression to compare:

Zstandard compression ratio is similar to the one of lbzip2 with default settings, but compression is quite faster, and much more power efficient. Compared to gzip, (p)zstd offers a better compression ratio, against with default settings, and somewhat comparable performance. pixz offers the best compression ratio, but takes a lot more time to compress, and uses more resources to decompress compared to Zstandard and Pigz. Pzstd with compression level 19 takes even more time to compress, and is getting close to pixz compression, but has the advantage of being much faster to decompress.

Compress & Decompress Files Faster with lbzip2 multi-threaded version of bzip2

December 16th, 2016 30 comments

Bzip2 is still one of the most commonly used compression tools in Linux, but it only works with a single thread, and I’ve been made aware that lbzip2 allows multi-threaded bzip2 compressions which should lead to much better performance on multi-core systems.

Tar with lbzip2 on a 8-core Processor - Click to Enlarge

Tar with lbzip2 on an 8-core Processor – Click to Enlarge

lbzip2 was not installed by default in my Ubuntu 16.04 machine, but it’s easy enough to install:

I have cloned mainline linux repository on my machine, so let’s see how long it takes to compress the directory with bzip2 (one core compression):

9 minutes and 22 seconds. Now let’s repeat the test with lbzip2 using all 8 cores from my AMD FX8350 processor:

2 minutes 32 seconds. Almost 4x times, not bad at all. It’s not 8 times faster because you have to take into account I/Os, and at the beginning the system is scanning the drive, using all 8-core but not all full throttle. The files were also stored in a hard drive, so I’d assume the performance difference should be even more noticeable from an SSD.

We can see both files are about the same size as they should be:

I’m not exactly sure why there’s about 771 KB difference as both tools offer the same compression.

That was for compression. What about decompression? I’ll decompress the lbzip2 compressed file with bzip2 first:

2 minutes and 49 seconds. Now let’s decompress the bzip2 compressed file with lbzip2:

45 seconds! Again the performance difference is massive.

If you want tar to always use lbzip2 instead of bzip2, you could create an alias:

Please note that this will cause a conflict (“Conflicting compression options”) when you try to compress files using -j /–bzip2 or -J, –xz options, so instead of tar, you may want to create another alias, for example tarfast.

lbzip2 is not the only tool to support multi-threaded bzip2 compression, as pbzip2 is another implementation. However, one report indicates that lbzip2 may be twice as fast as pbzip2 to compress files (decompression speed is about the same), which may be significant if you have a backup script…

tkaiser also tested various compression algorithms (gzip, pbzip2, lz4, pigz) for a backup script for Orange Pi boards running armbian, and measured overall performance piping his eMMC through the different compressors to /dev/null:

pigz looks the best solution here (25.2 MB/s) compared to pbzip2 (15.2 MB/s). lbzip2 has not been tested, and could offer an improvement over pigz both in terms of speed and compression based on the previous report, albeit actual results may vary depending on the CPU used.

How to Use Sonoff POW ESP8266 WiFi Power Switch with MQTT and ThingSpeak

December 11th, 2016 10 comments

ITEAD Studio’s Sonoff is a family of cheap home automation products based on ESP8266 WiSoC, and I’ve already tested Sonoff TH16 wireless switch with a humidity and temperature sensor using the stock firmware and eWelink app for Android or iOS. It works, but up to recently it required a registration to a cloud service (the company will now allow use from the local network), and the source code is closed. So for the second device under review, namely Sonoff POW wireless switch with a power consumption monitor, I decided to install ESPurna firmware working on ESP8266 Sonoff devices and NodeMCU, as it’s open source, supports Sonoff POW natively, includes a web interface to control the device from the LAN, and includes an MQTT client.

MQTT (Message Queuing Telemetry Transport) is a lightweight publish/subscribe messaging protocol used to control IoT sensors and devices, and it’s a popular method to gather data from client to a MQTT broker to push the data to the cloud or a local database.

iot-sensors-mqtt-cloud

So typically, you’d have a bunch of sensor nodes (like Sonoff devices) communicating over MQTT to an MQTT Broker in your local network, which could be an OpenWrt router or a Linux development board like a Raspberry Pi, which in turns gets the data the the cloud to services like AWS IoT, Xively, or ThingSpeak. It’s also possible to use Cloud services to control MQTT devices remotely through the MQTT broker.

I eventually plan to use NanoPi NEO board to run both MQTT and ThingSpeak locally (not connected to the cloud) in order to monitor the power consumption of my small office, but since I’m all new to this, I’ve started experimenting by connecting a 30W light to Sonoff POW, and use a desktop computer running Ubuntu 16.04 for MQTT and ThingSpeak.

sonoff-pow-connection

Click to Enlarge

Click to Enlarge

Since I’ve already installed ESPurna firmware to the device, I disconnected the USB to serial board (important since Sonoff POW board has a hot ground), and connected it to the mains (220V in my location). That means we already have an MQTT client which first I had to configure.

Click to Enlarge

Click to Enlarge

Since it was the first time I connected a load to the device, I went to ESPurna’s status menu to check power usage was reported, and my 30 Watts light bulb was drawing 27 Watts. Close enough. I changed the hostname to sonoff-office, and setup two SSID in order to connect Sonoff POW to my local network in client mode, instead of using it in Access Point mode by default. You’ll need to tap on Update each time you modify the settings. Since the SSID must be entered manually, please note that SSID are case sensitive, e.g. CNX-SOFTWARE is different from cnx-software.

Click to Enlarge

Click to Enlarge

I wanted to calibrate the power using the 30W light bulb, so I entered 30W in AC RMS Active Power field, and tapped on Update, but the web interface reported “no changes”. I’m not sure how to use that part. Finally the most important part for this tutorial is to set the MQTT settings with MQTT IP address, and leaving other fields unchanged. However, you can change MQTT Topic field for example replacing /test/switch/{identifier} by /myiotstuff/{identifier}.

Now that our MQTT client is configured, I need to install mosquitto MQTT broker in Ubuntu:

mosquitto-clients is not really needed, but I’ll use it to test the MQTT broker a little later. Once you installed it, the MQTT Broker should already run automatically.

The last line of the log above shows a client connection from Sonoff POW. Now, we need to check the topic, and since ESPurna documentation is still work in progress, you could either check out the source code, or IMHO more fun, capture MQTT packet with tcpdump or Wireshark as I’ve done below.

Wireshark MQTT Capture - Click to Enlarge

Wireshark MQTT Capture – Click to Enlarge

Here we can see that Sonoff POW will send a Publich Message with the power level using the topic “/test/switch/sonoff-office/power29”.  “/test/switch” is the string we’ve defined in the web interface, “sonoff-office” the hostname we’ve given to Sonoff Pow, and “power29” indicates 29 Watts of power is currently used.

We can also start a client in Ubuntu 16.04 terminal window to check more MQTT topics with # wildcard for sonoff-office host:

We can use MQTT to get the IP address, firmware and file system version, hearbeat message, power use, and relay status (on or off).

It’s all good, but now we need to do something to draw the data, and possibly analyze it. I selected ThingSpeak for this purpose since it can be installed in the local network, or through their service in the cloud. By the end of my testing, I’ve noticed ThingSpeak has a new MQTT API, meaning it should be possible to connect your MQTT broker directly to it, but for this guide I use mqspeak instead as a bridge between MQTT and ThingSpeak. It may still be useful, as the open source version of ThingSpeak is not updated anymore, and lacks the MQTT API.

You’ll need Python 3 and pip3 to install mqspeak:

Once it’s done, we’ll need to create a config files as explained on mqspeak’s github repo, and I created /etc/mqspeak.conf with the following content:

Brokers are used to configure MQTT broker IP address and port, as well as the topic(s) to subscribe to, while Channels take care of ThingSpeak configuration with the channel’s Id and write API key, update rate in seconds (15s minimum), update type (see github for details), and fields defined in your ThingSpeak’s channel(s), which will create later on. I wrote one broker for the power consumption topic, and other for the relay status. However, I eventually ignored the relay status, as it’s not updated often enough and cause ThingSpeak’s channel to only be updated when the relay changes status, even if there are power updates in the meantime. A workaround is to use two different channels for ThingSpeak.

mqspeak connects directly to api.thingspeak.com, so if you are using ThingSpeak cloud services, the next step is to register an account and setup one or more channels.

Extra Instructions for a local installation of ThingSpeak

However, if you’ve installed ThingSpeak in Ubuntu 16.04 or other Linux operating systems locally or on your own server, you’ll need to change the server in the source code, and reinstall mqspeak.

  1. Get the source code:
  2. Modify mqspeak/sending.py to replace api.thingspeak.com using HTTPS with localhost (or other IP address/URL where you’ve installed ThingSpeak) with HTTP:
  3. Install mqspeak

An improvement would be to install a signed SSL certificate, like the one offered by LetsEncrypt and configure the rails server to use https instead. I have not setup ThingSpeak server to start automatically yet, so I have to start it manually for now:

End of instructions specific to local installation.

The instructions specific for the local installation of ThingSpeak are now done, and all instructions below are valid for both the local installation and cloud service. Now open a web browser, go ThingSpeak (cloud or local), and click on “Get Started Now” in order to register an account.

Click to Enlarge

Click to Enlarge

Once it is done, login and click on “New Channel”.

Click to Enlarge

Click to Enlarge

Give it a name, a description, create fields as needed, for example power-consumption and power-status, and click on Save Channel.  Update /etc/mqspeak.conf accordingly with the fields’ name, and channel Id.

thingspeak-api-keyNow select API Keys tab to copy and paste the write API key into mqspeak.conf.

Now we can start mqspeak:

ESPurna firmware will send a power update every 60 seconds (this can be changed in code/src/pow.ino), so you’ll need a new message pop-up every 60 seconds with your channels Id and write API key. I’ve let it run for about one hour, and got the follow chart in ThingSpeak after turning on and off the lights from time to time.
thingspeak-power-consumptionThat’s pretty cool, so it only shows the current power in watt, and we’d probably want to get power consumption in kW/h per day, week and month at some time, and I have yet to study how to do that, Exporting the data to excel would be a workaround if this can not be handled in ThingSpeak. ThingSpeak.com (but not the open source version) offers some Matlab processing of the data, so that’d be another options.

The next steps would be to install MQTT and ThingSpeak in NanoPi NEO board, enable HTTPS in ThingSpeak, autostart rails server and mqspeak at boot time, make ESPurna firmware publish the “Power” topic more often than every 6 second, and find some way to generate useful kW/h consumption charts from the data stored in ThingSpeak within ThingSpeak, or but exporting the data.

How to Install ThingSpeak in Ubuntu 16.04

December 7th, 2016 11 comments

Last week-end I installed ESPurna open source firmware with MQTT server on Sonoff POW WiFi switch, and the next step is find a way to draw power consumption charts in some web based interface. We could do this in the IoT cloud with services like Xively or ThingSpeak, but since one of the goals of replacing the default firmware was not to rely on a proprietary cloud based solution, I decided to find a way to draw those chart in a local server, and it so happens that ThingSpeak is also open source with the code available on Github. Hardware platforms like NanoPi NEO / NEO Air, or Orange Pi Zero boards appear to be particularly well suited for the task of running an MQTT broker and Thingspeak, but at first I wanted to install ThingSpeak in my own Ubuntu 16.04 computer to have a try.

Click to Enlarge

Click to Enlarge

As you can see from the screenshot above I manage to do it, but it requires a bit more efforts than expected, as the project has not been updated since 2015, and does not work out of the box with the latest operating system.

I used various resources on the web including the instructions on Github, as well as this ThingSpeak script for Ubuntu 14.04, and a few other resources.

First we have to make sure Ubuntu 16.04 is fully upgraded:

Ubuntu 16.04 comes with Ruby 2.3, but we need the older Ruby 2.1.0 version for ThingSpeak, so let’s uninstall ruby to avoid conflicts:

Now we can install dependencies, Ruby 2.1.0, and Bundle:

Once this is done, we can get ThingSpeak source code and install it:

This looked successful so I moved on to database configuration:

It’s recommended to change the username and password in config/database.yml for test, development and production databases with your own for security purpose. Once it’s done, let’s try to create the databases:

Sadly it starts with an error:

So I checked mysql2 version and upgraded it to see if it would fix the issue:

The previous error is gone, but only to be replaced by a new one…

Finally, I found out (can’t find where anymore) that I had to edit Gemfile in ThingSpeak directory, and add an older version to mysql2:

Let’s update mysql2, and try to create the databases again:

Damn a permission error. I could not find a proper fix, so at this point the title of the post should possibly become “How NOT to install Thingspeak in Ubuntu 16.04”, as although it will work, the steps below makes the installation insecure since I simply give full databases’ access to thing user. But that will do since I’ll only use it in my LAN, and maybe somebody will point to a secure solution to the issue.

[Update: Thanks to Arthur, I’ve got a more secure solution . I’ve left both insecure and secure workaround for reference, but obviously you should use the secure one, especially it’s not hard]

Insecure (don’t use it, I just left it to show what you should not do):

Secure way (strongly recommended):

This time I can create the databases for Thingspeak:

So now we can go to the next step to load the database with some data required by Thingspeak to work:

Great! Yet another error:

After spending a while for a solution I eventually found it in Rails Github with the reason being that MySQL 5.7 used in Ubuntu 16.04 does ot allows for NULL key.

We’ll need to create config/initializers/abstract_mysql2_adapter.rb file with:

Then we need add the following line at the end of config/environment.db

and run the command again:

Success! Finally…

The final step is to start the server:

Now start your web browser and you can access your local Thingspeak installation @ http://localhost:3000.
I’ll now have to study a little more about Thingspeak, install MQTT, as well as one of the MQTT to Thingspeak bridges available on the web, and see if I can plot power consumption data there.