Heatshrink – An ultra-lightweight compression library for embedded systems

When I wrote about Bangle.js 2 JavaScript smartwatch yesterday, I noticed they used “Heatshrink compression” in ESPruino firmware.  I can’t remember ever reading about Heatshrink before, and indeed there are no results while searching on CNX Software.

Heatshrink is an open-source data compression library designed for resources-constrained embedded systems that works with as little as 50 bytes of RAM. That’s impressive, so let’s investigate.

Heatshrink

The library is written in C language and was released about 8 years ago on Github with the following key features:

  • Low memory usage – As low as 50 bytes with specific parameters, and usually under 300 bytes are needed.
  • Incremental, bounded CPU use – Input data is processed in tiny bites
  • Static or dynamic memory allocation
  • Released under an ISC license which allows you to use the library freely, even in commercial products.

The internal workings of the library are explained as follows:

Heatshrink is based on LZSS, since it’s particularly suitable for compression in small amounts of memory. It can use an optional, small index to make compression significantly faster, but otherwise can run in under 100 bytes of memory. The index currently adds 2^(window size+1) bytes to memory usage for compression, and temporarily allocates 512 bytes on the stack during index construction (if the index is enabled).

So let’s give it a try on my laptop, which I reckon is not exactly a resources-constrained embedded system.


After the build is complete we’ll get the libraries (libheatshrink_dynamic.a
and libheatshrink_static.a) which you could use with the header files to integrate into your own project, as well as one 109KB binary (27KB after being stripped) called heatshrink used to encode and decode files. Note that when compiled for an Arduino microcontroller with avr-gcc, the decoder only takes about 1 KB of storage space.

But before using the utilities, we must understand the three options:

  • window_sz2, -w – Sets the window size to 2^W bytes. The window size determines how far back in the input can be searched for repeated patterns. A window_sz2 of 8 will only use 256 bytes (2^8), while a window_sz2 of 10 will use 1024 bytes (2^10) with tradeoffs in terms of memory usage and compression ratio.  It can be set between 4 and 15.
  • lookahead_sz2, -l – Sets the lookahead size to 2^L bytes. The lookahead size determines the max length for repeated patterns that are found. For instance, if the lookahead_sz2 is 4, a 50-byte run of ‘a’ characters will be represented as several repeated 16-byte patterns (2^4 is 16). It’s usually set to around half the size of windows_sz2

Let compress/encode the latest Linux 5.14 changelog with heatshrink using 8/4 and 13/4 parameters:


It will take longer, and consume more memory and CPU cycles with a larger window size, but the compression ratio is also much better.


So you’d have to adjust this to optimize the values considering the resources available on your target. To decompress we just need to replace the first argument:


Decompression is quite faster. A quick check with diff shows the files are identical.

The Makefile also comes with a benchmark option (but I had to download the cantrbry.tar.gz file manually first) that will test various files and settings:


It would be nice to compare heatshrink to another popular compression library like gzip, and that’s exactly what Scott Vokes in a blog post after he released the library in 2013.

heatshrink vs gzip

gzip has a better compression ratio in every case, but obviously uses a lot more RAM than heatshrink, and with over 128KB of RAM used not suitable for many microcontrollers. If you are using the compression library with the lower settings you may want to check if it’s not actually increasing the size as HS 6,3 and HS 5,3 options in the table above did have some of the compressed files larger than the original. Still, it’s a nice option to have, and I can see somebody made an Arduino library based on Heatshrink as well.

Share this:

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK 5 ITX Rockchip RK3588 mini-ITX motherboard
Subscribe
Notify of
guest
The comment form collects your name, email and content to allow us keep track of the comments placed on the website. Please read and accept our website Terms and Privacy Policy to post a comment.
9 Comments
oldest
newest
RK
RK
3 years ago

Considering the subject is lzw compressions, I guess it’s a good a time to mention encode.ru moved to encode.su.

3 years ago

Looks like we think similarly.

Willy
3 years ago

Thanks for sharing this! The compression ratios are very good! I compared with SLZ that I created for the same memory considerations reason, that produces a gzip-compatible output stream and that is particularly suitable for use in HTTP proxies and servers, and I need to use larger blocks to achieve the same ratio as HSS8,4 (gzip output format is not that efficient). However the permanent state remains fixed at 24 bytes. Interestingly the compression ratio for kennedy.xls is quite poor in my case (66%), I’ll have to check why, I suspect that many literals are emitted and that their encoding… Read more »

repkid
repkid
3 years ago

nice, didn’t know about this one.

wanderer_
3 years ago

Ooh, very clever name!

dgp
dgp
3 years ago

I wonder how big this will be on something really crap like an 8051? If the code did compile to something like 50 bytes it could be useful for squashing static data like USB descriptors etc.

willy
willy
3 years ago

No it’s not 50 bytes of code, it’s 50 bytes of state to have to be kept between subsequent calls to the compressor. The code is a bit larger. Also it will possibly be very slow on a 8051 because decompression is already 100 times slower than gunzip on a PC. But in order to decompress small code once at boot that might be totally fine.

dgp
dgp
3 years ago

*re-reads article* – 50 bytes of RAM .. mm

I’m writing a replacement for the CH341 on the breadbee with a CH552 to allow for serial and for programming blank flash (SoC has a secret flashing interface on the UART pins) so I can finally sell them without getting in copyright trouble. Very close to filling the flash. Compressing the USB descriptors with something might have helped if the code wasn’t bigger than the descriptors. 🙂

fanoush
fanoush
3 years ago

BTW it is Espruino not ESPruino. The name has nothing to do with ESP32/8266 chips. Esp comes from ECMAScript=Javascript. Even the Javascript standards are called ESx https://www.espruino.com/Features

Boardcon Rockchip RK3588S SBC with 8K, WiFI 6, 4G LTE, NVME SSD, HDMI 2.1...