Embedded Device Security: Protecting Linux Systems with Modern Tools

CNXSoft: This is a guest port by Erik Wierich, Senior Engineer at RISCstar Solutions, demonstrating a practical security implementation for embedded devices using standard Linux tools like dm-verity and TPM 2.0. It covers threat models, filesystem security, and TPM-based encryption with working code examples.

Nowadays, it is (rightfully) impossible to put an embedded device into the market without comprehensive embedded device security measures. Most new devices store private data that we do not want to see leaked in dark corners of the Internet. We also want to avoid our device ending up as part of a botnet.

Linux has a large set of tools to help us with security. What has historically been lacking is a simple, off-the-shelf way to integrate these tools into a secure-by-default configuration. This post will demonstrate how modern tools simplify deployments while ensuring strong security.

Embedded Device Security Scope

When talking about embedded system security, we should also talk about what we are actually trying to protect against. It is impossible to generalize a threat model over all possible embedded devices, but we can try to collect some typical requirements.

Usually, we will find two variants of data on a system:

• Fixed binaries and libraries forming the operating system
• Variable configuration and user data that is changed at runtime

For the operating system, we need to ensure the integrity of the factory image. This guards against supply chain attacks and the booting of unauthorized software. However, for the purpose of this blog post, we won’t consider the operating system binaries as confidential.

We do consider the dynamic configuration and user data to be confidential. This data is only generated while the device is running and is not flashed at the factory or replaced during updates. We definitively want to protect this data from being read out on stolen devices.

There are way more options and scenarios that might be relevant to your specific device. But this gives us a good basis to build on!

Filesystem configuration

Sounds like a plan? Let’s move to execution…

Linux allows very flexible filesystem configurations. For a typical embedded device, it usually boils down to some places in /etc, /var, and /home being required to be writable while the rest can remain an immutable image.

We can implement the root file system (/) as immutable and mount the various writable locations. However, managing multiple exceptions from the norm is cumbersome. Instead, there is a much easier alternative: make / writable and mount a read-only /usr partition. /usr is where all the system files are stored. They are all immutable, and any other files can be automatically populated on the first boot. This allows us to drop the / partition from the factory image and only ship the /usr part.

This might seem like a slightly counterintuitive approach at first. But it makes managing the writable partition a lot easier. Having to only manage two partitions without any symlinks between them is the most obvious benefit. It also makes the cut between the different requirements for the two partitions easier. One will require integrity checks, and the other partition will require encryption.

Key management also becomes easier (and more secure) if we can create the keys on the device on first boot. Incorporating per-device encryption into a deployment that happens from a single factory image is a lot simpler if we create the encrypted partitions on the device. Otherwise, we will find ourselves having to do some awkward in-place re-encryption of these partitions. Another thing that also becomes easier with partitions created at the first boot is a factory reset mechanism. We can simply toss out the encrypted partitions, delete the keys, and start fresh by recreating them.

Now, the critical reader may be wondering how we survive the first boot attempt without having a / filesystem. Linux really needs it for switching to user-space eventually. Thus, we need to find a replacement. An initrd filesystem will come to the rescue—a minimal filesystem. Don’t worry, setting that one up will be a lot easier than one might assume.

Securing an Embedded Device Operating System

Dealing with an immutable filesystem makes integrity checks fairly easy. We can simply guard the entire partition block by putting it into a dm-verity partition. This builds a hash tree over the blocks of the filesystem, leading to eventually a single root-hash that can be used to verify the integrity of the entire hash tree and thus also the filesystem.

Now, we still need to somehow sign that root-hash to ensure it originates from our trusted build process. This is typically the part where complexity starts to creep in. One option is to bake the hash into the kernel commandline by appending usrhash=, which then will be signed together with the other boot artifacts. This seems straightforward, but complicates the build process as we now need to build the /usr partition before injecting the usrhash into the kernel commandline–or more likely: into a Unified Kernel Image (UKI). This leads to our /boot partition depending on the build of /usr or requires some patching of the built image to inject the usrhash= option.

The alternative is to have some metadata elsewhere that contains the hash and a signature over it. People have come up with various ways to do this. But lately, a group of systemd and Linux distribution maintainers has been formed to define standards for this. The uapi-group now provides a standard for securely auto-discovering the root-hash and its signature. Even better: We do not have to bother with the whole dm-verity setup at all. systemd-repart has support for it and will automate the whole thing for us. Systemd also comes with its counterpart systemd-gpt-auto-generator that does the discovery and mounting for us.

Side topic: systemd-repart

systemd-repart at its core does exactly what the name suggests: It allows repartitioning of a disk based on some simple configuration files.

Given the following configuration:

It will compare which partitions exist on the booted disk to the configuration and create the missing ones. The configuration format is simple, yet powerful.

But this is not even the best feature of systemd-repart. It can also create disk images from scratch!

An example:

With this config, we can call:

It will handle all the dm-verity setup, signing, and assign the well-known GPT partition UUIDs that systemd-gpt-auto-generator can detect to auto-mount.

With this, we will get our /usr burned into the factory image and then use systemd-repart again on the device to fill in the remaining parts.

Protecting Writable Data in Embedded Device Security

With the static part out of the way, let’s look at the dynamic / partition. We want to protect against this being dumped on stolen devices. So we will need some kind of encryption.

Asking a user for a pincode is an easy way to solve this. But this becomes impractical for headless appliances. Instead, we will need some kind of secure mechanism to only release some key materials when the system is in a well-defined state. Various chip vendors have provided such mechanisms for a while. But securely implementing those mechanisms is challenging and often requires getting an NDA to even peek at the relevant reference manual.

Luckily, we again have a standard to reach for: Trusted Platform Modules (TPMs). What sounds like a specific hardware solution is actually more of an API than actual hardware. TPMs can also be implemented in firmware solutions that are running in isolated environments on the main CPU. On embedded devices where we can have great control over the kind of firmware that we run, firmware TPMs can be a great way to get a standard-compliant TPM solution without needing any actual TPM hardware.

Covering the TPM 2.0 standard in detail is beyond the scope of this blog post. What matters is that the standard gives us a mechanism to bind keys to specific system states. These keys are then only accessible in those specific states. TPM and UEFI work well together, and there are registers reserved for various states of the firmware to measure into. uapi-group defines a list of registers and their allocation that are relevant for a typical Linux system.

For the scope of this post, we will only look at the register named “PCR #7”. This register accumulates all the state that impacts the secure boot state. It allows us to bind to the keys and signatures that were used to confirm the various boot components, effectively giving us a way to only reveal a key when the earlier chain was verified successfully. This guards the encryption keys as long as nothing can read the TPM after secure boot. With firmware TPMs, that means that our keys are safe if our secure boot mechanism is safe.

How to set all of this up? Easy: Use systemd-repart!

systemd-repart will automatically create an encrypted partition, enroll it with the TPM, and create a filesystem within. The filesystem will have the right partition UUID set so that it can be automatically detected by a systemd-based initrd.

Putting it together

Let’s put it all together. We will use mkosi to build our factory image. It integrates well with systemd tooling and saves us from building everything from scratch.

We will use a simple config to build a custom image based on Fedora packages:

Then we can drop our partition config into mkosi.repart/ (this will define the factory image partition layout). The one I showed in the earlier section will do fine.

Finally, we can drop the runtime partition config into mkosi.extra (which will be added into the image):

We can build and boot the system with

I provide the example config in a repo: https://github.com/riscstar/blogpost-secure-boot-userspace

Summary

This blog shows that security on embedded Linux systems does not need to be hard. We live in a world where we can simply pick components off the shelf and achieve better security than 90% of existing embedded deployments.

Of course, embedded device security always requires an analysis specific to the needs of the specific system. This post provides a good basis for creating a secure, encrypted root filesystem. It does that while keeping complexity low. But when giving security advice, one also has to present the caveats. This blog does not cover everything for secure embedded system design. I do not cover more advanced ways to narrow the key reveal to just the initrd. This can help to further limit the attack surface. You will also need rollback protection to prevent an attacker from booting an older, signed kernel with known security defects. Such a kernel can otherwise be exploited to access your confidential data.

Finally, I only demo a simple mkosi image here. It is not difficult to apply the same mechanisms to Yocto-built images as well. Stay safe and make sure your systems do as well!

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress. We also use affiliate links in articles to earn commissions if you make a purchase after clicking on those links.