Posts Tagged ‘elce 2017’
Orange Pi Development Boards

MINIX based Intel Management Engine Firmware & UEFI are Closed Source & Insecure, NERF to the Rescue!

November 7th, 2017 12 comments

You may have heard a few things about Intel Management Engine in recent months, especially as security issues have been found, the firmware is not easily upgradeable, and the EFF deemed it a security hazard asking Intel for ways to disable it.

In recent days, I’ve seen several media reports about the Management Engine being based on an Intel Quark x86-based 32-bit CPU running MINIX open-source operating system. Keep in mind, there’s nothing nefarious about MINIX, it’s just that Intel keeps its own developments on top closed. One of sources for the information is a blog post explaining how to disable Intel ME 11, but ZDNET also points to one of the talks at the Embedded Linux Conference Europe 2017 entitled “Replace Your Exploit-Ridden Firmware with Linux” by Ronald Minnich, Google which explains the problem, and proposes a solution to (almost) disable Intel’s ME, and replace UEFI by a small open source Linux kernel and ramdisk.

Click to Enlarge

To better understand about the issue, we’ll first need to talk about rings… yes… rings. Protection rings with numbers from -3 to +3 indicate the level of privileges with Ring 3 being the lowest priviledge, and Ring -3 giving full access to all hardware. It’s unrelated to OS privileges and normal users and root are are part of the same Ring 3 privileges.

Ring 0 to 3 are well documented, Ring -1 is for hypervisors like Zen so “known to mankind”, but while we know something about Ring -2 CPU, the code for the UEFI kernel and SMM half kernel are not always known, and in the case of Ring -3 kernels which include the Management Engine, Integrated Sensor Hub (ISH), and Innovation Engine (IE) we know very little about the hardware and software, despite them having the highest privilege.

We do know the Management Engine Ring -3 OS provides the following features.

Full Network manageability ICC Over Clocking
Regular Network manageability Protected Audio Video Path (PAVP)
Manageability IPV6
Small business technology KVM Remote Control (KVM)
Level III manageability Outbreak Containment Heuristic (OCH)
IntelR Anti-Theft (AT) Virtual LAN (VLAN)
IntelR Capability Licensing TLS
Service (CLS) Wireless LAN (WLAN)
IntelR Power Sharing Technology (MPC)

If you have no idea what some of the feature do, that’s OK, as even Richard is unclear. The important part is that the firmware has a full network stack, and web servers are running for remote management, which has recently become a serious problem as Intel found a vulnerability in “Intel Active Management Technology (AMT), Intel Standard Manageability (ISM), and Intel Small Business Technology” that can allow “an unprivileged attacker to gain control of the manageability features”.  It requires a firmware update, but considering this affect Intel’s first to seven generation, the bug is at least 9 years old, and most systems won’t be updated. Read this PDF for a detailed (71 pages) security evaluation of Intel’s ME

Beside the Management Engine, the presentation also goes through Ring -2 OS (UEFI), an extremely complex kernel (millions of lines of code) running on the main CPU, and whose security model is… obscurity. UEFI exploits also exist, and can be made permanent since UEFI can rewrite itself. The firmware also always runs, and exploits are undetectable by kernels and programs.

The solution proposed to address the privacy and security issues related to ME and UEFI is called NERF (Non-Extensible Reduce Firmware).

It’s only a partial solution because the system cannot fully boot without ME, but they’ve managed to reduce the size from 5MB to 300KB, removing the web server and IP stack in the process on Minnowboard MAX board thanks to me_cleaner. The SMM Ring -2 semi kernel can be disabled however.

UEFI stands for Unified Extensible Firmware Interface, so to simplify the code and make it less vulnerable to exploits they’ve made their implementation NON-extensible, and as a result it’s much simpler, as illustrated in the diagram below.


UEFI DXE stage is replaced with a single Linux kernel (tied to the BIOS vendor), and a 5.9MB firmware-based root file system written in Go ( If you are interesting in the details watching the 38 minutes ELCE 2017 presentation.

You may also be interested in the slides.

Learn More About Linux’s New GPIO User Space Subsystem & Libgpiod

November 3rd, 2017 10 comments

Sysfs was used to control GPIOs on Linux system with the GPIOs defined in /sys/class/gpio, but starting with Linux 4.8, a new GPIO interface for user space was introduced, with gpiochip becoming char devices foudn in /dev/gpiochip0, dev/gpiochip1, etc.. , and sysfs allegedly become deprecated.

But a quick check in NanoPi Duo with Linux 4.11 shows both GPIO user space interfaces appear to be enabled:

Nevertheless overtime, sysfs will die out, and the new subsystem will likely be used by all systems, so it might be useful to learn more about it.

One way to do that is to watch Bartosz Golaszewski’s ELCE 2017 talk entitled “New GPIO Interface for User Space” with the video embedded below. But I first I’ll summarize some of the key points.

Now GPIO handling from user space becomes similar to other char driver with ioctl, poll and read functions, and beside assigning numbers to GPIOs you can assign names. The API (in linux/gpio.h) provides access to chip info, line info, line request for values, reading values, settings values, line request for events (rise/falling edges), polling for events, and reading events. Bartosz goes into details for each function in his talk.

Since the kernel API is a bit complicated to use, there’s also a C library called libgpiod, which comes with some tools like gpiodetect, gpioinfo, gpioset, gpioget, gpiofind & gpiomon. Further work includes C++ and Python bindings, as well as a GPIO daemon and client. Example code for libgpiod:

Some example of user space tools found in libgpiod:

The first command return the list of gpio chips, their names, and number of lines, the second set the 3rd pin of Chip 1 to high, the third return the values of 5 different pins, and the last one monitor a pin to detect an event and return the time when it happened.

You can check out the latest libgpiod source code, or download the latest stable release of the code as a tarball. libgpiod is already available in meta-openembedded & buildroot, and packaged in Fedora and Arch Linux.

You may also be interested in the slides.

Lab in a Box Concept Embeds x86 Server and 6 ARM Boards into a PC Case for Automated Software Testing

November 3rd, 2017 7 comments

The Linux kernel now has about 20 millions line of code, Arm has hundreds of licensees making thousands of processors and micro-controllers, which end up in maybe hundreds of thousands of different designs, many of which are not using Linux, but for those that do, Linux must be tested to make sure it works. The same stands true for any large software used on multiple hardware platforms.

Manual testing is one way to do it, but it’s time consuming and expensive, so there are software and hardware continuous integration solutions to automate testing such as Linaro LAVA (Linaro Automated Validation Architecture), KernelCI automated Linux kernel testing, and Automotive Grade Linux CIAT that automatically test incoming patch series.

Both CIAT and KernelCI focus on Linux, and rely on LAVA, with KernelCI leveraging hardware contributed by the community, and proven to be effective as since it’s been implemented, failed build configs dropped from 51 with Linux 3.14 to zero today. However, settings the hardware and LAVA can be complicated and messy with all different boards lying around, so BayLibre engineers worked on an affordable “Lab in Box” concept to simplify administration and duplication of such systems in the hope of getting more people involved.

Click to Enlarge

They ended up with a nicely package system that fits into a desktop PC tower and includes:

  • ASRock Q1900B-ITX motherboard based on Intel Celeron J1900 with 8GB RAM and 120GB SSD running LAVA master and dispatcher
  • Devices Under Tests (DUT) will vary depending on your needs, but the demo system includes:
    • Renesas R-Car M3 Starter Kit
    • DragonBoard 410C
    • AML-S905-CC (LePotato) board
    • BeagleBone Black
    • Raspberry Pi 3
    • NXP SABRE Light
  • Connectivity / wiring
    • Network switch
    • USB hub
    • For each DUT board: power cable, serial debug cable, Ethernet cable
  • ACME Cape + Probes + Beaglebone Black to measure power consumption and control the DUTs
  • Power Supply – 530 Watt ATX power supply with +12V and +5V

Click to Enlarge

The system has been proven to work with complete continuous integration system fitted into a single PC case, and costing about 400 Euros excluding the DUTs. Software installation has also been simplified with partially automated software installations (WiP). However there may still need to work, as it’s been found to take a long time to build partially because it’s requires custom wiring for each DUT, boards need to support either 5 or 12V input, and DUT power consumption must be limited to 4A per pair of wires. This system also only supports board that fit into such case, and it’s not really scalable because using a larger case with more board may lead to excessive internal wiring. The Lab in a Box concept could be improved with a more powerful power supply, support for larger boards, and better documentation will also be provided. Baylibre may also work on a professional-grade “Lab in a Box” that fits into a rack.

Watch “Introducing the Lab in a Box Concept” by Patrick Titiano & Kevin Hilman, BayLibre for further details.

If you are short in time, you can also read the presentation slides.

As a side note, all Embedded Linux Conference Europe 2017 videos have been uploaded to YouTube.

Embedded Linux Conference & Open Source Summit Europe 2017 Schedule

August 27th, 2017 4 comments

The Embedded Linux Conference & IoT summit 2017 took place in the US earlier this year in February, but there will soon be a similar event with the Embedded Linux Conference *& Open Source Summit Europe 2017 to take up in Europe on October 23 – 25 in Prague, Czech Republic, and the Linux Foundation has just published the schedule. It’s always useful to find out what is being discussed during such events, even if you are not going to attend, so I went through the different sessions, and compose my own virtual schedule with some of the ones I find the most interesting.

Monday, October 23

  • 11:15 – 11:55 – An Introduction to SPI-NOR Subsystem – Vignesh Raghavendra, Texas Instruments India

Modern day embedded systems have dedicated SPI controllers to support NOR flashes. They have many hardware level features to increase the ease and efficiency of accessing SPI NOR flashes and also support different SPI bus widths and speeds.

In order to support such advanced SPI NOR controllers, SPI-NOR framework was introduced under Memory Technology Devices (MTD). This presentation aims at providing an overview of SPI-NOR framework, different types of NOR flashes supported (like SPI/QSPI/OSPI) and interaction with SPI framework. It also provides an overview of how to write a new controller driver or add support for a new flash device.

The presentation then covers generic improvements done and proposed while working on improving QSPI performance on a TI SoC, challenges associated when using DMA with these controllers and other limitations of the framework.

  • 12:05 – 12:45 – Free and Open Source Software Tools for Making Open Source Hardware – Leon Anavi, Konsulko Group

The open source hardware movement is becoming more and more popular. But is it worth making open source hardware if it has been designed with expensive proprietary software? In this presentation, Leon Anavi will share his experience how to use free and open source software for making high-quality entirely open source devices: from the designing the PCB with KiCAD through making a case with OpenSCAD or FreeCAD to slicing with Cura and 3D printing. The talk will also provide information about open source hardware licenses, getting started guidelines, tips for avoiding common pitfalls and mistakes. The challenges of prototyping and low-volume manufacturing with both SMT and THT will be also discussed.

  • 14:20 – 15:00 – Introduction to SoC+FPGA – Marek Vašut, DENX Software Engineering GmbH

In this talk, Marek introduces the increasingly popular single-chip SoC+FPGA solutions. At the beginning, the diverse chip offerings from multiple vendors are introduced, ranging from the smallest IoT-grade solutions all the way to large industrial-level chips with focus on their software support. Mainline U-Boot and Linux support for such chips is quite complete, and already deployed in production. Marek demonstrates how to load and operate the FPGA part in both U-Boot and Linux, which recently gained FPGA manager support. Yet to fully leverage the potential of the FPGA manager in combination with Device Tree (DT) Overlays, patches are still needed. Marek explains how the FPGA manager and the DT Overlays work, how they fit together and how to use them to obtain a great experience on SoC+FPGA, while pointing out various pitfalls.

  • 15:10 – 15:50 – Cheap Complex Cameras – Pavel Machek, DENX Software Engineering GmbH

Cameras in phones are different from webcams: their main purpose is to take high-resolution still pictures. Running preview in high resolution is not feasible, so resolution switch is needed just before taking final picture. There are currently no applications for still photography that work with mainline kernel. (Pavel is working on… two, but both have some limitations). libv4l2 is doing internal processing in 8-bit, which is not enough for digital photography. Cell phones have 10 to 12-bit sensors, some DSLRs do 14-bit depth.

Differences do not end here. Cell phone camera can produce reasonable picture, but it needs complex software support. Auto-exposure / auto-gain is a must for producing anything but completely black or completely white frames. Users expect auto-focus, and it is necessary for reasonable pictures in macro range, requiring real-time processing.

  • 16:20 – 17:00 – Bluetooth Mesh with Zephyr OS and Linux – Johan Hedberg, Open Source Technology Center, Intel

Bluetooth Mesh is a new standard that opens a whole new wave of low-power wireless use cases. It extends the range of communication from a single peer-to-peer connection to a true mesh topology covering large areas, such as an entire building. This paves the way for both home and industrial automation applications. Typical home scenarios include things like controlling the lights in your apartment or adjusting the thermostat. Although Bluetooth 5 was released end of last year, Bluetooth Mesh can be implemented on any device supporting Bluetooth 4.0 or later. This means that we’ll likely see very rapid market adoption of the feature.

The presentation will give an introduction to Bluetooth Mesh, covering how it works and what kind of features it provides. The talk will also give an overview of Bluetooth Mesh support in Zephyr OS and Linux and how to create wireless solutions with them.

  • 17:10 – 17:50 – printk() – The Most Useful Tool is Now Showing its Age – Steven Rostedt, VMware

printk() has been the tool for debugging the Linux kernel and for being the display mechanism for Linux as long as Linux has been around. It’s the first thing one sees as the life of the kernel begins, from the kernel banner and the last message at shutdown. It’s critical as people take pictures of a kernel oops to send to the kernel developers to fix a bug, or to display on social media when that oops happens on the monitor on the back of an airplane seat in front of you.

But printk() is not a trivial utility. It serves many functionalities and some of them can be conflicting. Today with Linux running on machines with hundreds of CPUs, printk() can actually be the cause of live locks. This talk will discuss all the issues that printk() has today, and some of the possible solutions that may be discussed at Kernel Summit.

  • 18:00 – 18:45 – BoF: Embedded Linux Size – Michael Opdenacker, Free Electrons

This “Birds of a Feather” session will start by a quick update on available resources and recent efforts to reduce the size of the Linux kernel and the filesystem it uses.

An ARM based system running the mainline kernel with about 3 MB of RAM will also be demonstrated. If you are interested in the size topic, please join this BoF and share your experience, the resources you have found and your ideas for further size reduction techniques!

Tuesday, October 24

  • 10:55 – 11:35 – Introducing the “Lab in a Box” Concept – Patrick Titiano & Kevin Hilman, BayLibre

Continuous Integration (CI) has been a hot topic for long time. With the growing number of architectures and boards, it becomes impossible for maintainers to validate a patch on all configurations, making it harder and harder to keep the same quality level without leveraging CI and test automation. Recent initiatives like LAVA,, Fuego, (…) started providing a first answer, however the learning curve remains high, and the HW setup part is not covered.

Baylibre, already involved in, decided, as part of the AGL project, to go one step further in CI automation and has developed a turnkey solution for developers and companies willing to instantiate a LAVA lab; called “Lab in a Box”, it aims at simplifying the configuration of a board farm (HW, SW).

Motivations, challenges, benefits and results will be discussed, with a demo of a first “Lab in a Box” instantiation.

  • 11:45 – 12:25 – Protecting Your System from the Scum of the Universe – Gilad Ben-Yossef, Arm Holdings

Linux based systems have a plethora of security related mechanisms: DM-Crypt, DM-Verity, Secure Boot, the new TEE sub-system, FScrypt and IMA are just a few examples. This talk will describe these the various systems and provide a practical walk through of how to mix and match these mechanisms and design them into a Linux based embedded system in order to strengthen the system resilience to various nefarious attacks, whether the system discussed is a mobile phone, a tablet, a network attached DVR, a router, or an IOT hub in a way that makes maximum use of the sometime limited hardware resources of such systems.

  • 14:05 – 14:45 – Open Source Neuroimaging: Developing a State-of-the-Art Brain Scanner with Linux and FPGAs – Danny Abukalam, Codethink

Neuroimaging is an established medical field which is helping us to learn more about how the human brain works, the most complex human organ. This talk aims to cover neuroimaging systems, from hobbyist to professional, and how open source has been used to build state-of-the-art systems. We’ll have a look the general problem area, why open source was a good fit, and some examples of solutions including a commercial effort that we have been involved in bringing to market. Typically these solutions consist of specialist hardware, a bespoke software solutions stack, and a suite to manage and process the vast amounts of data generated during the scan. Other points of interest include how we approached building a maintainable and upgradeable system from the outset. We’ll also talk about future plans for neuroimaging, future ideas for hardware & discuss areas lacking good open source solutions.

  • 14:55 – 15:35 – More Robust I2C Designs with a New Fault-Injection Driver – Wolfram Sang, Renesas

It has its challenges to write code for certain error paths for I2C bus drivers because these errors usually don’t happen on the bus. And special I2C bus testers are expensive. In this talk, a new GPIO based driver will be presented which acts on the same bus as the bus master driver under inspection. A live demonstration will be given as well as hints how to handle bugs which might have been found. The scope and limitations of this driver will be discussed. Since it will also be analyzed what actually happens on the wires, this talk also serves as a case study how to snoop busses with only Free Software and OpenHardware (i.e. sigrok).

  • 16:05 – 16:45 – GStreamer for Tiny Devices – Olivier Crête, Collabora

GStreamer is a complete Open Source multimedia framework, and it includes hundreds of plugins, including modern formats like DASH, HLS or the first ever RTSP 2.0 implementation. The whole framework is almost 150MB on my computer, but what if you only have 5 megs of flash available? Is it a viable choice? Yes it is, and I will show you how.

Starting with simple tricks like only including the necessary plugins, all the way to statically compiling only the functions that are actually used to produce the smaller possible footprint.

  • 16:55 – 17:35 – Maintaining a Linux Kernel for 13 Years? You Must be Kidding Me. We Need at Least 30? – Agustin Benito Bethencourt, Codethink Ltd

Industrial grade solutions have a life expectancy of 30+ years. Maintaining a Linux kernel for such a long time in the open has not been done. Many claim that is not sustainable, but corporations that build power plants, railway systems, etc. are willing to tackle this challenge. This talk will describe the work done so far on the kernel maintenance and testing front at the CIP initiative.

During the talk it will be explained how we decide which parts of the kernel to cover – reducing the amount of work to be done and the risk of being unable to maintain the claimed support. The process of reviewing and backporting fixes that might be needed on an older branch will be briefly described. CIP is taking a different approach from many other projects when it comes to testing the kernel. The talk will go over it as well as the coming steps. and the future steps.

Wednesday, October 24

  • 11:05 – 11:45 – HDMI 4k Video: Lessons Learned – Hans Verkuil, Cisco Systems Norway

So you want to support HDMI 4k (3840×2160) video output and/or video capture for your new product? Then this is the presentation for you! I will describe the challenges involved in 4k video from the hardware level, the HDMI protocol level and up to the kernel driver level. Special attention will be given to what to watch out for when buying 4k capable equipment and accessories such as cables and adapters since it is a Wild, Wild West out there.

  • 11:55 – 12:35 – Linux Powered Autonomous Arctic Buoys – Satish Chetty, Hera Systems 

In my talk/presentation, I cover the technical, and design challenges in developing an autonomous Linux powered Arctic buoy. This system is a low cost, COTS based, extreme/harsh environment, autonomous sensor data gathering platform. It measures albedo, weather, water temperature and other parameters. It runs on a custom embedded Linux and is optimized for efficient use of solar & battery power. It uses a variety of low cost, high accuracy/precision sensors and satellite/terrestrial wireless communications.

I talk about using Linux in this embedded environment, and how I address and solve various issues including building a custom kernel, Linux drivers, frame grabbing issues and results from cameras, limited power challenges, clock drifts due to low temperature, summer melt challenges, failure of sensors, intermittent communication issues and various other h/w & s/w challenges.

  • 14:15 – 14:55 – Linux Storage System Bottleneck for eMMC/UFS – Bean Huo & Zoltan Szubbocsev, Micron

The storage device is considered a bottleneck to the system I/O performance. This thinking drives the need for faster storage device interfaces. Commonly used flash based storage interfaces support high throughputs, eg. eMMC 400MB/s, UFS 1GB/s. Traditionally, advanced embedded systems were focusing on CPU and memory speeds and these outpaced advances in storage speed improvements. In this presentation, we explore the parameters that impact I/O performance. We describe at a high level how Linux manages I/O requests coming from user space. Specifically, we look into system performance limitations in the Linux eMMC/UFS subsystem and expose bottlenecks caused by the software through Ftrace. We show existing challenges in getting maximum performance of flash-based high-speed storage device. by this presentation, we want to motivate future optimization work on the existing storage stack.

  • 15:05 – 15:45 – New GPIO Interface for User Space – Bartosz Golaszewski

Since Linux 4.8 the GPIO sysfs interface is deprecated. Due to its many drawbacks and bad design decisions a new user space interface has been implemented in the form of the GPIO character device which is now the preferred method of interaction with GPIOs which can’t otherwise be serviced by a kernel driver. The character device brings in many new interesting features such as: polling for line events, finding GPIO chips and lines by name, changing & reading the values of multiple lines with a single ioctl (one context switch) and many more. In this presentation, Bartosz will showcase the new features of the GPIO UAPI, discuss the current state of libgpiod (user space tools for using the character device) and tell you why it’s beneficial to switch to the new interface.

  • 16:15 – 16:55 – Replace Your Exploit-Ridden Firmware with Linux – Ronald Minnich, Google

With the WikiLeaks release of the vault7 material, the security of the UEFI (Unified Extensible Firmware Interface) firmware used in most PCs and laptops is once again a concern. UEFI is a proprietary and closed-source operating system, with a codebase almost as large as the Linux kernel, that runs when the system is powered on and continues to run after it boots the OS (hence its designation as a “Ring -2 hypervisor”). It is a great place to hide exploits since it never stops running, and these exploits are undetectable by kernels and programs.

Our answer to this is NERF (Non-Extensible Reduced Firmware), an open source software system developed at Google to replace almost all of UEFI firmware with a tiny Linux kernel and initramfs. The initramfs file system contains an init and command line utilities from the u-root project, which are written in the Go language.

  • 17:05 – 17:45 – Unikernelized Real Time Linux & IoT – Tiejun Chen, Vmware

Unikernel is a novel software technology that links an application with OS in the form of a library and packages them into a specialized image that facilitates direct deployment on a hypervisor. But why these existing unikernels have yet to gain large popularity broadly? I’ll talk what challenges Unikernels are facing, and discuss exploration of if-how we could convert Linux as Unikernel, and IoT could be a valuable one of use cases because the feature of smaller size & footprint are good for those resource-strained IoT platforms. Those existing unikernels are not designed to address those IoT characters like power consumption and real time requirement, and they also doesn’t support versatile architectures. Most existing Unikernels just focus on X86/ARM. As a paravirtualized unikenelized Linux, especially Unikernelized Real Time Linux, really makes Unikernels to succeed.

If you’d like to attend the real thing, you’ll need to register and pay a registration fee:

  • Early Registration Fee: US$800 (through August 27, 2017)
  • Standard Registration Fee: US$950 (August 28, 2017 – September 17, 2017)
  • Late Registration Fee: US$1100 (September 18, 2017 – Event)
  • Academic Registration Fee: US$200 (Student/Faculty attendees will be required to show a valid student/faculty ID at registration.)
  • Hobbyist Registration Fee: US$200 (only if you are paying for yourself to attend this event and are currently active in the community)

There’s also another option with the Hall Pass Registration ($150) if you just want to network on visit with sponsors onsite, but do not plan to attend any sessions or keynotes.