Archive

Posts Tagged ‘real-time’

A Look at Three Options to Develop Real-Time Linux Systems on Application Processors – HMP, Real-Time Linux and Xenomai

October 15th, 2016 6 comments

This is a guest post by written by Guilherme Fernandes, Raul Muñoz, Leonardo Veiga, Brandon Shibley, all working for Toradex.

Introduction

Application processor usage continues to broaden. System-on-Chips, usually powered by ARM Cortex-A cores, are taking over several spaces where small ARM Cortex-M, and other microcontroller devices, have traditionally dominated. This trend is driven by several facts, such as:

  • The strong requirements for connectivity, often related to IoT and not only from a hardware point of view, but also related to software, protocols and security
  • The need for highly interactive interfaces such as multi-touch, high resolution screens and elaborate graphical user interfaces;
  • The decreasing price of SoCs, as consequence of its volume gain and new production capabilities.

Typical cases exemplifying the statement above are the customers we see every day starting a product redesign upgrading from a microcontroller to a microprocessor. This move offers new challenges as the design is more complicated and the operating system abstraction layer is much re complex. The difficulty of hardware design using an application processor is overcome by the use of reference designs and off-the-shelf alternatives like computer-on-modules or single board computers. On the operating system layer, the use of embedded Linux distributions is widespread in the industry. An immense world of open source tools is available simplifying the development of complex and feature rich embedded systems. Such development would be very complicated and time consuming if using microcontrollers. Despite all the benefits, the use of an operating system like Linux still raises a lot of questions and distrust when determinism and real-time control application topics are addressed.

A common approach adopted by developers is the strategy of separating time-critical tasks and regular tasks onto different processors. Hence, a Cortex-A processor, or similar, is typically selected for multimedia and connectivity features while a microcontroller is still employed to handle real-time, determinism-critical tasks. The aim of this article is to present some options developers may consider when developing real-time systems with application processors. We present three possible solutions to provide real-time capability to application processor based designs.

Heterogeneous Multicore Processing

The Heterogeneous Multicore Processing (HMP) approach is a hardware solution. Application processors like the NXP i.MX7 series, the NXP i.MX6SoloX and the upcoming NXP i.MX8 series present a variety of cores with different purposes. If we consider the i.MX7S you will see a dual core processor composed of a Cortex-A7 core @ 800MHz side-by-side with a Cortex-M4 core @ 200MHz. The basic idea is that user interface and high-speed connectivity are implemented on an abstracted OS like Linux with the Cortex-A core while, independently and in parallel, executing control tasks on a Real-Time OS, like FreeRTOS, with the Cortex-M core. Both cores are able to share access to memory and peripherals allowing flexibility and freedom when defining which tasks are allocated to each core/OS. Refer to Figure 1.

NXP i.MX7 Block Diagram (Click to Enlarge)

Figure 1 – NXP i.MX7 Block Diagram (Click to Enlarge)

Some of the advantages of using the HMP approach are:

  • Legacy software from microcontrollers can be more easily reused;
  • Firmware update (M4 core) is simplified as the firmware may be a file at the filesystem of the Cortex-A OS;
  • Increased flexibility of choosing which peripherals will be handled by each core. Since it is software defined, future changes can be made without changing hardware design.

More information on developing applications for HMP-based processors are available at these two articles:

Toradex, Antimicro and The Qt Company collaboratively built a robot showcasing this concept. The robot – named TAQ – is an inverted pendulum balancing robot designed with the Toradex Computer on Module Colibri iMX7. The user interface is built upon Linux with the QT framework running on the Cortex-A7 and the balancing/motor control is deployed on the Cortex-M4. Inter-core communication is used to remote control the robot and animate its face as seen in the short video below.

Real-Time Linux

The second approach we present in this article is software related. Linux is not a real-time operating system, but there are some initiatives which have greatly improved the determinism and timeliness of Linux. One of these efforts is the Real-Time Linux project. Real-Time Linux is a series of patches (PREEMPT_RT) aimed at adding new preemption options to the Linux Kernel along with other features and tools to improve its suitability for real-time tasks. You can find documentation on applying the PREEMPT_RT patch to the Linux kernel and developing applications for it at the official Real-Time Linux Wiki (formerly here).

We did some tests using the PREEMPT_RT patches on a Colibri iMX6DL to exemplify the improvement in real-time performance. The documentation on preparing the Toradex Linux image to deploy the PREEMPT_RT patch is available at this link. We developed a simple application which toggles a GPIO at a 2.5KHz (200µs High / 200µs Low). The GPIO output is connected to a scope where we measure the resulting square wave and evaluate the real output timings. The histograms below show the comparison between the tests on a standard Linux kernel configured for Voluntary Preemption (top) and a PREEMPT_RT patched Linux kernel configured for Real-time Preemption (bottom). The x-axis represents the period of the square wave sample and the y-axis represents the number of samples which measured with such a period. The table below the chart presents the worst and average data.

Click to Enlarge

Figure 2: Histogram of the square wave generated using the standard Kernel (top) and Preempt-RT kernel (bottom) – Click to Enlarge

Description

Samples

Smallest (µs)

Worst Case for 99% of  Samples (µs)

Worst Case (µs)

Median (µs)

Average (µs)

Default Kernel

694,780

36

415

4,635

400

400

PREEMPT_RT Kernel

683,593

369

407

431

400

400

Table 1: Comparison between Default Kernel and real-time Kernel when generating a square wave.

An example software system using the PREEMP_RT patch is provided by Codesys Solutions. They rely on the Real-Time Linux kernel, together with the OSADL (Open Source Automation Development Lab), to deploy their software PLC solution which is already widespread throughout the automation industry across thousands of devices. The video below presents the solution running on a Apalis iMX6Q.

Xenomai

Xenomai is another popular framework to make Linux a real-time system. Xenomai achieves this by adding a co-kernel to the Linux kernel. The co-kernel will handle time-critical operations and will have higher priority than the standard kernel. To use the real-time capabilities of Xenomai the real-time APIs (aka libcobalt) must be used to interface user-space applications with the Cobalt core, which is responsible for ensuring real-time performance.

dual-core-xenomai-configuration

Figure 3: Dual Core Xenomai Configuration

Documentation on how to install Xenomai on your target device can be found at the Xenomai website. Additionally, there is a variety of Embedded Hardware which is known to work as indicated in the hardware reference list, which includes the whole NXP i.MX SoC series.

To validate the use of Xenomai on the i.MX6 SoC we also developed a simple experiment. The target device was the Colibri iMX6DL by Toradex. We ran the same test approach as described above for the Real-Time Linux extension. Some parts of the application code used to implement the test are presented below to highlight the use of Xenomai APIs.

The results comparing Xenomai against a standard Linux kernel are presented in the chart below. Once again, the real-time solution provides a clear advantage – this time with even greater distinction – over the time-response of the standard Linux kernel.

Click to Enlarge

Figure 3: Histogram of the square wave generated using the standard Kernel (top) and Xenomai (bottom) – Click to Enlarge

Description

Samples

Smaller (µs)

Worst Case for 99% of Samples (µs)

Worst Case (µs)

Median (µs)

Average (µs)

Default Kernel

694,780

36

415

4,635

400

400

Xenomai Implementation

1,323,521

386

402

414

400

400

Table 2: Comparison between Default Kernel and Xenomai implementation when generating a square wave.

Conclusion

This article presented a brief overview of some solutions available to develop real-time systems on application processors running Linux as the target operating system. This is a starting point for developers who are aiming to use microprocessors and are concerned about real-time control and determinism.

We presented one hardware-based approach, using Heterogeneous Multicore Processing SoCs and two software based approaches namely: Linux-RT Patch and Xenomai. The results presented do not intend to compare operating systems or real-time techniques. Each of them has strong and weak points and may be more or less suitable depending on the use case.

The primary takeaway is that several feasible solutions exist for utilizing Linux with application processors in reliable real-time applications.

ARM Unveils Cortex-R52 ARMv8-R CPU Core for Safety-Critical Systems

September 20th, 2016 1 comment

ARM has introduced their very first ARMv8-R real-time 32-bit CPU core with Cortex-R52 designed for safety-critical applications in the automotive, industrial and health-care markets. It has been designed to address higher workloads with increased performance (up to 35%) compared to Cortex-R5 processor.

Click to Enlarge

Click to Enlarge

The processor should be used in systems capable of fulfilling IEC 61508 SIL 3 and ISO 26262 ASIL D functional safety requirements. ARM explains the new processor address both random errors for example bit flipping from radiation, and systemic errors more related to software or design faults.

functional-safety-random-systematic-faults

The latter can be addresses with the right development processes, including following aforementioned functional safety standards, but random errors require some extra hardware features such as ECC memory, or dual core lock step processors, where instructions are run on two processors simultaneously and results compared.

Normally, the whole software stack must be validated and certified on safety-critical systems, even for part of the code that may not be safety-critical. This is a time-consuming and costly endeavor however, and as software becomes ever more complex becomes an issue. So Cortex R52 cores also implement a Level 2 MPU running monitor or hypervisor software, which can help separating safety code, critical safety code and non-safety code.

arm-processor-real-time-coreCortex-R52 cores would typically be used in conjunction with Cortex-A cores running non-safety code, and offering higher performance, throughput, and more peripherals. Some current processors featuring Cortex-Rxx cores include Xilinx Zynq UltraScale+ MPSoC (Cortex-R5), and Renesas R-Car H3 automotive SoC (Cortex-R7).

You may want to visit ARM Cortex-R52 product page for a few more details.

Embedded Linux Conference Europe 2014 Schedule – IoT, ARM vs x86, Optimization, Power Management, Debugging…

August 21st, 2014 2 comments

The Embedded Linux Conference Europe (ELC 2014), CloudOpen, and LinuxCon Europe will jointly take place at the Congress Centre Düsseldorf, in Germany on October 13 – 15, 2014. The 3-day events will consists of keynotes, presentations, and tutorials. Each day will open with two or three keynotes by speakers including  Jim Zemlin (Executive Director, Linux Foundation), and Jono Bacon (XPRIZE), followed by presentation and tutorials. There will be 45 presentations for ELCE, 58 for LinuxCon, and 47 for CloudOpen, I’ll make a virtual schedule with a few sessions part of the Embedded Linux Conference Europe “track”.

ELCE_2014

Monday, October 13

When faced with a performance problem, the initial steps towards a solution include identifying the sections of code responsible and the precise reasons they are time-consuming. To this end, the ‘perf’ profiling tools provide valuable insight into the characteristics of a program. The presentation will show, using real-world examples, how the ‘perf’ tools can be used to pinpoint the parts of a program in need of optimization.

It’s not uncommon to produce embedded Linux based devices that end up with long and inconvenient boot times – yet eliminating boot time delays can be difficult and time consuming. Furthermore once a minimal boot time has been achieved it’s often just as difficult to maintain it through subsequent software development.

In this presentation, Andrew unfolds 12 keys lessons learned in his experience of boot time reduction. These lessons provide an insight into the common causes of boot time delays, why they are present and how they can be overcome. In describing these lessons Andrew will also take you on a journey that indicates why file system benchmarks should probably be ignored (with respect to boot time reduction) and a journey that illustrates that the Linux kernel is rarely the worst offender for boot delays.

With the introduction of Bluetooth Smart (aka Low Energy), the ubiquity of Bluetooth is more and more present. Millions of devices support Bluetooth Low Energy and with Bluetooth 4.1 specification, they are ready for the Internet of Things. This presentation will give an overview of Bluetooth Low Energy, and its usage for the Internet of Things. It will also introduce 6loWPAN over Bluetooth and show the possibilities this opens for Linux.

With experience developing community based open hardware for both the ARM based PandaBoard project and the x86 based MinnowBoard project, this presentation will provide a detailed comparison of the pros and cons of each platform with highlights of what each platform can learn from the other. Not only limited to the hardware aspect of the platforms, but also discuss community, software, corporate and general embedded aspects.

For almost as long as there have been deployments of Linux, there has been someone wondering “how can I get the device started quicker?” and “how do I configure some redundancy, easily, in case something goes wrong?”. And for the longest time, the answer has been “hack this and this and that” or “hire these consultants, they have done it before”. In this presentation, Tom will show what you need to turn on and the prep work required for, getting a lot of those items out of the box in U-Boot, what the hardware (and/or ROM) needs to do, and the what works is left going forward.

Got a question, comment, gripe, praise, or other communication for the Yocto Project and/or OpenEmbedded? Or maybe you’d just like to learn more about these projects and their influence on the world of embedded Linux? Feel free to join us for an informal BoF.

Tuesday, October 14

While user experiences are increasingly moving to 3D, rendering of 2D content remains at the core of how we interact with computer applications today. Skia is an open-source project maintained by Google whose goal is to bring the best 2D graphics library to a variety of targets, from mobile to desktop and embedded. Skia is used in highly popular projects like Mozilla Firefox, the Chromium browser and Android.

This talk will introduce Skia to developers and users, giving an overview of its design, architecture and features. It will also discuss briefly how hardware acceleration improves performance of Skia in the context of new devices, form-factors and the industry shift to mobile; with focus set on Linux and Android platforms.

The 4.4 KitKat release includes the results of “Project Svelte”: a set of tweaks to the operating system to make it run more easily on devices with around 512 MB RAM. This is especially important for people working with Android Wearables and “Embedded Android”, that is, implementing Android on devices at the lower end of the Android ecosystem. A large part of the problem is knowing how much RAM is really being used. Android offers a variety of tools for the purpose: procrank, procmem, meminfo and procstats, which Chris covers in the first part of the talk. In the second part, he takes a real-world example and show the practical steps you can take to optimize memory use including tuning the size of the Dalvik heap, enabling KSM (Kernel samepage merging) and swap to zRAM.

Android has relied from its early days on the Linux kernel for sandboxing the processes it runs. Yet, the permission model presented to app developers is significantly different from the Unix permission model. What’s the relationship between those two models? How is Android’s app security framework tied to the Linux kernel’s security model? More recently, Android has started using SELinux and has been extended by SEAndroid to support similar functionality. How is SELinux used by Android and what is SEAndroid about? Furthermore, how does Android provide support for multiple users?

This talk will explore Android’s security model in great detail and explain how the functionality found in the kernel is used to isolate user processes and the SE enhancements are leveraged by Android. As we’ll see, there are quite a few moving parts in Android’s security model.

Since last year, Free Electrons has been working on supporting the SoCs from Allwinner, a Chinese SoC vendor, in the mainline kernel. These SoCs are cheap, wide-spread, backed by a strong community and, until last year, only supported by an out-of-tree kernel. Through this talk, Maxime will share the status of this effort: the status a year ago, what solutions were in place, where we are currently, and what to expect from the future. He will also focus on the community around these SoCs, the work that is done there, etc.

Enlightenment Foundation Library is a set of libraries designed to use the full potential of any hardware to do great UI. It has been designed with the embedded devices in mind, but it is a desktop class toolkit. Being done in C, it is providing a stable API/ABI, high efficiency, low memory and low battery usage for all kind of Linux devices. Enabling development of modern UI adapted to any hardware that run Linux. These are the reason why Samsung uses it in its Tizen devices. This talk, after a short overview of what this libraries cover, will focus on this year improvement, and where it is heading. It will also be an opportunity to learn about project around EFL that will help people develop product with it. And it would also be a good opportunity to see where EFL are used with some real use case.

Wednesday, October 15

A major issue the community faces is the lack of power measurement (PM) instrumentation, coupled with poor integration: development boards not designed for it, expensive high-precision lab equipment not accessible to hobbyists (plus limited Linux support), limited low-cost solutions (precision, sampling rate) to monitor high-performance SoC (System On Chips) platforms (e.g. smartphones, tablets, IoT, …). After a brief introduction to the problematic (PM techniques, sense resistor / ADC selection, …) and a comparative study of existing solutions, this presentation will focus on a new upcoming initiative to close these gaps and bring a full-blown multi-channel but low-cost power (and temperature) measurement equipment to the community, including the definition of an open standard PM connector. After having covered motivations, challenges, key decisions, a live demo will close the talk.

In 2013, at the Embedded Linux Conference in Europe in Edinburgh, there was a race between a dog and a blimp. It was said that despite the dogs win, that the blimp had participated in the miracle of flight. In 2014, John wants to show that the brains of that dog can be transplanted and that it too, can participate in the miracle of flight. The talk is mainly targeting taking an off the shelf embedded platform, Minnowboard Max, and it’s use in UAVs, specifically quad-copters. With the ability to do real time computer vision, as well as various GPIO capabilities he will explore the directions that significantly more autonomous UAVs can take with Linux and embedded platforms using, mostly, off the shelf components.

There have been many presentations on what a device tree looks like and how to create a device tree. This talk instead examines how the Linux kernel uses a device tree. Topics include the kernel device tree framework, device creation, resource allocation, driver binding, and connecting objects. Troubleshooting will consider initialization, allocation, and binding ordering; kernel configuration; and driver problems.

Providing real-time capabilities to a general purpose operating system is an outstanding technical problem, and Linux Preempt-RT has been developed for 10 years for this goal. In this presentation, Jim proposes a lightweight open source para-virtualization layer, called “rtmux”, using resource-multiplexing techniques to provide a highly deterministic RT environment for Linux/ARM. Typically, less than 500 lines modification against Linux kernel are required to enable rtmux accompanied by POSIX/PSE51 compatible runtime.

During the last 2.5 years, a team of engineers at Free Electrons has been involved in mainlining the support for several ARM processors from Marvell, converting the not-so-great vendor-specific BSP into mainline quality code progressively merged upstream. This effort of several hundreds working days, has led to the integration of hundreds of patches in the kernel. Through this talk, Thomas will share some lessons learned regarding this mainlining effort, which could be useful to other engineers involved in ARM SoC support, as well as detail the steps Free Electrons engineers have gone through, the mistakes made and how they’ve been solved, as well as their overall experience on this project.

To make your own schedule matching your interests, you can check out the events’ program.

To attend the conference, you can register online.

The fees are listed as follows:

  • All-access Registration Fee – $600 until August 22 (tomorrow), $750 until October 2, and $850 afterwards
  • Attendee Networking Pass Registration – No access to conference sessions. $250 until August 22, $300 afterwards.
  • Student Registration Fee – $200 (valid student id required).
  • Registration Discount Scholar – $300. For active open source community members who can’t be sponsored by their company. .

Fees are significantly higher than last year, because there are only all-in-one (ELCE, CloudOpen and LinuxCon )options, and you can’t simply register to one single event.

Understanding PREEMPT_RT (The Real-Time Patch) – ELCE 2012

January 16th, 2013 No comments

Steven Rostedt, working at Red Hat, talks about Real-Time Linux at the Embedded Linux Conference Europe, in Spain on November 6, 2012.

Abstract:

The real-time patch (which provides CONFIG_PREEMPT_RT), has been around since 2005. Started by Ingo Molnar and maintained by Thomas Gleixner and several others, it has grown from a hobby RTOS into a very serious contender. Several distributions (Red Hat, SuSE, Debian, Ubuntu) supply a kernel version that includes this patch. The embedded world has started adding the -rt patch to their own devices that they ship. But do the embedded developers understand what the -rt patch supplies? Programming for real time, and especially when writing kernel code requires special knowledge to avoid real time traps. This talk will explain what the real time patch provides and special programming tips that will ensure embedded developers will get the best from their devices.

Real-Time Linux Option in Make menuconfig: "Fully Preempt Linux Kernel (RT)"

Real-Time Linux Option in Make menuconfig: “Fully Preemptible Kernel (RT)”

He goes through the following key points during the presentation:

  • Real-time OS definition – Deterministic, does not mean fast (but still nice), meet deadlines.
  • Goal of PREEMP_RT – 100% Preemptible kernel and quick reaction
  • Different levels of preemption in Linux:
    • No preemption – Do as most possible with as little scheduling overhead. Use for server in Linux 2.4
    • Voluntary preemption – Schedule only at “preemption points”
    • Preemptible Kernel – CONFIG_PREEMPT. Preempt anywhere except within spin_locks
    • Preemptible Kernel (Basic RT) – For debugging, it will most probably go away…
    • Fully Preemptible Kernel – PREEMPT_RT_FULL. Preempts everywhere except from preempt_disable and interrupts disabled.
  • Details of PREEMPT_RT in the Linux kernel – priorities, spin_locks, interrupts, threaded interrupts, etc…

You can also download the slides for this tutorial/presentation. You may also want to access the source code via the Git repo, or get the PREEMPT_RT patches at http://www.kernel.org/pub/linux/kernel/projects/rt/. Full details cane be found on the Real-Time Linux Wiki.