How-Tos & Training Materials
This page contains information and links for embedded Linux and Android development, and focuses on the “system” aspect (low level, OS configuration), so you won’t find that much information about application development, and many concepts discussed in this page can also be applied to “big Linux” development. Although the information provided is supposed to be target independent, some materials are board specific if it can help understanding more general ideas.
- Setting up an Embedded Linux Development Machine
- Configuration Management: Version Control, Bug Tracking and Continuous Integration
- Developing with Virtual Machines
- Emulators and Virtual Hardware Platforms
- ARM Linux development
- Linux Kernel Development
- Embedded Android Development
- Training Materials about Embedded Linux and Android
- Common Software Packages for Embedded Systems
- ARM Programming
- Fixing Common Compile Error Messages
- Programming Books
- Software QA
- Miscellaneous Tips and How-Tos
- Where and How to Ask Questions
Setting up an Embedded Linux Development Machine – Top
Embedded Linux development is usually done in a Linux machine, so you’ll have to set it up to exchange files between the build machine and the target boards, debug the target board, install a toolchain to cross-compile software, etc…. If you are new to Linux, Ubuntu is a good distribution for ARM Linux development. To get started with Ubuntu 12.04, you may want to download “Getting Started with Ubuntu 12.04“, a free 143-page guide.
Once you have installed Linux in your machine, the first thing you need to do is to install a cross toolchain which is often provided by the Silicon manufacturer, but there are now toochains part of common Linux Distribution such as:
- Linaro GCC Toolchain (ARM) – Originally only for Ubuntu, but it’s now available as a tarball, so any Linux distribution can use this toolchain, and there is also a Windows version. Here’s how to install Linaro ARM gcc in Ubuntu.
- Emdebian (ARM, A64, M68k, mips, mipsel, powerpc, s390 and sparc) – Another toolchain available as a binary package in Debian. This link explains how to install ARM Emdebian toolchain.
- GCC on Redhat/Fedora (ARM) – This cross-toolchain can be installed in Fedora and RedHat using yum.
- Sourcery CodeBench Lite Edition is another cross-toolchain widely use for embedded Linux development.
The Yocto Project and Buildroot are tools to generate a full Linux distribution for embedded systems, and they will also generate a cross-toolchain for you. I shortly address those in ARM Linux development rootfs section below.
So now you’ve got a toolchain installed on your machine, you need to setup a method to exchange files between the target board and your machine via the network in order to speed up development. There are several server available to achieve this feat:
- NFS – NFS stands for Network File System, and in my experience, is the most commonly use to exchange files between the build machine and the target board. After you build your app, you copy it in the NFS server, and run it from there in the target board. Nice and easy. See how you can install NFS in Ubuntu.
- TFTP – The Transfer File Transport Protocol is usually used to load files in the bootloader, it can be used to load a system image (e.g. ramdisk) via the network either to debug the system (kernel + rootfs) or to perform a network boot in order to save on BOM cost (e.g. no Flash needed) and maintenance, as files stored in a central server may be easier to maintain that accessing individual files. In Ubuntu, you can use apt-get install atftpd or apt-get install tftpd to install a TFTP server.
- CIFS/SAMBA – Aka “Windows Share”, this is not really used unless you also need to use Windows. Here’s how to configure a samba server in Linux.
You’ll also need to install some other tools such as a version control client (git, svn, cvs…), debuggers, and more. But more on that below. You’ll most probably use a serial console to access the board using minicom or screen (Linux), or hyperterminal or teraterm (Windows).
Configuration Management: Version Control, Bug Tracking and Continuous Integration – Top
This section is not specific to embedded software, but you’ll need a version control system to manage your code, especially if you work as a team, but it’s also useful even if you work by yourself on a project since you can easily keep track of changes. There are several version control systems available, CVS is the oldest, and less and less used, SVN is more recent and still used in some projects, but the most used solution is now Git.
You can install Git yourself on Linux or Windows, or use online services such as Gitorious or Github. Online services are free for public repository and small teams, but you need to pay to host closed source projects.
For bug tracking, you can install tools such as Bugzilla or Trac. Bugzilla is the most widely used, and Trac is actually more than just a Bug tracker, as it is used to manage complete projects and integrates with svn. There is a more complete list on Wikipedia.
Finally, you’ll most probably want to automatically build your code regularly, either daily/nightly using scripts for example, either using continuous integration (build after each commit) tools to make sure your build complete. Such builds can also be used by the QA team, as they can indicate which build failed to pass some tests. Some interesting tools include Jenkins-CI, Buildbot or BuildHive (Jenkins-CI on Github).
Developing with Virtual Machines – Top
If you have to use Window XP/7 has your main machine, you can still do Linux development with x86 virtualization packages such as Oracle VirtualBox or VMWare Player, although if you are a team of developers, you could use a Linux server and do development via an SSH terminal window using Putty for example.
Installing Linux distributions in VirtualBox or VMWare Player is really easy, so I won’t explain how to do this here. One the thing you probably want to have is the ability to copy/paste text between the Windows host and the Linux guest OS. I’ve explained how to do this in VirtualBox in a previous post, since this is not enabled by default. The simplest solution to exchange files between the host and the guest OSes is probably to setup a network share.
Emulators and Virtual Hardware Platforms – Top
Whether you are a student with a very limited budget, or a more experienced engineer needing to work on a project before the hardware is ready, you can get started with cross-development using emulators such as qemu, which can emulate a lot of targets such as ARM, MIPS, PCC and more.
Qemu is often provided as a binary package in Linux distributions, and you could get install it using “apt-get install qemu-system” in Debian or Ubuntu for example. In some cases, you may want to use the latest qemu version, in that case you’ll have to build it. Here’s how to build qemu for ARM emulator.
Now that you’ve got qemu up-and-running, you can try it. I’ve written the instructions to emulate a Gumstix Overo board (TI OMAP3 – ARM Cortex A8) as well as ARM Versatile Express in qemu, and you could use this to get started with ARM development.
There are limits to developing with emulators however due to the low performance and the lack of peripheral support, so usually any thing that access the hardware and is graphics intensive (although OpenGL ES is now supported) cannot usually be run in qemu. The solution to this issue is the implementation of virtual hardware platforms that can run on power machines and emulate the CPU(s) and all peripherals of an SoC. I haven’t tried it myself yet, but If you want to have a go, you may want to try Impera OPVsim simulator. It’s free and there are a a few models available.
ARM has recently released fast-models for ARMv8. I’ve written instructions to use it with OpenEmbedded 64-bit ARM port.
ARM Linux development – Top
This section is specific to ARM and Linux, and I assume you’ve already setup your build machine as specified above.
You’ve usually got 3 components in an Embedded Linux systems, which are called in sequence at boot time.
- The bootloader (e.g. U-boot)
- The Linux kernel
- The root file system (rootfs)
I’ll give examples showing how to each, as the method is more as less the same:
- Building u-boot for Pandaboard (ARM Cortex A9 board)
- Building Raspberry Pi Kernel (ARM11 board)
- Building the Linux Kernel in Ubuntu (for Pandaboard). This method is very simple, but works only in Ubuntu 12.04 and is quite specific.
Once you get the bootloader and Linux kernel ready, you’ll need a rootfs that contains all userspace files required to complete the system boot. Here are some methods to get a root file system for ARM:
- Generate a custom ARM rootfs with rootstock. Only available for Ubuntu, and this has been deprecated in Ubuntu 12.04 (rootstock script is gone), but I put it here for reference, and because the new method (multiarch) does not seem to be as reliable as expected.
- Getting an ARMEL or ARMHF rootfs in Ubuntu 12.04 – You are now being asked to download available rootfs, instead of generating them with rootstock.
- Getting Debian ARM rootfs – There are basically two methods: debootstrap (example) and multistrap (more powerful and better adapted to embedded systems). Alternatively, you can also look into EmDebian Grip images.
- Generate a Linux distribution for embedded systems – Some tools will help you generate your own distribution and tools specifically for your applications such as the Yocto Project, buildroot and Open Embedded.
If you want to cross-compile ARM packages you can now use MultiArch in Ubuntu 12.04, This will download and install ARM packages (dependencies), and cross-build packages from sources for ARM targets.
Linux Kernel Development – Top
If you want to learn more about kernel development, there are some must read books:
- Linux Device Drivers, Third Edition - Available for free @ http://lwn.net/Kernel/LDD3/ for Linux 2.6 drivers and beyond. In case you prefer an hardcopy it’s also available on Amazon.
- Linux Kernel Development (3rd Edition) – Available on Amazon as paperback or kindle editions.
- Linux Kernel in a Nutshell – Available for free online.
There’s also a great list of books and how-tos about Linux kernel development on Linuxquestions.org
Once you’ve developed drivers for the kernel and/or fix a bug, you’ll want to submit a patch, but you’ll have to follow the rules in the provided link or it will be declined.
Embedded Android Development – Top
Whether you plan to develop apps for Android, or just want to tinker with the system (with ADB), you’ll need to install the Android SDK. Here’s how to install the Android SDK on Windows XP and Ubuntu 12.04
If you want to learn more about Linaro Android and how to install it on a Pandaboard, you can do so with the Linaro Android Tutorial by Zach Pfeffer, the tech lead of the Linaro Android working group (He’s always entertaining). Zach also has another very interesting presentation where he explains how to port Android to a new platform with Linaro.
If you want to learn more about Android internals, and use Android in embedded systems, check out Opersys “Android Variants, Hacks, Tricks and Resources” presentation slides (AnDevCon II) as well as their “Embedded Android Workshop” slides.
Training Materials about Embedded Linux and Android – Top
In this section, I’ll list some external resources and I (and many others) find very good to get started with embedded Linux or/and embedded Android.
Free Electrons is a French technology company offering embedded Linux consulting services and training, abd there have excellent embedded Linux training material about Linux kernel and driver development (labs.pdf), embedded Linux using IGEPv2 board (TI OMAP3) and a more than 500 pages presentation about embedded Linux system development (slides.pdf). The PDF downloads may not be the latest versions, and the latest documents are available in their repository. Here’s how to build the documentation.
Opersys is another training company – You’ll find the owner,Karim J. Yaghmour, gives presentation at nearly all Linux and Android events – that specializes in embedded software development courses. They offer their embedded Android and Linux materials for free.
The Linux foundation also provides free Linux training videos, although most are not directly related to Embedded software, there are still interesting Have a look at those free Linux tutorials.
Every year, there are events such as the Embedded Linux Conference and the Linux Conference, and videos are recorded (usually by Free Electrons mentioned above) and made available freely. I find those videos very informative, and by watching them you’ll learn many of the latest Embedded Linux, open source and Linux developments. I’ve uploaded a few to YouTube, written summaries and provided links to presentation slides if available:
- ELCE 2011 videos
- ELC 2012 videos
- ELCE 2012 videos
- Linuxcon 2011 videos
- Android Builder Summit 2012 Videos
The Embedded Linux Quick Start Guide presentation by Chris Simmons at ELCE 2010 is one of the most popular, and within three hours will walk you maybe aspect of embedded software development, with the last section being an hands-on session with NXP LPC3250 Stick (ARM9). Please note the second video is very noisy until 17:00 because of renovation being done at the time.
NPTEL has uploaded a 50-hour course about embedded systems by Dr.Santanu Chaudhury,Dept. of Electrical Engineering, IIT Delhi, on Youtube. This is not specific to embedded Linux, as it deals with all sorts of embedded systems, but it still seems very interesting.
The University of Edinburgh has made available their Embedded Systems Courses slides. Again this is not specific to Linux, although some parts are, but still interesting. Examples are using MQX RTOS and Freescale Kinetis K70.
Common Software Packages for Embedded Systems – Top
In “Big” Linux, when you want to install a web server you may think about Apache, a database you may think about Mysql, but those are generally way too resources intensive for embedded systems with limited memory and processing power.
So instead you’ll use lightweight alternatives:
- Web servers
- Databases – Berkeley DB, SQLite and more.
- NTFS support for embedded systems
ARM Programming – Top
This section is not specific to Linux and Android, but you may have to use this if you want to optimized ARM performance or the battery life of your products.
- ARM development Cortex-A programmer guide – All you need to dig down at the register level and assembler level for ARM Cortex-A processor (Cortex A5, A7, A8 A9 and A15). The latest version 3.0 is available at https://silver.arm.com/download/download.tm?pv=1296010 but requires free registration.
- ARM Neon Tutorial in C and Assembler – Presented by Doulos, another training company.
- Compile with ARM Thumb2 to reduce memory footprint and improve performance. (A bit old, but hopefully still relevant).
You can also visit ARM infocenter which a massive amount of information on ARM programming.
Fixing Common Compile Error Messages – Top
When I browse some Linux or programming forums I often read people post “help” posts about errors during the configure and make stage of a build due to missing files or packages. A few years ago, It used to be that you had to do some research to find the right software package, but now thanks to packaging tools is very easy to find the missing software package and such questions should not show up in forums. I’ll give instructions using Ubuntu/Debian apt tools:
- “fatal error: header.h: No such file or directory compilation terminated”. You can use apt-file to find the missing package.
- “checking for PACKAGE… configure: error: Package requirements (package_name) were not met: No package ‘package_name’ found”. In that case you can use “aptitude search package_name” to get a list of packages containing “package_name”. and install the relevant one.
- Finding configure option for a given package. Not an error as such, but it’s quite common that you have to cross-build a library yourself in embedded Linux as your system does not support packaging utils such as apt-get or yum, and may have to setup dependencies paths (headers and libraries) manually. A quick way to list those is to use pkg-config.
Programming Books – Top
Here are some links to programming books:
- Top 5 Most Influential Programming Books Ever as per slashdot users.
- Embedded Software Programming Books
- Free e-Book (PDF) about C programming language – Essential C
Debugging – Top
Here are some links to debugging tools, tips and how-tos to debug software in embedded systems:
- oprofile – Equivalent of gprof for embedded systems (now deprecated and replaced by perf in the latest Linux versions)
- gdb and gdb insight – gdb is the defacto debugger on Linux (userspace) and gdb insight is a graphical interface for gdb (although apparently not developed in recent years). If you’re debugging ARM (Cortex-A) platforms, you may also consider using ARM DS5 CE. kgdb is the equivalent of gdb for kernel debugging.
- Linux kernel debugging with kgdb and OpenOCD. Presentation at Linaro Connect in Q4 2011, OpenOCD support is now much better.
- strace – A debugging utility that monitor the system calls ans signals. Particularly useful to debug binary only packages.
- Check open files for a process – lsof can do this, but if your system does not include it, you can simply use the sample code provided in this link.
- How-to list dynamic libraries used by a program. This can be useful to see which libs may be missing, or if you program uses the wrong lib.
- How-to list the dynamic libraries as they are opened at runtime. This may help you locate the problematic library in case of crash.
- How to find memory leaks the easy way – This shows how to detect and find the location of memory leaks, but adding debug code. Particularly useful if your platform does not support tools such as Valgrind.
- Debugging kernel panics in uClinux – If you are new to uClinux, you’ll probably get a 100% of getting kernel panics when running your applications because of stack overflow. This explains how to work around this.
Software QA – Top
In this section, I’ll write a bit about testing and list simple techniques to improve the source code by using some tools.
About 2 years ago, I wrote a post on how to share testing tasks between developers and testers when testing embedded software. this is for reference only, because other people will have a difference point of view.
Here are several resources, developers can do to improve code quality:
- 17 Best practise to write safer C code
- Remove code duplicate with tools such as CCFinderX (for C/C++ only).
- Using static analysis tools and techniques to detect some bugs at build time.
- Software Code Review Study and Best Practices or let developers cross read each others code to detect problems early.
Finally, an interesting study found that to speed up development developers should use test automation, static analysis and virtual system prototyping/simulation (VSPS) tools, yet very few do.
Miscellaneous Tips and How-To – Top
Your application may need to support different languages, you can read the post entitled “Languages, Unicode and Charset” to learn why your app should use Unicode, how to convert different charset to unicode, and how to detect the language used in a file using tools and C code.
You should at least have basic code rules which indicate how to format your code, and many software can be setup to do that automatically. One of the rule is often to ban tabs in the code, so everybody uses (4) spaces instead of tabs to make the code well formatted for anybody. If you use vi/vim as your editor, you can set it to replace tabs by spaces automatically.
If you’re going to work on embedded Linux, you’ll have to use open source packages (one of them being the kernel), you should learn about the different open source licenses such as GPL, LGPL, Apache, MIT…
Here’s a method to redirect all output to a log file, it’s particularly useful for long builds.
You should really have documentation with your software, and you could write that separatly, but there are tools such as Doxygen which can generate the documentation automatically using variables, function names and specific comments in your source code.
Even though, with NAND flash it’s more common to use ext-2/3/4 nowadays, JFFS2 is still use in many embedded systems using NOR flash. You may need to access those images in a Linux PC to add files and/or see what’s inside. Here’s how to mount a JFFS2 image in Linux.
Where and How to Ask Questions – Top
Here’s a list of site useful to developers
- stackoverflow – This is where you should ask questions related to programming
- xda-developers forum – Excellent forum to find out how to hack mobile devices either to work with Android or Linux distributions. Information is all over of the place (due to the site popularity) and it may be difficult to find exact information, but I can tell you it’s there somewhere
- Linux Questions forums – If you’ve got a problem with Linux, this is the place to ask. They also have section related to Embedded Linux, the Linux Kernel and hardware.
- The board or silicon vendor – If you bought a development board, you should ask to your resellers, as many boards have online support. It should be the most obvious place to ask. Some companies such as Texas Instruments or Atmel have very active online communities, so you should ask in their forums to get answers from experts, instead of asking in more general websites such as Linux Questions.
If you think “Help me!”, “My board does not work!” or the grammatically incorrect ‘How do?” are good subjects for a forum post, I strongly recommend you read “How To Ask Questions The Smart Way“.
Before you ask a questions, make sure you’ve at least done some research with an online search tool such as “Google” (some people do not appear to know it exists in that case: http://lmgtfy.com/), provide an explicit title, explain what you’ve already tried yourself and give as much details as possible. If you have a long log, do not post it directly in the post, but post the relevant snippet, or if you are not sure where the problem is, copy/paste your log in pastebin.com and provide a link.
You are more likely to get an accurate answer that way, and you’ll also avoid ire from the community.
Let me know if you are more suggestions, find errors, obsolete instructions, etc.. in the comment section.