Create Minimal Debian Upstream Images with Debos and Armbian

[Update June 28: Post updated with correct procedure thanks to Collabora help]

Armbian provides lightweight Debian or Ubuntu images for various Arm Linux SBC, and over the years has become the recommended source for stable firmware images for boards part of Orange Pi & Banana Pi families, and others.

Uncompressed images are still over 1GB and come with Armbian-specific tools, kernel and bootloader. If you’d like to leverage Armbian images, but instead create a Debian upstream image with only the packages you intend to use, Collabora explains how to do just that with Orange Pi Zero +2 H5 and Libre Computer AML-S905X-CC (aka Le Potato) boards using Debos Debian OS builder.

Debos Armbian Debian Orange Pi Zero Plus 2 H5

I’ve decided to give a try at the instructions for Orange Pi Zero Plus2 H5 in my laptop running Ubuntu 18.04 to better understand how this all works.

I’ll assume you’ve already installed Docker, and made sure you’ve got it working as a non-root user, so we can install Debos as follows:


Configuration for Debos is made through a YAML files. Collabora has already provided such file for Orange Pi Zero 2+ board: orangepi0p2.yaml. You’ll need to modify it to your version of Armbian as I did below. Note getting the latest version number for Armbian is critical, or the script will fail with something like:


I’ve also added comments explaining a bit more about each sections:


The next command requires KVM for QEMU, and we should run it as normal user:


Logout and login again. Now in theory this command should create  our image:


But sadly, I’ve run into permissions errors, I’ve been unable to fix after a couple of hours:


My username is part of kvm and docker groups:


and the KVM modules are indeed loaded:


I got stuck here, but with Frédéric Danis’ help (see comments below), I could solve the permission issue:


The /recipes/scripts directory is however missing, and that’s because I wrongly assume debos would just need to be feed the yaml file. Instead we also need to get scripts and overlays directories from gitlab, and copy our updated yaml file into debos-armbian2debian/1-ArmbianKernel directory:


Now we can enter the directory, and run the previous command to generate the minimal image:


After a few minutes, the image should be generated:


You’ll find it in the working directory:


Now you can flash the image to your SD card as follows:

Support CNX Software - Donate via PayPal or become a Patron on Patreon

19
Leave a Reply

avatar
5 Comment threads
14 Thread replies
2 Followers
 
Most reacted comment
Hottest comment thread
9 Comment authors
ChasXFrédéric DanisantonioJean-Luc Aufranc (CNXSoft)Andrej Shadura Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
tkaiser
Guest
tkaiser

> Uncompressed images are still over 1GB and come with Armbian-specific tools, kernel and bootloader.

To save a lot of space for the uncompressed image with Armbian simply choose btrfs instead of ext4 at image creation time. You can also save additional space by using Armbian’s customization features and skip most of if not all packages referenced in the $PACKAGE_LIST_ADDITIONAL variable.

And by skipping ‘Armbian-specific tools’ you choose both lower performance and less reliability. I really wonder whether those people obsessed by small image sizes and ‘Debian upstream’ ever did performance comparisons or checked what Armbian does to increase life expectancy of the install media…

dimtass
Guest
dimtass

I consider armbian to be the Ubuntu of the embedded SoCs. Is the center of the distros universe for many reasons. Nevertheless, I like to see other alternatives from time to time, because the more the better in terms of choices for different use cases. Also people learn by doing those things. So I’m glad seeing new stuff and people try to do whatever they like to do.

tkaiser
Guest
tkaiser

> I consider armbian to be the Ubuntu of the embedded SoCs. Is the center of the distros universe for many reasons.

I couldn’t agree less since for me Armbian is not even a distro 🙂

It’s a flexible build system able to generate bootable Linux images from scratch in various flavors: 3 x Debian and 3 x Ubuntu. Majority of work deals with collecting and consolidating kernel patches (and as such Armbian is often months or even years ahead of ‘upstream Debian’ wrt hardware support for vast majority of interesting boards). The other important part is SBC specific tweaks — see below. Other than that Armbian is either pure Debian or pure Ubuntu.

> Also people learn by doing those things

Sure, learning is important. But I would try to focus on the important stuff and that’s:

Most SD cards and especially those with small capacity suck when it’s about random IO. They were made for sequential transfers but perform poorly when a huge amount of data has to be written in a random fashion. Installing/updating packages is exactly this: almost pure random I/O. As such I fail to understand the desire to generate ‘minimal’ SBC images at least when it’s about general purpose distros like Debian/Ubuntu.

Also important: ‘Write amplification’ when writing various small chunks of data on flash storage like SD cards is magnitudes higher compared to writing a whole image containing all the necessary stuff in a sequential fashion. Generating a ‘minimal’ image and afterwards installing the needed packages will not only take way longer but also increases wear out on the SD card drastically. The relevant learning here is: if you love your SD card (or flash storage in general) use a flexible build system and add all the packages to a reasonably large SD card image which then gets written sequentially in a single step.

How to deal with low memory situations and horribly low performing storage? Avoid swap on SD card of course. After a lot of research on ZRAM this is default on all Armbian images now. Swap to compressed block devices in RAM performs awesome and causes no harm to SD card. The whole stuff is fully configurable in Armbian and while defaults are rather conservative even a massive ZRAM overcommitment can be configured and works beautifully (taks that need 3GB RAM can run on 1GB boards with Armbian with a slight performance decrease though). What’s there with ‘upstream Debian’? Swap on SD card or nothing.

I’ve already mentioned ‘Write amplification’. A logfile where a few bytes get written to every few seconds will result in an awful lot of writes at the flash layer below (likely hundred times or even more). That’s what armbian-ramlog is for, keeping logs in RAM and writing them all in a batch to storage every hour and as such resulting in magnitudes less writes at the flash layer. This combined with other measures like increased commit interval for filesystem writes (keeping amount of ‘dirty pages’ high) increases life expectancy of SD cards dramatically.

Same happens on Armbian desktop images where browser caches are located in RAM and ‘profile sync daemon’ is enabled for every local user resulting in browser ‘profiles’ being kept in RAM since otherwise ‘surfing the web’ will be both slow and also drastically increases wear on SD card (same for eMMC — we have several reports of worn out eMMC on Linux Desktop installs, and SD cards dying is more rule than exception with this use case)

Speaking about performance: isn’t it great that with majority of kernels on ARM IRQ/SMP/HMP affinity is broken? If all interrupts are processed on cpu0 and CPU intensive tasks also sit there then with high system load this CPU core can become a nice bottleneck slowing everything down (especially true on big.LITTLE designs where cpu0 is almost always a little core). Distributing the IRQ processing on different CPU cores is one of the Armbian tweaks (unfortunately nobody there caring any more of such stuff in the meantime).

And then part of ‘Armbian-specific tools’ is also something allowing to diagnose/report problems: armbianmonitor. IMO prerequisit in support situations to get a clue what’s going on on a remote SBC where someone struggles with something.

IMO that’s the important learnings (not related to Armbian ‘as distro’ but just stuff Armbian partially focuses/focused on and implements). Debootstrapping Debian in yet another variant can also be fun but if the outcome is something performing a lot lower and sending SD cards to the bin more frequently… then you focused on the wrong goals IMO.

SBC are different than ‘real PC’. Not addressing this is IMO just an excercise in ignorance. 🙂

dim
Guest
dim

You’re right! I’ve miss-used the word distro. Of course it’s a build system. I do the same mistake often for SBC and BSP.

Since I’m building quite a lot of distros (I mean proper distros this time :p) for different architectures like ARM and x86, I’ve had to build distros that were a few MBs in size that the only thinks they did was just to run a single process in the initramfs, up to distros of 10ths of GBs that were running a huge application framework on top. This is why I’ve said there’s room for everything. Whatever the client needs. And sometimes they ask for weird stuff.

Also things like RAM logs and storage buffers are pretty much the standard in most of the projects I’ve worked with (using eMMC though, not many projects had SD cards). Also the affinity or the rtirq for preempt-rt kernels is also something I’m using a lot since I’ve mostly worked on audio devices for the last year. I haven’t used ZRAM yet, because I didn’t had to, but I would like to have a project at some point to use it! But in my case these are custom distros for custom SBCs.

It’s nice that Armbian has all these things! I wasn’t aware of those tools in the image, I’ll have a look to compare. Anyway, it’s pretty obvious that is made from people that are aware of the embedded domain and its needs and it’s not just a naive port from the desktop to the embedded. As I’ve said is my favorite to use when I want to test something on a SoC that is supported and I’m also using a lot of patches from there when I need to work with that SoC.

Anyway, I like seeing new things whatever that is and I like when I have time to look at it and be able to keep even 1 thing from the whole thing.

tkaiser
Guest
tkaiser

> Armbian … made from people that are aware of the embedded domain

Don’t think so. Armbian’s origins are ‘energy efficient and inexpensive server stuff’… on ARM since ~5 years ago something like a Cubieboard, Cubietruck, CuBox i4 or a BananaPi seemed like an ideal choice for small servers due to ‘native SATA’ and so on while x86 offers were way more expensive and wasted a lot more energy.

Project matured, in early 2016 we started to support H3 boards (most probably the biggest mistake in the project’s history) but still the focus was on server, NAS, now also Linux as desktop and shortly after also this retro gaming crap. I believe ’embedded’ with its own constraints was never a consideration since why would an embedded project need an Ubuntu or Debian userland or an installation size of around 1 GB?

The ‘collecting and consolidating kernel patches’ approach Armbian is a lot about provides a great benefit for embedded projects since the mainlining process on ARM takes an awful lot of time and once a certain hardware is fully supported by mainline kernel it’s likely already EoL. Here I’m with you that Armbian might provide some benefits for the embedded domain (providing a kernel with hardware support for a specific SoC/board 6 to 18 months prior to ‘being mainlined’) but other than that I don’t think there’s much focus on this. And the real heroes here aren’t the Armbian guys but those people contributing to mainline Linux kernel and providing their efforts in a usable way (thinking of Icenowy, jernej, Ondrej, wens — just to name few of the sunxi guys — and countless others)

It’s dealing with the limitations. I’m a server guy for example and learned a lot the last few years dealing with these small beasts. One direct benefit is being able to cramp 3 times more VMs in a virtualization host compared to before 🙂

And a side product is improving things on ARM and trying to make stuff like ZRAM a default on Linux boxes with small memory footprint (hey, even Windows 10 does something like this today!). Not sure where this will end though since currently nobody from ‘team Armbian’ takes care about such issues any more. The sole focus of the project is adding more and more hardware to the build system to ensure it’s impossible to provide proper support. When moving to newer kernel versions for example performance often decreases but nobody cares about.

antonio
Guest
antonio

why h3 boards support was a mistake?

Tony mac
Guest
Tony mac

As tkaiser pointed out, if you want to get this “into the bushes” on making your own image, the Armbian build system supports doing just about anything you could hope for. And, skipping the Armbian utilities/bsp packages would be disastrous on several boards, and at the very least ill-advised on the rest.

Gustavo Padovan
Guest

Hi, I am Gustavo Padovan, from Collabora. Thanks for trying this out, most of the time our people are using Debian to run debos (I usually do it from Fedora). This issue is new to us, can you confirm that your user has access to /dev/kvm? A ‘ls /dev/kvm’ will tell us that.
I’m trying to understand where in the stack this bug is so I can work with the team here to fix this if it happens to be in debos itself. I’ll be happy if you can check that for us.

Thansks,

Gustavo

Andrej Shadura
Guest
Andrej Shadura

Hi,

I’m not sure how exactly debos works in Docker, but to get it going on your Ubuntu you need to create empty files at the locations it expects (i.e. kernel modules). Ubuntu has those modules compiled into the kernel while Debian ships them as modules, and it seems debos at the moment verifies the file is there before doing a modprobe call.

Andrej Shadura
Guest
Andrej Shadura

I have to clarify this only applies to running debos outside of the docker image and directly in the host Ubuntu system.

ChasX
Guest
ChasX

Recently I took a different approach to accomplish more or less the same to get Ubuntu 18.04 minimal onto an Allwinner board. Instead of running a builder I just downloaded the ARM (netboot) installation initrd from Ubuntu and ran it using qemu emulating the full ARM processor itself. It is quite slow to run the installer this way but in the end you’ll have the same Ubuntu installation as you’d have running the installer on general i386/AMD64 architectures which I find quite favorable because the resulting base installation is exactly as I have on my PC.

I used ‘https://gist.github.com/takeshixx/686a4b5e057deff7892913bf69bcb85a’ as a reference to get things up and running. Downloaded a kernel and the right initrd for the installer.

When the installer was done I compiled u-boot for the Allwinner board and also compiled the most recent 4.19 mainline kernel without modules but with cherry-picked drivers that I needed. Put u-boot, the kernel and the installation on an SD card. It works really well.

I think you can also run the original Debian installer inside ‘qemu-system-arm’ to get a base Debian installation for your Allwinner board,

EDIT: just tried it. Downloaded ‘vmlinuz’ and ‘initrd.gz’ from http://ftp.nl.debian.org/debian/dists/stretch/main/installer-armhf/current/images/netboot/ and then ran (on my amd64 Ubuntu 18.04 host):

dd if=/dev/zero of=disk.img bs=1M count=4096
qemu-system-arm -kernel vmlinuz -initrd initrd.gz -append “root=/dev/ram” -no-reboot -nographic -m 1024 -M virt -hda disk.img

You can go through the installer this way up to the point where it cannot find the virtio network card. The ubuntu installer did not have that problem.