How to Sandbox an arm64 GCC on aarch64 Hardware with armv7 Userspace

CNXSoft: Guest post by Blu about setting up arm64 toolchain on 64-bit Arm hardware running a 32-bit Arm (Armv7) rootfs.

Life is short and industry progress is never fast enough in areas we care about. That’s an observation most of us are familiar with. One would think that by now most aarch64 desktops would be running arm64 environments, with multi-arch support when needed.

Alas, as of late 2019, chromeOS on aarch64 is still shipping an aarch64 kernel and an armhf userspace. And despite the fine job by the good folks at chromebrew, an aarch64 chromeOS machine in dev mode ‒ an otherwise excellent road-warrior ride, is stuck with 32-bit armhf. Is that a problem, some may ask? Yes, it is ‒ aarch64 is the objectively better arm ISA outside of MCUs, from gen-purpose code to all kinds of ISA extensions, SIMD in particular. That shows in contemporary compiler support and in the difference in quality of codegen. Particularly with the recent introduction of ILP32 ABI for aarch64 (AKA arm64_32) and Thumb2 notwithstanding (unsupported under aarch64), there won’t be a good reason one would want to build aarch32 code on an aarch64 machine soon.

Arm64 GCC Armv7 Userspace

As a long-time arm chromebook user, I have been using a sandboxed arm64 compiler on my chromebooks. A recent incident (read: author’s sloppiness) destroyed that sandbox on my daily ride while I was away from home, so I had to do it again from scratch, over a precious mobile data plan. This has led me to document the minimal set of steps required to set up the sandbox, and I’m sharing those with you. Please note, that the steps would be valid for any setup of 3.7-or-newer aarch64 kernel & armhf userspace, but in the case of chromeOS some extra constraints apply ‒ the root fs is sternly immutable to the user.

Under chromeOS dev-mode a writable fs space is available under /usr/local, so this is where we’ll set up our sandbox.


We need to choose a good source of arm64 packages ‒ some (non-rolling) distro with up-to-date package base will do. That casually means Debian or Ubuntu ‒ we go with the latter due to its better refresh rate of packages. Next we need to decide which version of the compiler we want, as that will determine what system libraries we’ll need. For personal reasons I’ve decided upon gcc 8.2.0. Here’s the list of packages one needs to download from http://ports.ubuntu.com/ubuntu-ports/pool/main/ for the gcc version of choice:


In case of difficulties locating those packages on the main Ubuntu repo, one can use the global registry http://ports.ubuntu.com/ubuntu-ports/ls-lR.gz.

Next we ‘deploy’ those packages. We can do it one by one, e.g. if we’re curious to check what’s in each package:


Or en masse:


That was the straight-forward part. The next part is less so.

An aarch64 kernel can natively run statically-linked binaries no problem. Such binaries are a rarity on desktop Linux, though ‒ most of the binaries you’ll encounter or compile yourself will be dynamically-linked. For each architecture supported by the platform, there must be a corresponding dynamic loader. On arm multi-arch nowadays those would be:


For aarch64 and armhf, respectively; on non-multi-arch there would be only one, naturally. If you inquire about any given dynamically-linked binary, you’d see those loaders reported immediately after the ELF type:


The part that reads “dynamically linked, interpreter /lib/ld-linux-armhf.so.3” tells what dynamic loader should be used to launch each binary, with path and all. As we cannot modify /lib on chromeOS, we cannot place an aarch64 loader there, so we will need to revert to other means.

As many are familiar, the binary of gcc is just a facade that invokes a bunch of actual business binaries to carry out a build. In the context of our sandbox, those are:


Now, the first two were delivered through our .deb packages, while the latter two are normally mere symlinks to the actual aarch64-linux-gnu-* binaries, but we don’t have those symlinks yet, and we don’t need them as symlinks anyway.

Each time our arm64 gcc will try to execute one of those, it will fail due to lack of system-wide loader for aarch64. So how do we solve that? Easily ‒ we manually specify the loader that should be used with any given arm64 binary we want to run. So let’s shadow those binaries by some scripts of the same names to do our bidding, while preserving the original binaries for cc1 and collect2 as .bak files at their original locations:


Eventually we want the following scripts at their corresponding locations:

  • /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/cc1:
  • /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/collect2:
  • /usr/local/root64/usr/bin/as:
  • /usr/local/root64/usr/bin/ld:

Last but not least, we need a gcc top-level script to set up some things for us (placed somewhere in PATH, ideally):

  • gcc-8.sh:

Let’s test out contraption with the following sample code:


First, with the native armhf gcc-8.2.0 compiler, courtesy of chromebrew:


Next with the arm64 gcc-8.2.0 we just set up:


Please note, that since we have not exported an arm64-aware LD_LIBRARY_PATH yet, we have to specify that on the command line, right before invoking the aarch64 loader.

Click to Enlarge

Well, that’s all, folks. I hope this comes of use to some aarch64+armhf users out there.

Support CNX Software - Donate via PayPal, become a Patron on Patreon, or buy review samples
Subscribe
Notify of
guest
23 Comments
oldest
newest most voted
dgp
dgp
1 year ago

Seems like a perfect example of why you don’t get yourself locked into an ecosystem where you can’t just install your own kernel and userland.

blu
blu
1 year ago

That’s entirely a matter of priorities. I don’t want to ‘install my own kernel and userland’ — I’d rather leave that to distro providers, like for instance google, who are quite good at what they do. The majority of chromeos dev-mode users are fine with chromeos and rich 3rd party repos like chromebrew. This is an article for those who want aarch64 on top of that, demonstrating it’s trivial to get aarch64 apps and full toolchains going on chromeos.

dgp
dgp
1 year ago

> rich 3rd party repos like chromebrew.

With a whole 1026 packages apparently.

>demonstrating it’s trivial to get aarch64 apps and full toolchains going on chromeos.

Doesn’t look trivial. i.e. in the time you’ve taken to hack your ubuntu toolchain into chromeos I could have installed all of the pre-built cross toolchains in Debian and have GCC9 for a bunch of targets along with qemu to run the binaries locally if I needed to.

This seems like “making a-rod for your own back”.

blu
blu
1 year ago

> With a whole 1026 packages apparently. *Apparently* chromebrew is not trying to be a debian repo replicant — it is driven by the needs of a handful of devs (as in chromeos *dev-mode*). So it’s only the packages that those chromeos users want that find their way into the repo. For the needs of *those* devs, it’s quite a nice repo; from my experience with the repo, it’s provided the vast majority of the packages I’ve needed so far. > Doesn’t look trivial. i.e. in the time you’ve taken to hack your ubuntu toolchain into chromeos I could have… Read more »

dgp
dgp
1 year ago

>*Apparently* chromebrew is not trying to be a debian repo replicant You said it was *rich* not me. Only having 1026 packages probably means coverage of packages there is pretty shallow.. and a quick scan through the list of packages (https://github.com/skycocker/chromebrew/tree/master/packages) doesn’t seem to suggest any richness either. A lot of it seems to be toy stuff like quakespasm. >the approach is quite trivial, yes. IMHO writing an article that’s multiple paragraphs and includes images to explain how to do something contradicts with the claim that what is described in the article is trivial. Maybe you mean “straightforward”. >It’s a… Read more »

blu
blu
1 year ago

> You said it was *rich* not me. Yes, and I’ll repeat it — it’s rich for my purposes and those of devs like me. Clearly not rich for people who deal with absolutes. > Maybe you mean “straightforward”. No, dgp, I mean trivial, as in ordinary, as in nothing extraordinary to get an arm64 package running on an armv7 userspace. If you need to translate that to ‘straightforward’ so it could fit your worldviews — be my guest, I’m not stopping you. > I hate to go all meta on you but.. And here I thought you were nothing… Read more »

dgp
dgp
1 year ago

>it’s rich for my purposes and those of devs like me. Yet you have to write a wordy description of how to get basic tools on your system. I’m surprised your boss hasn’t spotted you constantly messing with that crap, forced the company credit card into your hand and told you to just buy a macbook already. For devs like me.. just my AndroidStudio instance would make your chromebook burst into flames somewhere between hitting the build button and the projected week it would take to build. >No, dgp, I mean trivial, as in ordinary, That’s not what trivial means… Read more »

blu
blu
1 year ago

I appreciate linguistic bikeshedding and political drivel, but that about concludes our totally-not-meta discussion. If you figure out something meaningful to say on the subject, I’ll participate. Until then, enjoy your day, dgp.

dgp
dgp
1 year ago

Usually I close the tab and forget about it because this is always what you do and it’s easy to tell you’re getting to this point because you try spinning your “I’m the only *dev* in the world” (No true Scotsman) elitist nonsense, calling people kids and so on… but I just found this tab open after looking for another so lets have some more fun. >I appreciate linguistic bikeshedding Words mean things. If I tell my boss a change is trivial that does not mean that it will take a whole day to implement and require the full test… Read more »

blu
blu
1 year ago

dgp, there’s just one individual on this site I’ve called ‘a kid’.. And here’s but an example of why: > Even if trivial was redefined to ordinary that still isn’t ordinary. You have to manually put your machine into dev mode and then manually extract packages and faff around to get the bastardised packages to run. Sure, let’s “redefine” trivial: https://www.merriam-webster.com/dictionary/trivial trivial, adjective 2 : commonplace, ordinary > manually put your machine into dev mode Another trivial step, done once and forgotten about by chromeos dev-mode users.. > manually extract packages and faff around to get the bastardised packages to… Read more »

rwoodsmall
rwoodsmall
1 year ago
blu
blu
1 year ago

Nice job! I was not aware of it — I’ll check it out ASAP.

theguyuk
theguyuk
1 year ago

As a question why have Android OS boxes stayed 32 bit software while the SoC are 64 bit ?

blu
blu
1 year ago

Most phone vendors don’t want to deal with ISA transitions, and aarch32 and aarch64 are different ISAs. Also, due to (1) larger pointers in 64-bit code and (2) absence of the compressed arm ISA known as Thumb on aarch64, most arm64 binaries tend to be larger. Larger binaries, larger distros, less space for photos, mp3s and apps. Lastly, some pointer-chasing code can be slower in 64-bit again thanks to larger data records, but that is addressed by arm64_32. The natural answer to all those points above under linux is called multi-arch support — on arm that’d be aarch64 kernel and… Read more »

Diego
Diego
1 year ago

Is there an easy way figuring out if my android phone has an armhf userspace?

blu
blu
1 year ago

Not an android user here, but chances are that if you have a recent android and armv8 hw, userspace is already effectively multi-arch (multi-lib in android terms). Get an android app with a known 64-bit version, i.e. Geekbench, and see what it says on your phone.

Diego
Diego
1 year ago

Thanks blu

Well honestly I don’t care enough to go that far finding out 😉

Just out of curiosity what are you using if not android?

blu
blu
1 year ago

I was a sworn s30/40 user for many years, then firefoxos until it went under, at which point I was forced into ios. I’ve written (system-level) code for android, but I’ve never owned an android phone : )

On the subject of android, checking the timelines again, it turns out android got 64-bit multi-lib earlier than I remembered — since lolipop.

Diego
Diego
1 year ago

I also had the impression it was 64 bit from a long time but got curious.

I never was a nokia fan couldn’t wait till android is coming. It never happened, till i lost interest…

Lol then Samsung came now im with android as i cant use ios (due to allergy 😛 ) its for me the least bad of the horrifying options. No more with Samsung from along time though

dgp
dgp
1 year ago

>As a question why have Android OS boxes stayed 32 bit software while the SoC are 64 bit ? TBH I doubt it has anything to do with size and was just a timing issue. Almost all apps that ship with native binaries only shipped with armv7 and x86 and they didn’t want to make that work (i.e. ship multiple versions of all of the android system libraries, make the interactions work correctly, get AndroidStudio etc compatible) at that time and it was much easier just to continue with armv7 for the userland. Now that no one makes new armv7… Read more »

Laurent
Laurent
1 year ago

> Almost all apps that ship with native binaries only shipped with armv7 and x86

Many native Android apps lacked x86 support and that’s why Intel had to write an emulator to run ARM code on their pitiful attempt at making x86-based SoC for phones.

dgp
dgp
1 year ago

>Many native Android apps lacked x86 support That might have been the case back at android 2 but the x86 emulator has been the main development environment for a lot of people for a long time now so if you don’t have x86 libraries you’ll be annoying almost every android developer for no good reason. I don’t even think you could have a 100% on the metal native app back then. A game I worked on a good 8 or more years ago shipped with arm and x86 libraries. >and that’s why Intel had to write an emulator to run… Read more »

Advertisements