Posted on by Jean-Luc Aufranc (CNXSoft) - 23 Comments on How to Sandbox an arm64 GCC on aarch64 Hardware with armv7 Userspace

How to Sandbox an arm64 GCC on aarch64 Hardware with armv7 Userspace

CNXSoft: Guest post by Blu about setting up arm64 toolchain on 64-bit Arm hardware running a 32-bit Arm (Armv7) rootfs.

Life is short and industry progress is never fast enough in areas we care about. That’s an observation most of us are familiar with. One would think that by now most aarch64 desktops would be running arm64 environments, with multi-arch support when needed.

Alas, as of late 2019, chromeOS on aarch64 is still shipping an aarch64 kernel and an armhf userspace. And despite the fine job by the good folks at chromebrew, an aarch64 chromeOS machine in dev mode ‒ an otherwise excellent road-warrior ride, is stuck with 32-bit armhf. Is that a problem, some may ask? Yes, it is ‒ aarch64 is the objectively better arm ISA outside of MCUs, from gen-purpose code to all kinds of ISA extensions, SIMD in particular. That shows in contemporary compiler support and in the difference in quality of codegen. Particularly with the recent introduction of ILP32 ABI for aarch64 (AKA arm64_32) and Thumb2 notwithstanding (unsupported under aarch64), there won’t be a good reason one would want to build aarch32 code on an aarch64 machine soon.

Arm64 GCC Armv7 Userspace

As a long-time arm chromebook user, I have been using a sandboxed arm64 compiler on my chromebooks. A recent incident (read: author’s sloppiness) destroyed that sandbox on my daily ride while I was away from home, so I had to do it again from scratch, over a precious mobile data plan. This has led me to document the minimal set of steps required to set up the sandbox, and I’m sharing those with you. Please note, that the steps would be valid for any setup of 3.7-or-newer aarch64 kernel & armhf userspace, but in the case of chromeOS some extra constraints apply ‒ the root fs is sternly immutable to the user.

Under chromeOS dev-mode a writable fs space is available under /usr/local, so this is where we’ll set up our sandbox.


We need to choose a good source of arm64 packages ‒ some (non-rolling) distro with up-to-date package base will do. That casually means Debian or Ubuntu ‒ we go with the latter due to its better refresh rate of packages. Next we need to decide which version of the compiler we want, as that will determine what system libraries we’ll need. For personal reasons I’ve decided upon gcc 8.2.0. Here’s the list of packages one needs to download from http://ports.ubuntu.com/ubuntu-ports/pool/main/ for the gcc version of choice:


In case of difficulties locating those packages on the main Ubuntu repo, one can use the global registry http://ports.ubuntu.com/ubuntu-ports/ls-lR.gz.

Next we ‘deploy’ those packages. We can do it one by one, e.g. if we’re curious to check what’s in each package:


Or en masse:


That was the straight-forward part. The next part is less so.

An aarch64 kernel can natively run statically-linked binaries no problem. Such binaries are a rarity on desktop Linux, though ‒ most of the binaries you’ll encounter or compile yourself will be dynamically-linked. For each architecture supported by the platform, there must be a corresponding dynamic loader. On arm multi-arch nowadays those would be:


For aarch64 and armhf, respectively; on non-multi-arch there would be only one, naturally. If you inquire about any given dynamically-linked binary, you’d see those loaders reported immediately after the ELF type:


The part that reads “dynamically linked, interpreter /lib/ld-linux-armhf.so.3” tells what dynamic loader should be used to launch each binary, with path and all. As we cannot modify /lib on chromeOS, we cannot place an aarch64 loader there, so we will need to revert to other means.

As many are familiar, the binary of gcc is just a facade that invokes a bunch of actual business binaries to carry out a build. In the context of our sandbox, those are:


Now, the first two were delivered through our .deb packages, while the latter two are normally mere symlinks to the actual aarch64-linux-gnu-* binaries, but we don’t have those symlinks yet, and we don’t need them as symlinks anyway.

Each time our arm64 gcc will try to execute one of those, it will fail due to lack of system-wide loader for aarch64. So how do we solve that? Easily ‒ we manually specify the loader that should be used with any given arm64 binary we want to run. So let’s shadow those binaries by some scripts of the same names to do our bidding, while preserving the original binaries for cc1 and collect2 as .bak files at their original locations:


Eventually we want the following scripts at their corresponding locations:

  • /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/cc1:
  • /usr/local/root64/usr/lib/gcc/aarch64-linux-gnu/8/collect2:
  • /usr/local/root64/usr/bin/as:
  • /usr/local/root64/usr/bin/ld:

Last but not least, we need a gcc top-level script to set up some things for us (placed somewhere in PATH, ideally):

  • gcc-8.sh:

Let’s test out contraption with the following sample code:


First, with the native armhf gcc-8.2.0 compiler, courtesy of chromebrew:


Next with the arm64 gcc-8.2.0 we just set up:


Please note, that since we have not exported an arm64-aware LD_LIBRARY_PATH yet, we have to specify that on the command line, right before invoking the aarch64 loader.

Chromebook Arm64 gcc
Click to Enlarge

Well, that’s all, folks. I hope this comes of use to some aarch64+armhf users out there.

Share this:
FacebookTwitterHacker NewsSlashdotRedditLinkedInPinterestFlipboardMeWeLineEmailShare

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK Pi 4C Plus

23 Replies to “How to Sandbox an arm64 GCC on aarch64 Hardware with armv7 Userspace”

  1. Seems like a perfect example of why you don’t get yourself locked into an ecosystem where you can’t just install your own kernel and userland.

    Reply

    1. That’s entirely a matter of priorities. I don’t want to ‘install my own kernel and userland’ — I’d rather leave that to distro providers, like for instance google, who are quite good at what they do. The majority of chromeos dev-mode users are fine with chromeos and rich 3rd party repos like chromebrew. This is an article for those who want aarch64 on top of that, demonstrating it’s trivial to get aarch64 apps and full toolchains going on chromeos.

      Reply

      1. > rich 3rd party repos like chromebrew.

        With a whole 1026 packages apparently.

        >demonstrating it’s trivial to get aarch64 apps and full toolchains going on chromeos.

        Doesn’t look trivial. i.e. in the time you’ve taken to hack your ubuntu toolchain into chromeos I could have installed all of the pre-built cross toolchains in Debian and have GCC9 for a bunch of targets along with qemu to run the binaries locally if I needed to.

        This seems like “making a-rod for your own back”.

        Reply

        1. > With a whole 1026 packages apparently.

          *Apparently* chromebrew is not trying to be a debian repo replicant — it is driven by the needs of a handful of devs (as in chromeos *dev-mode*). So it’s only the packages that those chromeos users want that find their way into the repo. For the needs of *those* devs, it’s quite a nice repo; from my experience with the repo, it’s provided the vast majority of the packages I’ve needed so far.

          > Doesn’t look trivial. i.e. in the time you’ve taken to hack your ubuntu toolchain into chromeos I could have installed all of the pre-built cross toolchains in Debian and have GCC9 for a bunch of targets along with qemu to run the binaries locally if I needed to.

          I thought I wouldn’t need to explain that, but since some people will do their best to miss the apparent: this is a not meant to substitute a package management system. For being ‘not a package manager’, the approach is quite trivial, yes.

          And of course it scratches my back. I said as much as in the third paragraph. It’s a primer of how to get aarch64 binaries on an aarch64-backed armv7 userspace, based on what binaries *I* needed *this* time. I thought that much was apparent, alas.

          Reply

          1. >*Apparently* chromebrew is not trying to be a debian repo replicant

            You said it was *rich* not me. Only having 1026 packages probably means coverage of packages there is pretty shallow.. and a quick scan through the list of packages (https://github.com/skycocker/chromebrew/tree/master/packages) doesn’t seem to suggest any richness either. A lot of it seems to be toy stuff like quakespasm.

            >the approach is quite trivial, yes.

            IMHO writing an article that’s multiple paragraphs and includes images to explain how to do something contradicts with the claim that what is described in the article is trivial. Maybe you mean “straightforward”.

            >It’s a primer of how to get aarch64 binaries on an aarch64-backed

            I hate to go all meta on you but you realise I wasn’t talking about your article anymore right? I was following up on your comment that was trying to suggest that locking yourself into an environment that is fighting against you isn’t as crazy as it looks.

          2. > You said it was *rich* not me.

            Yes, and I’ll repeat it — it’s rich for my purposes and those of devs like me. Clearly not rich for people who deal with absolutes.

            > Maybe you mean “straightforward”.

            No, dgp, I mean trivial, as in ordinary, as in nothing extraordinary to get an arm64 package running on an armv7 userspace. If you need to translate that to ‘straightforward’ so it could fit your worldviews — be my guest, I’m not stopping you.

            > I hate to go all meta on you but..

            And here I thought you were nothing but meta from your first comment. Silly me.

            > .. I was following up on your comment that was trying to suggest that locking yourself into an environment that is fighting against you isn’t as crazy as it looks.

            And I answered to that right away, but it went unnoticed by you (instead you decided to go after if chromebrew was rich), so let me restate: the ‘enviroment that is fighting against’ me is a question of *my* priorities. In my 35+y experience with PCs there has not been an environment which hasn’t fought me in one way or another. The thing is, there are more than enough benefits in chromeos for me to overcome whatever resistance the environment poses. Apropos, that same chromebook dual-boots into archlinux arm64, has debian arm64 as a KVM guest, and could easily get a crouton chroot environment with a userspace of my choice, yet I spend only a small fraction of my time with the device outside of chromeos — the latter does *most* of the things I care about better than the other environments.

          3. >it’s rich for my purposes and those of devs like me.

            Yet you have to write a wordy description of how to get basic tools on your system.
            I’m surprised your boss hasn’t spotted you constantly messing with that crap, forced the company credit card into your hand and told you to just buy a macbook already.

            For devs like me.. just my AndroidStudio instance would make your chromebook burst into flames somewhere between hitting the build button and the projected week it would take to build.

            >No, dgp, I mean trivial, as in ordinary,

            That’s not what trivial means in this context. Something that is trivial to do would not require paragraphs to explain. i.e. changing a light bulb is trivial hence there being next to no instructions on the package. Even if trivial was redefined to ordinary that still isn’t ordinary. You have to manually put your machine into dev mode and then manually extract packages and faff around to get the bastardised packages to run.

            >as in nothing extraordinary to get an arm64 package running on an armv7 userspace.

            You don’t think manually extracting debian packages isn’t extraordinary? It seems very much “going beyond what is usual, regular, or customary” to me. *shrug*

            >me is a question of *my* priorities.
            >And I answered to that right away,

            Because that’s something that is totally subjective. Some people actually seem to like doing things totally ass backwards because of things like; https://en.wikipedia.org/wiki/Sunk_cost https://en.wikipedia.org/wiki/Escalation_of_commitment and so on.

            >instead you decided to go after if chromebrew was rich

            Because a quick skim of the chromebrew repo should have made it totally obvious it’s rich in no way shape or form and probably a security hazard considering so many of the packages there haven’t been updated in 2 years.

            >In my 35+y experience with PCs there has not been an environment
            >which hasn’t fought me in one way or another.

            If you can install whatever you like then there is no reason to fight anything. Debian decides very little for you and provides almost all of the combinations you can imagine where possible.

            Chromeos is literally designed as a prison from the get go. They’ve taken perfectly good free software, removed as much of your freedom as possible and then added in Google spyware as an added bonus.

            >the latter does *most* of the things I care about better than the other environments.

            And we both know why chromeos works better. It’s because of the binary driver mess that means that chromeos is the only thing that actually functions *properly* on those machines and means those machines will become a rotting pile of remote exploits as soon as Google decides you aren’t worth spying on anymore.

          4. I appreciate linguistic bikeshedding and political drivel, but that about concludes our totally-not-meta discussion. If you figure out something meaningful to say on the subject, I’ll participate. Until then, enjoy your day, dgp.

          5. Usually I close the tab and forget about it because this is always what you do and it’s easy to tell you’re getting to this point because you try spinning your “I’m the only *dev* in the world” (No true Scotsman) elitist nonsense, calling people kids and so on… but I just found this tab open after looking for another so lets have some more fun.

            >I appreciate linguistic bikeshedding

            Words mean things. If I tell my boss a change is trivial that does not mean that it will take a whole day to implement and require the full test suite to be run to verify it. As a native English speaker your usage of the word trivial does not mean what you eventually tried to make it mean. Sure say that’s bikeshedding but go and ask other native speakers what trivial means in this context and you’ll get the same answer. I’ll coin a term for you in return “linguistic dishonesty”; Using a word with it’s generally accepted meaning initially only to try to retroactively change the meaning of the word if it doesn’t work out.

            > and political drivel,

            I don’t care about politics. Not wanting spyware you can’t remove pre-installed on your machines isn’t a political argument. Sure go ahead and throw away all of the freedoms that are granted to you via the GPL etc I don’t actually care if *you* do that but you can’t honestly say having Google or whoever’s spyware on a machine is good thing. TBH I’m not sure if you can work on projects that require NDAs and so on while working on a machine that you know sends a huge amount of usage data to a third party.

            Anyhow if I was going to make a political argument it wouldn’t be about making your own life difficult.. it would probably be about giving money to a company that is so internally messed up they had a mass hug and cry session after not being able to intentionally mess with an election well enough to get their preferred party elected.

            Fortunately the GPL etc grant me the freedom to do whatever the hell I like with lots of software they’ve touched and not give a crap about all of the shady business they get up to.

            >I’ll participate.

            I think that would require you to have better arguments than “actually words mean what I say they mean now” and “I’m the only proper dev”. If you need to build some toolchains I have a i7 4790K/GTX1050ti machine spare you can have for the cost of shipping.

          6. dgp, there’s just one individual on this site I’ve called ‘a kid’.. And here’s but an example of why:

            > Even if trivial was redefined to ordinary that still isn’t ordinary. You have to manually put your machine into dev mode and then manually extract packages and faff around to get the bastardised packages to run.

            Sure, let’s “redefine” trivial: https://www.merriam-webster.com/dictionary/trivial

            trivial, adjective

            2 : commonplace, ordinary

            > manually put your machine into dev mode

            Another trivial step, done once and forgotten about by chromeos dev-mode users..

            > manually extract packages and faff around to get the bastardised packages to run.

            Which part of this article being a primer of arm64-over-armhf-userspace *sans a package manager* you don’t get? I mean, this is only the 3rd or 4th time you chase your own tail with ‘so much manual labor’.

            > TBH I’m not sure if you can work on projects that require NDAs and so on while working on a machine that you know sends a huge amount of usage data to a third party.

            Ever heard of corporate google drive and corporate google mail? Apparently not.

            See, your issue is that you have so little experience in (and ergo, understanding of) the matters you come edgy about, that you need to stay absolutely meta via bikeshedding and political rhetorics in quite a few discussions I’ve had with you. BTW, teenagers get easily triggered when an adult calls them ‘a kid’. Are you a teenager, per chance?

    1. Most phone vendors don’t want to deal with ISA transitions, and aarch32 and aarch64 are different ISAs. Also, due to (1) larger pointers in 64-bit code and (2) absence of the compressed arm ISA known as Thumb on aarch64, most arm64 binaries tend to be larger. Larger binaries, larger distros, less space for photos, mp3s and apps. Lastly, some pointer-chasing code can be slower in 64-bit again thanks to larger data records, but that is addressed by arm64_32.

      The natural answer to all those points above under linux is called multi-arch support — on arm that’d be aarch64 kernel and userspace, with explicit support for armhf binaries. Canonical (ubuntu touch) and Google (android, chromeOS) both decided to ship aarch64 kernel with armhf userpaces. Ubuntu touch is gone, whereas google officially mandate aarch64 in android since last month — https://developer.android.com/distribute/best-practices/develop/64-bit. We can expect chromeOS to follow suit.

      Reply

        1. Not an android user here, but chances are that if you have a recent android and armv8 hw, userspace is already effectively multi-arch (multi-lib in android terms). Get an android app with a known 64-bit version, i.e. Geekbench, and see what it says on your phone.

          Reply

          1. Thanks blu

            Well honestly I don’t care enough to go that far finding out 😉

            Just out of curiosity what are you using if not android?

          2. I was a sworn s30/40 user for many years, then firefoxos until it went under, at which point I was forced into ios. I’ve written (system-level) code for android, but I’ve never owned an android phone : )

            On the subject of android, checking the timelines again, it turns out android got 64-bit multi-lib earlier than I remembered — since lolipop.

          3. I also had the impression it was 64 bit from a long time but got curious.

            I never was a nokia fan couldn’t wait till android is coming. It never happened, till i lost interest…

            Lol then Samsung came now im with android as i cant use ios (due to allergy 😛 ) its for me the least bad of the horrifying options. No more with Samsung from along time though

    2. >As a question why have Android OS boxes stayed 32 bit software while the SoC are 64 bit ?

      TBH I doubt it has anything to do with size and was just a timing issue. Almost all apps that ship with native binaries only shipped with armv7 and x86 and they didn’t want to make that work (i.e. ship multiple versions of all of the android system libraries, make the interactions work correctly, get AndroidStudio etc compatible) at that time and it was much easier just to continue with armv7 for the userland.

      Now that no one makes new armv7 phones it’s much easier to have period of cross over before the force all apps that only have armv7 native libraries out of the store. They have already done part of that by nuking apps that have an older target SDK.

      Reply

      1. > Almost all apps that ship with native binaries only shipped with armv7 and x86

        Many native Android apps lacked x86 support and that’s why Intel had to write an emulator to run ARM code on their pitiful attempt at making x86-based SoC for phones.

        Reply

        1. >Many native Android apps lacked x86 support

          That might have been the case back at android 2 but the x86 emulator has been the main development environment for a lot of people for a long time now so if you don’t have x86 libraries you’ll be annoying almost every android developer for no good reason. I don’t even think you could have a 100% on the metal native app back then. A game I worked on a good 8 or more years ago shipped with arm and x86 libraries.

          >and that’s why Intel had to write an emulator to run ARM code on their
          >pitiful attempt at making x86-based SoC for phones.

          The amount of apps with native libraries isn’t actually that high AFAIK. It’s mainly games that use cross platform engines like unity. Whether or not Intel’s attempt to get into the SoC market was a good idea aside not being able to run games that people want to play wouldn’t have been a great plan would it?

          Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Khadas VIM4 SBC
Khadas VIM4 SBC