Automatic updates are great because they keep your system up-to-date with the latest features and/or security fixes. That’s as long as the firmware is not messed up of course, as Bootlin and others found out when they discovered their Raspberry Pi board(s) had become inaccessible after an ill-fated Raspbian update.
What happened is that raspi-copies-and-fills package, which implements optimized low-level memory functions for the ARM processor, was updated on March 11th, and the update somehow made some programs completely fail to run. This explains why Bootlin guys were unable to access their Raspberry Pi over SSH.
The fix is simple, as long as you have physical access to your Raspberry Pi’s micro SD card, remove it from the board, and insert it into your computer, and…:
- Repair the rootfs partition with
1e2fsck -f /dev/mmcblk0p2 - Delete etc/ld.so.preload
Unmount the micro SD card, and reinsert it into your Raspberry Pi board. If you’ve installed your Raspberry Pi in a hard to access location, it looks like there’s no solution to repair your micro SD card remotely since SSH or other network connection methods won’t work.
The good news is that Raspbian team acted quickly to resolve the issue, and a fix has already been committed. That means it should now be safe to upgrade Raspbian to the latest version.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress
Not really a big issue. Rpi engineers focus on extreme performance with optimal code. Mistakes happen.
> Rpi engineers focus on extreme performance with optimal code
Doesn’t seem so. They focus on backwards compatibility even if it could hurt performance on the vast majority of RPi out there (those with the ARMv7 and ARMv8 cores) and introduces upgrade hassles?
The
raspi-copies-and-fillspackage provideslibarmmem.sovia/etc/ld.so.preloadwhich is described as ‘Replacement memcpy and memset functionality for the Raspberry Pi with the intention of gaining greater performance. ‘Raspberry Pi’ as in the first versions and Zeros using the horribly outdated ARM11 core.But there seems to be no performance gains with RPi 3B/3B+ anyway: https://bugs.launchpad.net/raspbian/+bug/1573473
In fact it’s https://github.com/bavison/arm-mem/commits/master minus the last two commits.
mistakes happen, but it’s QA/testers work not to release untested code to the wild. especially one that breakes device in such obvious way. that says about quality.
>mistakes happen, but it’s QA/testers work not to release untested code to the wild.
Very hard to test every single package in a distro. Bugs like this have happened in glibc (sse optimisation that broke memcpy between overlapping locations IIRC) and weren’t noticed for months after the fact.
‘werent noticed for months’ is not exactly the same as ‘openssh doesnt run after upgrade’ range of bug
From the commit logs for armmem it looks like the breakage here happened a month ago and it’s only just being noticed now because of it breaking raspbian.
Maybe the rpi guys should have tested better but I think they probably shouldn’t be preloading a library that replaces libc functionality for everything that’s running either.
> Rpi engineers focus on extreme performance
Excellent 🙂 Obviously as usual you have no idea what you’re talking about Jerry but this one is really really fun!
Jerry is apparently leveling up.
dont feed the troll, otherwise it levels up?
The approach itself sounds somewhat interesting, see https://github.com/bavison/arm-mem/commits/master — they build libarmmem-v6l.so and libarmmem-v7l.so separately but I really wonder whether someone did any extensive testing in which areas performance should improve. And I also wonder how a commit from Mon, 21 Jan 2019 could cause harm just two days ago? But maybe they update the Raspbian repositories only from time to time? At least this is where QA obviously failed: raspi-copies-and-fills (0.10) stretch; urgency=medium * Revert ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 -- Serge Schneider Mon, 11 Mar 2019 18:05:30 +0000 raspi-copies-and-fills (0.8) stretch; urgency=medium * Update to ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 - Add a NEON implementation of… Read more »
@tkaiser, that commit is just strlen, though, contrary to its commit message.
https://github.com/bavison/arm-mem/commit/ea0bde27d823bd27f2e38ea0913f124b540c5ecc#commitcomment-32722342
Wait, why are the raspi guys reverting ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 (strlen) when they should be reverting ea0bde27d823bd27f2e38ea0913f124b540c5ecc (memcmp) ?
Probably because they do what Jerry suggests them
Interesting: https://github.com/bavison/arm-mem/commit/ea0bde27d823bd27f2e38ea0913f124b540c5ecc#commitcomment-32807513
> I really wonder whether someone did any extensive testing in which areas performance should improve I decided to give it a try myself, chose the slowest SBC I’ve lying around (RPi 2B) and tested not with Raspbian but with an Armbian based OMV image (a debootstrapped Debian Stretch armhf) through the following conditions: * No changes: http://ix.io/1Du0 * libarmmem-v7l.so: http://ix.io/1Duh * libarmmem-v7l.so and /usr/bin/tvservice -o: http://ix.io/1Dub The only benchmark number that improves with libarmmem-v7l.so in place is memcpy which is pretty irrelevant for the stuff I’m interested in (server tasks). To test through a bit more different use cases… Read more »
It took another two days to repeat the Phoronix benchmark test twice, once without libarmmem-v7l.so preloaded, the other time with: https://openbenchmarking.org/result/1903178-SP-1903164SP81 TL;DR: libarmmem-v7l.so shows no improvements or even slows almost everything slightly down, the only two benchmarks that could benefit were tinymembench/memcpy and Primesieve v7.1. So unless you want to show silly mbw benchmark scores (like the RPi people do with every new iteration of their outdated hardware) better deinstall the raspi-copies-and-fills package that is part of Raspbian. My test environment was an OMV image for RPi (which is based on Armbian which is essentially a clean debootstrapped Debian armhf)… Read more »
Might be an issue for those people that have integrated rpi products into their own stuff and now have a bunch of broken stuff out there. I used to know a guy that had a few hundred pis in different cities.. hope he didn’t remote update all of those.
pearls of wisdom from the clown himself .. I was looking forward to it
DANGER!
mkfs.ext4 = format the partition = delete everything !
You want to use the (forced) filecheck command : fsck -f /dev/mmcblk0p2
Yes. Thanks. I just copied the command from Bootlin blog post without thinking about it. I changed that to e2fsck.
Haha everybody makes mistakes like that, did you actually execute that command? I have problems too, but mine are on a eMMC. e2fsck segfaults over there =P
Mkfs, are you sure you don’t mean fsck ?
I saw the typo in the bootlin article yesterday and assumed they were being snarky. 🙂 Best way to improve the performance of an Rpi is to just format the SD card… LOL.
Where can the source code of these additional Raspbian packages like
raspi-copies-and-fillsbe found?https://archive.raspberrypi.org/debian/pool/main/r/raspi-copies-and-fills/
Thanks, so it’s https://github.com/bavison/arm-mem minus strlen-v7l.S and test-strlen.c
and a proper commit history?Ah, the packages are here: https://github.com/RPi-Distro
It’s so unfortunate and sad that Rpi didn’t pick different SoC vendor 🙁
Imagine what would have been 🙁
I get this frightening message trying to run e2fsck . Is it ok to continue? Will I lose my data?
pi@raspberrypi:~ $ e2fsck -f /dev/mmcblk0p2
e2fsck 1.43.4 (31-Jan-2017)
/dev/mmcblk0p2 is mounted.
WARNING!!! The filesystem is mounted. If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.
Do you really want to continue? no
check aborted.
You need to remove the card from your Raspberry Pi, and run that command inside your computer.
Wait… If you are doing this in your Raspberry Pi that means you are not affected by the issue in this post…
Thanks, looks like I am impacted but the solution that seems to work is this in case someone else looks at this thread…
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
sudo rm /etc/ld.so.preload
then reboot
LD_PRELOAD will be ignored if you use sudo since it’s setuid. You’ll have to get root first using sudo and then use LD_PRELOAD=… rm /etc/…. At this point if you can issue these commands I suspect you’re not affected and don’t need to set LD_PRELOAD at all.
Or he was already logged in and is using an older openssh server binary (which has probably been deleted, so the inode will free if he logs out and he’ll be locked out).
The purpose of the fsck was because you couldn’t log into these broken rpi boards and you had been forced to power them off without shutting them down. So, the fs was in a dirty state. If you didn’t have to do that, then you’re fine to just do the other steps.
Oh my oh my…