Raspberry Pi Suddenly Not Working? You May Have to Repair your microSD Card

Automatic updates are great because they keep your system up-to-date with the latest features and/or security fixes. That’s as long as the firmware is not messed up of course, as Bootlin and others found out when they discovered their Raspberry Pi board(s) had become inaccessible after an ill-fated Raspbian update.

Raspberry Pi Out of Order

What happened is that raspi-copies-and-fills package, which implements optimized low-level memory functions for the ARM processor,  was updated on March 11th, and the update somehow made some programs completely fail to run. This explains why Bootlin guys were unable to access their Raspberry Pi over SSH.

The fix is simple, as long as you have physical access to your Raspberry Pi’s micro SD card, remove it from the board, and insert it into your computer, and…:

  1. Repair the rootfs partition with
  2. Delete etc/ld.so.preload

Unmount the micro SD card, and reinsert it into your Raspberry Pi board. If you’ve installed your Raspberry Pi in a hard to access location, it looks like there’s no solution to repair your micro SD card remotely since SSH or other network connection methods won’t work.

The good news is that Raspbian team acted quickly to resolve the issue, and a fix has already been committed. That means it should now be safe to upgrade Raspbian to the latest version.

Support CNX Software - Donate via PayPal or become a Patron on Patreon
Subscribe
Notify of
guest
36 Comments
oldest
newest most voted
Jerry
Jerry
1 year ago

Not really a big issue. Rpi engineers focus on extreme performance with optimal code. Mistakes happen.

tkaiser
tkaiser
1 year ago

> Rpi engineers focus on extreme performance with optimal code

Doesn’t seem so. They focus on backwards compatibility even if it could hurt performance on the vast majority of RPi out there (those with the ARMv7 and ARMv8 cores) and introduces upgrade hassles?

The raspi-copies-and-fills package provides libarmmem.so via /etc/ld.so.preload which is described as ‘Replacement memcpy and memset functionality for the Raspberry Pi with the intention of gaining greater performance. ‘Raspberry Pi’ as in the first versions and Zeros using the horribly outdated ARM11 core.

But there seems to be no performance gains with RPi 3B/3B+ anyway: https://bugs.launchpad.net/raspbian/+bug/1573473

tkaiser
tkaiser
1 year ago

In fact it’s https://github.com/bavison/arm-mem/commits/master minus the last two commits.

h. n.
h. n.
1 year ago

mistakes happen, but it’s QA/testers work not to release untested code to the wild. especially one that breakes device in such obvious way. that says about quality.

dgp
dgp
1 year ago

>mistakes happen, but it’s QA/testers work not to release untested code to the wild.

Very hard to test every single package in a distro. Bugs like this have happened in glibc (sse optimisation that broke memcpy between overlapping locations IIRC) and weren’t noticed for months after the fact.

k c
1 year ago

‘werent noticed for months’ is not exactly the same as ‘openssh doesnt run after upgrade’ range of bug

dgp
dgp
1 year ago

From the commit logs for armmem it looks like the breakage here happened a month ago and it’s only just being noticed now because of it breaking raspbian.
Maybe the rpi guys should have tested better but I think they probably shouldn’t be preloading a library that replaces libc functionality for everything that’s running either.

willy
willy
1 year ago

> Rpi engineers focus on extreme performance
Excellent 🙂 Obviously as usual you have no idea what you’re talking about Jerry but this one is really really fun!

blu
blu
1 year ago

Jerry is apparently leveling up.

k c
1 year ago

dont feed the troll, otherwise it levels up?

tkaiser
tkaiser
1 year ago

The approach itself sounds somewhat interesting, see https://github.com/bavison/arm-mem/commits/master — they build libarmmem-v6l.so and libarmmem-v7l.so separately but I really wonder whether someone did any extensive testing in which areas performance should improve. And I also wonder how a commit from Mon, 21 Jan 2019 could cause harm just two days ago? But maybe they update the Raspbian repositories only from time to time? At least this is where QA obviously failed: Shell raspi-copies-and-fills (0.10) stretch; urgency=medium * Revert ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 -- Serge Schneider Mon, 11 Mar 2019 18:05:30 +0000 raspi-copies-and-fills (0.8) stretch; urgency=medium * Update to ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 - Add a NEON implementation… Read more »

blu
blu
1 year ago

@tkaiser, that commit is just strlen, though, contrary to its commit message.

blu
blu
1 year ago

Wait, why are the raspi guys reverting ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 (strlen) when they should be reverting ea0bde27d823bd27f2e38ea0913f124b540c5ecc (memcmp) ?

willy
willy
1 year ago

Probably because they do what Jerry suggests them

tkaiser
tkaiser
1 year ago

> I really wonder whether someone did any extensive testing in which areas performance should improve I decided to give it a try myself, chose the slowest SBC I’ve lying around (RPi 2B) and tested not with Raspbian but with an Armbian based OMV image (a debootstrapped Debian Stretch armhf) through the following conditions: * No changes: http://ix.io/1Du0 * libarmmem-v7l.so: http://ix.io/1Duh * libarmmem-v7l.so and /usr/bin/tvservice -o: http://ix.io/1Dub The only benchmark number that improves with libarmmem-v7l.so in place is memcpy which is pretty irrelevant for the stuff I’m interested in (server tasks). To test through a bit more different use cases… Read more »

tkaiser
tkaiser
1 year ago

It took another two days to repeat the Phoronix benchmark test twice, once without libarmmem-v7l.so preloaded, the other time with: https://openbenchmarking.org/result/1903178-SP-1903164SP81 TL;DR: libarmmem-v7l.so shows no improvements or even slows almost everything slightly down, the only two benchmarks that could benefit were tinymembench/memcpy and Primesieve v7.1. So unless you want to show silly mbw benchmark scores (like the RPi people do with every new iteration of their outdated hardware) better deinstall the raspi-copies-and-fills package that is part of Raspbian. My test environment was an OMV image for RPi (which is based on Armbian which is essentially a clean debootstrapped Debian armhf)… Read more »

dgp
dgp
1 year ago

Might be an issue for those people that have integrated rpi products into their own stuff and now have a bunch of broken stuff out there. I used to know a guy that had a few hundred pis in different cities.. hope he didn’t remote update all of those.

admin
admin
1 year ago

pearls of wisdom from the clown himself .. I was looking forward to it

Alain
Alain
1 year ago

DANGER!
mkfs.ext4 = format the partition = delete everything !
You want to use the (forced) filecheck command : fsck -f /dev/mmcblk0p2

frank
1 year ago

Haha everybody makes mistakes like that, did you actually execute that command? I have problems too, but mine are on a eMMC. e2fsck segfaults over there =P

Yann
Yann
1 year ago

Mkfs, are you sure you don’t mean fsck ?

David Willmore
David Willmore
1 year ago

I saw the typo in the bootlin article yesterday and assumed they were being snarky. 🙂 Best way to improve the performance of an Rpi is to just format the SD card… LOL.

tkaiser
tkaiser
1 year ago

Where can the source code of these additional Raspbian packages like raspi-copies-and-fills be found?

tkaiser
tkaiser
1 year ago

Thanks, so it’s https://github.com/bavison/arm-mem minus strlen-v7l.S and test-strlen.c and a proper commit history?

Ah, the packages are here: https://github.com/RPi-Distro

xnc-hardware
xnc-hardware
1 year ago

It’s so unfortunate and sad that Rpi didn’t pick different SoC vendor 🙁
Imagine what would have been 🙁

Scott
Scott
1 year ago

I get this frightening message trying to run e2fsck . Is it ok to continue? Will I lose my data?

[email protected]:~ $ e2fsck -f /dev/mmcblk0p2
e2fsck 1.43.4 (31-Jan-2017)
/dev/mmcblk0p2 is mounted.

WARNING!!! The filesystem is mounted. If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.

Do you really want to continue? no
check aborted.

Scott
Scott
1 year ago

Thanks, looks like I am impacted but the solution that seems to work is this in case someone else looks at this thread…

LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
sudo rm /etc/ld.so.preload

then reboot

willy
willy
1 year ago

LD_PRELOAD will be ignored if you use sudo since it’s setuid. You’ll have to get root first using sudo and then use LD_PRELOAD=… rm /etc/…. At this point if you can issue these commands I suspect you’re not affected and don’t need to set LD_PRELOAD at all.

David Willmore
David Willmore
1 year ago

Or he was already logged in and is using an older openssh server binary (which has probably been deleted, so the inode will free if he logs out and he’ll be locked out).

The purpose of the fsck was because you couldn’t log into these broken rpi boards and you had been forced to power them off without shutting them down. So, the fs was in a dirty state. If you didn’t have to do that, then you’re fine to just do the other steps.

Diego
Diego
1 year ago

Oh my oh my…

Advertisements