Raspberry Pi Suddenly Not Working? You May Have to Repair your microSD Card

Automatic updates are great because they keep your system up-to-date with the latest features and/or security fixes. That’s as long as the firmware is not messed up of course, as Bootlin and others found out when they discovered their Raspberry Pi board(s) had become inaccessible after an ill-fated Raspbian update.

Raspberry Pi Out of Order

What happened is that raspi-copies-and-fills package, which implements optimized low-level memory functions for the ARM processor,  was updated on March 11th, and the update somehow made some programs completely fail to run. This explains why Bootlin guys were unable to access their Raspberry Pi over SSH.

The fix is simple, as long as you have physical access to your Raspberry Pi’s micro SD card, remove it from the board, and insert it into your computer, and…:

  1. Repair the rootfs partition with
  2. Delete etc/ld.so.preload

Unmount the micro SD card, and reinsert it into your Raspberry Pi board. If you’ve installed your Raspberry Pi in a hard to access location, it looks like there’s no solution to repair your micro SD card remotely since SSH or other network connection methods won’t work.

The good news is that Raspbian team acted quickly to resolve the issue, and a fix has already been committed. That means it should now be safe to upgrade Raspbian to the latest version.

36
Leave a Reply

avatar
6 Comment threads
30 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
17 Comment authors
DiegoScottxnc-hardwarek cadmin Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Jerry
Guest
Jerry

Not really a big issue. Rpi engineers focus on extreme performance with optimal code. Mistakes happen.

tkaiser
Guest
tkaiser

> Rpi engineers focus on extreme performance with optimal code

Doesn’t seem so. They focus on backwards compatibility even if it could hurt performance on the vast majority of RPi out there (those with the ARMv7 and ARMv8 cores) and introduces upgrade hassles?

The raspi-copies-and-fills package provides libarmmem.so via /etc/ld.so.preload which is described as ‘Replacement memcpy and memset functionality for the Raspberry Pi with the intention of gaining greater performance. ‘Raspberry Pi’ as in the first versions and Zeros using the horribly outdated ARM11 core.

But there seems to be no performance gains with RPi 3B/3B+ anyway: https://bugs.launchpad.net/raspbian/+bug/1573473

tkaiser
Guest
tkaiser

In fact it’s https://github.com/bavison/arm-mem/commits/master minus the last two commits.

h. n.
Guest
h. n.

mistakes happen, but it’s QA/testers work not to release untested code to the wild. especially one that breakes device in such obvious way. that says about quality.

dgp
Guest
dgp

>mistakes happen, but it’s QA/testers work not to release untested code to the wild.

Very hard to test every single package in a distro. Bugs like this have happened in glibc (sse optimisation that broke memcpy between overlapping locations IIRC) and weren’t noticed for months after the fact.

k c
Guest

‘werent noticed for months’ is not exactly the same as ‘openssh doesnt run after upgrade’ range of bug

dgp
Guest
dgp

From the commit logs for armmem it looks like the breakage here happened a month ago and it’s only just being noticed now because of it breaking raspbian.
Maybe the rpi guys should have tested better but I think they probably shouldn’t be preloading a library that replaces libc functionality for everything that’s running either.

willy
Guest
willy

> Rpi engineers focus on extreme performance
Excellent πŸ™‚ Obviously as usual you have no idea what you’re talking about Jerry but this one is really really fun!

blu
Guest
blu

Jerry is apparently leveling up.

k c
Guest

dont feed the troll, otherwise it levels up?

tkaiser
Guest
tkaiser

The approach itself sounds somewhat interesting, see https://github.com/bavison/arm-mem/commits/master — they build libarmmem-v6l.so and libarmmem-v7l.so separately but I really wonder whether someone did any extensive testing in which areas performance should improve.

And I also wonder how a commit from Mon, 21 Jan 2019 could cause harm just two days ago? But maybe they update the Raspbian repositories only from time to time? At least this is where QA obviously failed:

blu
Guest
blu

@tkaiser, that commit is just strlen, though, contrary to its commit message.

blu
Guest
blu

Wait, why are the raspi guys reverting ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 (strlen) when they should be reverting ea0bde27d823bd27f2e38ea0913f124b540c5ecc (memcmp) ?

willy
Guest
willy

Probably because they do what Jerry suggests them

tkaiser
Guest
tkaiser

> I really wonder whether someone did any extensive testing in which areas performance should improve

I decided to give it a try myself, chose the slowest SBC I’ve lying around (RPi 2B) and tested not with Raspbian but with an Armbian based OMV image (a debootstrapped Debian Stretch armhf) through the following conditions:

* No changes: http://ix.io/1Du0
* libarmmem-v7l.so: http://ix.io/1Duh
* libarmmem-v7l.so and /usr/bin/tvservice -o: http://ix.io/1Dub

The only benchmark number that improves with libarmmem-v7l.so in place is memcpy which is pretty irrelevant for the stuff I’m interested in (server tasks).

To test through a bit more different use cases I thought I would try a popular set of kitchen-sink benchmarks which are usually not sufficient to compare different systems but in this case (same compiler, same OS version, same libs and so on) it should work. Unfortunately the execution mode of these Phoronix benchmarks is that stupid that benchmark execution for 21 simple tests takes over half a day (WTF?).

So my test ran into an SSH idle timeout: https://pastebin.com/raw/3ir2wksu

Anyway, same picture: with libarmmem-v7l.so in place (that’s the results labeled ‘Raspberry Pi 2 B’ compared to ‘Raspberry Pi 2 B Armbian’ without) only with irrelevant tests (tinymembench’s memcpy or PostgreSQL write buffer test which is IO bound) running with libarmmem-v7l.so shows significantly better results. With the Primesieve, Lame encoding and ‘Hierarchical INTegration’ tests libarmmem-v7l.so shows slightly better performance while with all other tests it’s the opposite or performance is identical.

Still don’t know why RPi folks think preloading the libarmmem library on Raspbian would be a good idea (at least if it’s about the somewhat modern RPi ARMv7/ARMv8 variants and not the horribly outdated single core ARM11 thingies).

tkaiser
Guest
tkaiser

It took another two days to repeat the Phoronix benchmark test twice, once without libarmmem-v7l.so preloaded, the other time with:

https://openbenchmarking.org/result/1903178-SP-1903164SP81

TL;DR: libarmmem-v7l.so shows no improvements or even slows almost everything slightly down, the only two benchmarks that could benefit were tinymembench/memcpy and Primesieve v7.1.

So unless you want to show silly mbw benchmark scores (like the RPi people do with every new iteration of their outdated hardware) better deinstall the raspi-copies-and-fills package that is part of Raspbian.

My test environment was an OMV image for RPi (which is based on Armbian which is essentially a clean debootstrapped Debian armhf) and raspi-copies-and-fills 0.10.

dgp
Guest
dgp

Might be an issue for those people that have integrated rpi products into their own stuff and now have a bunch of broken stuff out there. I used to know a guy that had a few hundred pis in different cities.. hope he didn’t remote update all of those.

admin
Guest
admin

pearls of wisdom from the clown himself .. I was looking forward to it

Alain
Guest
Alain

DANGER!
mkfs.ext4 = format the partition = delete everything !
You want to use the (forced) filecheck command : fsck -f /dev/mmcblk0p2

Yann
Guest
Yann

Mkfs, are you sure you don’t mean fsck ?

David Willmore
Guest
David Willmore

I saw the typo in the bootlin article yesterday and assumed they were being snarky. πŸ™‚ Best way to improve the performance of an Rpi is to just format the SD card… LOL.

tkaiser
Guest
tkaiser

Where can the source code of these additional Raspbian packages like raspi-copies-and-fills be found?

Peter
Guest
tkaiser
Guest
tkaiser

Thanks, so it’s https://github.com/bavison/arm-mem minus strlen-v7l.S and test-strlen.c and a proper commit history?

Ah, the packages are here: https://github.com/RPi-Distro

xnc-hardware
Guest
xnc-hardware

It’s so unfortunate and sad that Rpi didn’t pick different SoC vendor πŸ™
Imagine what would have been πŸ™

Scott
Guest
Scott

I get this frightening message trying to run e2fsck . Is it ok to continue? Will I lose my data?

[email protected]:~ $ e2fsck -f /dev/mmcblk0p2
e2fsck 1.43.4 (31-Jan-2017)
/dev/mmcblk0p2 is mounted.

WARNING!!! The filesystem is mounted. If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.

Do you really want to continue? no
check aborted.