Raspberry Pi Suddenly Not Working? You May Have to Repair your microSD Card

Automatic updates are great because they keep your system up-to-date with the latest features and/or security fixes. That’s as long as the firmware is not messed up of course, as Bootlin and others found out when they discovered their Raspberry Pi board(s) had become inaccessible after an ill-fated Raspbian update.

Raspberry Pi Out of Order

What happened is that raspi-copies-and-fills package, which implements optimized low-level memory functions for the ARM processor,  was updated on March 11th, and the update somehow made some programs completely fail to run. This explains why Bootlin guys were unable to access their Raspberry Pi over SSH.

The fix is simple, as long as you have physical access to your Raspberry Pi’s micro SD card, remove it from the board, and insert it into your computer, and…:

  1. Repair the rootfs partition with
  2. Delete etc/ld.so.preload

Unmount the micro SD card, and reinsert it into your Raspberry Pi board. If you’ve installed your Raspberry Pi in a hard to access location, it looks like there’s no solution to repair your micro SD card remotely since SSH or other network connection methods won’t work.

The good news is that Raspbian team acted quickly to resolve the issue, and a fix has already been committed. That means it should now be safe to upgrade Raspbian to the latest version.

Share this:
FacebookTwitterHacker NewsSlashdotRedditLinkedInPinterestFlipboardMeWeLineEmailShare

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

36 Replies to “Raspberry Pi Suddenly Not Working? You May Have to Repair your microSD Card”

  1. Not really a big issue. Rpi engineers focus on extreme performance with optimal code. Mistakes happen.

    1. > Rpi engineers focus on extreme performance with optimal code

      Doesn’t seem so. They focus on backwards compatibility even if it could hurt performance on the vast majority of RPi out there (those with the ARMv7 and ARMv8 cores) and introduces upgrade hassles?

      The raspi-copies-and-fills package provides libarmmem.so via /etc/ld.so.preload which is described as ‘Replacement memcpy and memset functionality for the Raspberry Pi with the intention of gaining greater performance. ‘Raspberry Pi’ as in the first versions and Zeros using the horribly outdated ARM11 core.

      But there seems to be no performance gains with RPi 3B/3B+ anyway: https://bugs.launchpad.net/raspbian/+bug/1573473

    2. mistakes happen, but it’s QA/testers work not to release untested code to the wild. especially one that breakes device in such obvious way. that says about quality.

      1. >mistakes happen, but it’s QA/testers work not to release untested code to the wild.

        Very hard to test every single package in a distro. Bugs like this have happened in glibc (sse optimisation that broke memcpy between overlapping locations IIRC) and weren’t noticed for months after the fact.

          1. From the commit logs for armmem it looks like the breakage here happened a month ago and it’s only just being noticed now because of it breaking raspbian.
            Maybe the rpi guys should have tested better but I think they probably shouldn’t be preloading a library that replaces libc functionality for everything that’s running either.

    3. > Rpi engineers focus on extreme performance
      Excellent 🙂 Obviously as usual you have no idea what you’re talking about Jerry but this one is really really fun!

      1. The approach itself sounds somewhat interesting, see https://github.com/bavison/arm-mem/commits/master — they build libarmmem-v6l.so and libarmmem-v7l.so separately but I really wonder whether someone did any extensive testing in which areas performance should improve.

        And I also wonder how a commit from Mon, 21 Jan 2019 could cause harm just two days ago? But maybe they update the Raspbian repositories only from time to time? At least this is where QA obviously failed:

          1. Wait, why are the raspi guys reverting ce0fcb7273e39a3c51e3b44dd6b82c04d1fba443 (strlen) when they should be reverting ea0bde27d823bd27f2e38ea0913f124b540c5ecc (memcmp) ?

          2. > I really wonder whether someone did any extensive testing in which areas performance should improve

            I decided to give it a try myself, chose the slowest SBC I’ve lying around (RPi 2B) and tested not with Raspbian but with an Armbian based OMV image (a debootstrapped Debian Stretch armhf) through the following conditions:

            * No changes: http://ix.io/1Du0
            * libarmmem-v7l.so: http://ix.io/1Duh
            * libarmmem-v7l.so and /usr/bin/tvservice -o: http://ix.io/1Dub

            The only benchmark number that improves with libarmmem-v7l.so in place is memcpy which is pretty irrelevant for the stuff I’m interested in (server tasks).

            To test through a bit more different use cases I thought I would try a popular set of kitchen-sink benchmarks which are usually not sufficient to compare different systems but in this case (same compiler, same OS version, same libs and so on) it should work. Unfortunately the execution mode of these Phoronix benchmarks is that stupid that benchmark execution for 21 simple tests takes over half a day (WTF?).

            So my test ran into an SSH idle timeout: https://pastebin.com/raw/3ir2wksu

            Anyway, same picture: with libarmmem-v7l.so in place (that’s the results labeled ‘Raspberry Pi 2 B’ compared to ‘Raspberry Pi 2 B Armbian’ without) only with irrelevant tests (tinymembench’s memcpy or PostgreSQL write buffer test which is IO bound) running with libarmmem-v7l.so shows significantly better results. With the Primesieve, Lame encoding and ‘Hierarchical INTegration’ tests libarmmem-v7l.so shows slightly better performance while with all other tests it’s the opposite or performance is identical.

            Still don’t know why RPi folks think preloading the libarmmem library on Raspbian would be a good idea (at least if it’s about the somewhat modern RPi ARMv7/ARMv8 variants and not the horribly outdated single core ARM11 thingies).

          3. It took another two days to repeat the Phoronix benchmark test twice, once without libarmmem-v7l.so preloaded, the other time with:

            https://openbenchmarking.org/result/1903178-SP-1903164SP81

            TL;DR: libarmmem-v7l.so shows no improvements or even slows almost everything slightly down, the only two benchmarks that could benefit were tinymembench/memcpy and Primesieve v7.1.

            So unless you want to show silly mbw benchmark scores (like the RPi people do with every new iteration of their outdated hardware) better deinstall the raspi-copies-and-fills package that is part of Raspbian.

            My test environment was an OMV image for RPi (which is based on Armbian which is essentially a clean debootstrapped Debian armhf) and raspi-copies-and-fills 0.10.

    4. Might be an issue for those people that have integrated rpi products into their own stuff and now have a bunch of broken stuff out there. I used to know a guy that had a few hundred pis in different cities.. hope he didn’t remote update all of those.

  2. DANGER!
    mkfs.ext4 = format the partition = delete everything !
    You want to use the (forced) filecheck command : fsck -f /dev/mmcblk0p2

    1. Yes. Thanks. I just copied the command from Bootlin blog post without thinking about it. I changed that to e2fsck.

      1. Haha everybody makes mistakes like that, did you actually execute that command? I have problems too, but mine are on a eMMC. e2fsck segfaults over there =P

    1. I saw the typo in the bootlin article yesterday and assumed they were being snarky. 🙂 Best way to improve the performance of an Rpi is to just format the SD card… LOL.

  3. Where can the source code of these additional Raspbian packages like raspi-copies-and-fills be found?

  4. It’s so unfortunate and sad that Rpi didn’t pick different SoC vendor 🙁
    Imagine what would have been 🙁

  5. I get this frightening message trying to run e2fsck . Is it ok to continue? Will I lose my data?

    pi@raspberrypi:~ $ e2fsck -f /dev/mmcblk0p2
    e2fsck 1.43.4 (31-Jan-2017)
    /dev/mmcblk0p2 is mounted.

    WARNING!!! The filesystem is mounted. If you continue you ***WILL***
    cause ***SEVERE*** filesystem damage.

    Do you really want to continue? no
    check aborted.

    1. Wait… If you are doing this in your Raspberry Pi that means you are not affected by the issue in this post…

      1. Thanks, looks like I am impacted but the solution that seems to work is this in case someone else looks at this thread…

        LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
        sudo rm /etc/ld.so.preload

        then reboot

        1. LD_PRELOAD will be ignored if you use sudo since it’s setuid. You’ll have to get root first using sudo and then use LD_PRELOAD=… rm /etc/…. At this point if you can issue these commands I suspect you’re not affected and don’t need to set LD_PRELOAD at all.

      2. Or he was already logged in and is using an older openssh server binary (which has probably been deleted, so the inode will free if he logs out and he’ll be locked out).

        The purpose of the fsck was because you couldn’t log into these broken rpi boards and you had been forced to power them off without shutting them down. So, the fs was in a dirty state. If you didn’t have to do that, then you’re fine to just do the other steps.

Leave a Reply

Your email address will not be published. Required fields are marked *

Khadas VIM4 SBC
Khadas VIM4 SBC