Checking Out Machine Check Exception (MCE) Errors in Linux

Machine Check Exception Error Linux

I recently reviewed ODROID-H2 with Ubuntu 19.04, and noticed some errors messages in the kernel log of the Intel Celeron J4105 single board computer while running SBC-Bench benchmark:


I did not know what do make of those errors, but I was told I would get more details with mcelog which can be installed as follows:


There’s just one little problem: it’s not in Ubuntu 19.04 repository, and a bug report mentions mcelog is not deprecated, and remove from Ubuntu 18.04 Bionic onwards. Instead, we’re being told the mcelog package functionality has been replaced by rasdaemon.

But before looking into the utilities, let’s find out what Machine Check Exception (MCE) is all about from ArchLinux Wiki:

A machine check exception (MCE) is an error generated by the CPU when the CPU detects that a hardware error or failure has occurred.

Machine check exceptions (MCEs) can occur for a variety of reasons ranging from undesired or out-of-spec voltages from the power supply, from cosmic radiation flipping bits in memory DIMMs or the CPU, or from other miscellaneous faults, including faulty software triggering hardware errors.

Hardware error should probably be taken seriously. Let’s investigate how to run the tools. First, I try to install mcelog from Ubuntu 16.04:


Oh good! It could install… Let’s run some commands:


Nothing interesting shows up here, but the file /var/log/mcelog is now up, and we can see details about the errors:


But let’s also try the recommended rasdaemon to see if we can get similar details.

Installation:


It looks like the service will not start automatically upon installation, so a reboot may be needed, or simply run the following command:


I ran a few commands and at first, it looked like some driver may be needed:


This should be related to EDAC drivers that are used for ECC memory according to a thread on Grokbase. Gemini Lake processors do not support ECC memory, so I probably don’t need it.

Running one more command to show the summary of errors, and we’re getting somewhere:


12 corrected error related to the L2 cache. We can get the full details with the appropriate command:


The status is green which means everything still works, but the utility reports a “large number of corrected cache errors”, and the “system (is) operating, but might lead to uncorrected errors soon” (See source code). It happens only a few times a day, and I’m not sure what can be done about the cache since it’s not something that can be changed as it’s embedded into the processor, maybe it’s just an issue with the processor I’m running. If somebody has an ODROID-H2 running, it may be useful to check out the kernel log with dmesg to see if you’ve got the same errors. If you do, please also indicate whether you have a board from the first batch (November 2018) or one of the new ODROID-H2 Rev B boards.

Share this:

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK Pi 4C Plus
Subscribe
Notify of
guest
The comment form collects your name, email and content to allow us keep track of the comments placed on the website. Please read and accept our website Terms and Privacy Policy to post a comment.
39 Comments
oldest
newest
tkaiser
tkaiser
4 years ago

> the file /var/log/mcelog is now up

That’s how we monitor x86 commodity hardware: installing the mcelog package, then defining a primitive rule watching the size of /var/log/mcelog. Once the size exceeds 0 you have a problem.

As for the occurrences of these 2nd level cache errors: most probably they only occur once data is pumped through the CPU cores (e.g. running 7-zip or cpuminer as part of sbc-bench for example).

mce error
mce error
1 year ago

Could someone provide me with some help? I use a small firewall with IPFire and get mce errors. How is this to be interpreted? # du -sh mcelog 24K    mcelog 12 # du -sh mcelog24K    mcelog # cat /var/log/mcelog Kernel does not support page offline interface Hardware event. This is not a software error. MCE 0 CPU 2 BANK 4 ADDR 1c4d35220 TIME 1665391460 Mon Oct 10 10:44:20 2022 MCG status: MCi status: Corrected error MCi_ADDR register valid MCA: Instruction CACHE Level-2 Instruction-Fetch Error STATUS 8400000000010151 MCGSTATUS 0 MCGCAP 806 APICID 4 SOCKETID 0 MICROCODE 411 CPUID Vendor Intel Family… Read more »

theguyuk
theguyuk
4 years ago

Might be worth also posting on Odroid H2 forum, issues or Ubuntu, maybe raise the number of board users reading this?

theguyuk
theguyuk
4 years ago

Seem to be incompatible memory posts on Odroid forum. Could it be memory or hardware chips driver issues?

willy
willy
4 years ago

Cache issues in CPUs do happen. I’ve been hit many times with those in UltraSparc processors for example (and my U5 is sick again due to this for the second time by the way). I think modern CPUs use parity to better resist issues caused by heat and other random events. It’s “simple” enough to drop a line and re-read it when reading inconsistent data (it’s more difficult when changes are present but nothing prevents from re-reading the faulty word if it was not modified). In case that’s what you’re facing, there could be a corner case of locally-modified data… Read more »

Laurent
Laurent
4 years ago

Some Intel chips go beyond parity checking and have ECC on L2. But I don’t know if this chip supports it, Intel doesn’t seem to document that openly.

Found a command to check that: sudo dmidecode and look for “Error Correction Type”. On my i7-8650U it shows L1D is parity protected, L2 is one-bit ECC and L3 it multi-bit ECC.

willy
willy
4 years ago

Yes I’m pretty sure that high-end chips do have ECC, but here we’re talking about an Atom basically. Well, marketingly speaking it’s a celeron 🙂

Laurent
Laurent
4 years ago

willy, you’re correct for DRAM ECC, here we’re talking about the internal caches 😉

willy
willy
4 years ago

Laurent, I was also speaking about internal caches 🙂 I’m sure I’ve seen it mentioned a few times in the past for Xeons and have some memories of seeing things like “L2 cache ECC” in some server BIOSes in the past. Note that it might have been quite old since my memory associates this to L2 not L3. Also IBM’s POWER8 and above definitely use ECC for L2 & L3.

willy
willy
4 years ago

The link doesn’t work here, it displays “privatebin is a minimalist …”.

willy
willy
4 years ago

Tried already, and again, but same result. It’s no big deal though, don’t worry 🙂

Eversor
Eversor
4 years ago

Hi there,

This is not normal. One error once in a couple months due to cosmic rays, that is normal.

You are being warned because at that rate there is something wrong with the hardware or too much EMI from another device.

tkaiser
tkaiser
4 years ago

> at that rate there is something wrong with the hardware

So far only ECCed (corrected) single bit flips in L2 cache. Harms slightly performance and will only be a real issue once two bits flip at the same time. While I wouldn’t trust such a CPU that much for the average use case a Gemini Lake box is taken (media center, desktop) this shouldn’t matter that much.

David Willmore
David Willmore
4 years ago

Physics is physics. You’re going to get bit flips in caches. Beta particles from C-14 decomposition. Beta particles from the Si itself, etc. Then there’s cosmic rays that you just can’t avoid as they’re everywhere. As transistors get smaller and smaller, Johnson–Nyquist noise becomes a bigger issue. The error may have even occured due to a transmission error on the chip–the right value may have been stored, but it got corrupted between the storage element and the ECC block. As long as the ECC is catching single bit errors at a relatively low rate, there’s noting to worry about. If… Read more »

willy
willy
4 years ago

Sometimes there are 4 in the same second, this is far too much, something is busted in this machine, and its ability to recover from all the events you enumerated is affected by this existing one. The probability of two-bit *unrelated* errors remains very low. But if the hardware is defective, the probability of two-bit errors in the same cache line cannot be dismissed.

Eversor
Eversor
4 years ago

That is usually coil whine and is not the issue.

I’d try setting a lower maximum clock and see if the problem goes away. I’d want to check if power regulation/supply isn’t deficient.

If the problem is the hardware, a torture test like Prime 95 will also make it much worse.

David Willmore
David Willmore
4 years ago

I don’t use an x86 MB/memory combo without first doing at least 24 hours of memtest86 and then Prime95 on torture test. That helps validate memroy, processor, power delivery, etc.

Mikko Rantalainen
Mikko Rantalainen
3 years ago

I used to run memtest86 for 24 hours, too. But then I got burned with broken RAM that memtest86 was not able to find no matter how long I tried. Didn’t know about Prime95 at that time but running sha1sum over all files in the filesystem repeatedly returned different hashes for the same files. And the problem went away by removing 2 of the 4 memory cards. As such, I think running memtest86 is just waste of time. Prime95 / mprime -t is much better option. However, you need to manually adjust the RAM usage to cover all chips. If… Read more »

tkaiser
tkaiser
4 years ago

> I’ve gone to the BIOS and changed two settings

And you lost around 1/3 of the CPU performance 🙂

Smells a bit like unstable high DVFS OPP…

Eversor
Eversor
4 years ago

Yeah, any of the modes be the cause of this unless they’re bugged. I agree with Kaiser it’s DVFS bug probably. Could also be something on the hardware design – like capacitors – which can’t handle higher current swings.

Eversor
Eversor
4 years ago

BTW, I once had an issue with a motherboard BIOS where it would have correctable and uncorrectable L1/L2 errors when doing light loads.

The board was initially K8 only but eventually supported Phenom 1 and 2.

Somewhere something broke, as the motherboard induced errors if I enabled frequency scaling on the K8 but worked fine with a Phenom II – which used a different version of C&Q.

K8 CPU worked fine if set at any fixed speed, just couldn’t change on demand. Prob was changing clocks too fast, without letting the VRM stabilize at the higher voltages first.

theguyuk
theguyuk
4 years ago

Have Odroid replied yet?

willy
willy
4 years ago

We could have hoped better, like “given that yours shows problems we cannot reproduce it definitely indicates a hardware issue and a risk of accelerated aging, we’re going to replace it”. Their response is a bit disappointing.

willy
willy
4 years ago

OK that’s understandable then. Still they’d better make some statements like “oh we know our early samples were not perfect” than let the doubt exist about their hardware.

Eversor
Eversor
4 years ago

I agree with Willy – most manufacturers would want the board back for further testing. Doesn’t give much confidence that they will do proper Q/A.

Avra
Avra
4 years ago

Thank you for reporting MCE problem on Gemini Lake. It helped me not to buy it. I already have MCE errors on Apollo Lake Asrock J3455-ITX and thought that problem will not show on Gemini Lake. Unfortunately it does not seam to be the case… avra@falcon:~$ uname -a Linux falcon 4.19.0-0.bpo.2-amd64 #1 SMP Debian 4.19.16-1~bpo9+1 (2019-02-07) x86_64 GNU/Linux avra@falcon:~$ dmesg | grep microcode dmesg: read kernel buffer failed: Operation not permitted avra@falcon:~$ sudo dmesg | grep microcode [sudo] password for avra: [ 0.961770] mce: [Hardware Error]: PROCESSOR 0:506c9 TIME 1564253320 SOCKET 0 APIC 0 microcode 1e [ 3.731218] microcode: sig=0x506c9,… Read more »

Hugh
Hugh
4 years ago

This is an error in the processor. It could be induced by bad power or inappropriate clock speed. Other than that, it is likely a busted processor chip.

If you are just using the machine for benchmarking, who cares.
Otherwise, I would discard the board.
(I’m assuming that the processor is not socketted.)

The right way of thinking about ECC is as a life preserver. Your boat is sinking, but you can float long enough to get to another boat. You don’t use it to try to keep the old boat floating.

Lithopsian
Lithopsian
2 years ago

The “Drivers not loaded” message is a bit spurious. It just looks in /proc/modules for anything having edac in the name and assumes that is good, regardless of whether it is the right driver for your machine. If it doesn’t see anything, for example if the correct edac driver is built in to the kernel, then it reports that the driver isn’t loaded. So the message has nothing to do with MCE and may or may not be reporting the status of EDAC monitoring.

Khadas VIM4 SBC