Checking Out Machine Check Exception (MCE) Errors in Linux

Machine Check Exception Error Linux

I recently reviewed ODROID-H2 with Ubuntu 19.04, and noticed some errors messages in the kernel log of the Intel Celeron J4105 single board computer while running SBC-Bench benchmark:


I did not know what do make of those errors, but I was told I would get more details with mcelog which can be installed as follows:


There’s just one little problem: it’s not in Ubuntu 19.04 repository, and a bug report mentions mcelog is not deprecated, and remove from Ubuntu 18.04 Bionic onwards. Instead, we’re being told the mcelog package functionality has been replaced by rasdaemon.

But before looking into the utilities, let’s find out what Machine Check Exception (MCE) is all about from ArchLinux Wiki:

A machine check exception (MCE) is an error generated by the CPU when the CPU detects that a hardware error or failure has occurred.

Machine check exceptions (MCEs) can occur for a variety of reasons ranging from undesired or out-of-spec voltages from the power supply, from cosmic radiation flipping bits in memory DIMMs or the CPU, or from other miscellaneous faults, including faulty software triggering hardware errors.

Hardware error should probably be taken seriously. Let’s investigate how to run the tools. First, I try to install mcelog from Ubuntu 16.04:


Oh good! It could install… Let’s run some commands:


Nothing interesting shows up here, but the file /var/log/mcelog is now up, and we can see details about the errors:


But let’s also try the recommended rasdaemon to see if we can get similar details.

Installation:


It looks like the service will not start automatically upon installation, so a reboot may be needed, or simply run the following command:


I ran a few commands and at first, it looked like some driver may be needed:


This should be related to EDAC drivers that are used for ECC memory according to a thread on Grokbase. Gemini Lake processors do not support ECC memory, so I probably don’t need it.

Running one more command to show the summary of errors, and we’re getting somewhere:


12 corrected error related to the L2 cache. We can get the full details with the appropriate command:


The status is green which means everything still works, but the utility reports a “large number of corrected cache errors”, and the “system (is) operating, but might lead to uncorrected errors soon” (See source code). It happens only a few times a day, and I’m not sure what can be done about the cache since it’s not something that can be changed as it’s embedded into the processor, maybe it’s just an issue with the processor I’m running. If somebody has an ODROID-H2 running, it may be useful to check out the kernel log with dmesg to see if you’ve got the same errors. If you do, please also indicate whether you have a board from the first batch (November 2018) or one of the new ODROID-H2 Rev B boards.

Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.

Support CNX Software - Donate via PayPal, become a Patron on Patreon, or buy review samples
Subscribe
Notify of
guest
36 Comments
oldest
newest most voted
tkaiser
tkaiser
1 year ago

> the file /var/log/mcelog is now up

That’s how we monitor x86 commodity hardware: installing the mcelog package, then defining a primitive rule watching the size of /var/log/mcelog. Once the size exceeds 0 you have a problem.

As for the occurrences of these 2nd level cache errors: most probably they only occur once data is pumped through the CPU cores (e.g. running 7-zip or cpuminer as part of sbc-bench for example).

theguyuk
theguyuk
1 year ago

Might be worth also posting on Odroid H2 forum, issues or Ubuntu, maybe raise the number of board users reading this?

theguyuk
theguyuk
1 year ago

Seem to be incompatible memory posts on Odroid forum. Could it be memory or hardware chips driver issues?

willy
willy
1 year ago

Cache issues in CPUs do happen. I’ve been hit many times with those in UltraSparc processors for example (and my U5 is sick again due to this for the second time by the way). I think modern CPUs use parity to better resist issues caused by heat and other random events. It’s “simple” enough to drop a line and re-read it when reading inconsistent data (it’s more difficult when changes are present but nothing prevents from re-reading the faulty word if it was not modified). In case that’s what you’re facing, there could be a corner case of locally-modified data… Read more »

Laurent
Laurent
1 year ago

Some Intel chips go beyond parity checking and have ECC on L2. But I don’t know if this chip supports it, Intel doesn’t seem to document that openly.

Found a command to check that: sudo dmidecode and look for “Error Correction Type”. On my i7-8650U it shows L1D is parity protected, L2 is one-bit ECC and L3 it multi-bit ECC.

willy
willy
1 year ago

Yes I’m pretty sure that high-end chips do have ECC, but here we’re talking about an Atom basically. Well, marketingly speaking it’s a celeron 🙂

Laurent
Laurent
1 year ago

willy, you’re correct for DRAM ECC, here we’re talking about the internal caches 😉

willy
willy
1 year ago

Laurent, I was also speaking about internal caches 🙂 I’m sure I’ve seen it mentioned a few times in the past for Xeons and have some memories of seeing things like “L2 cache ECC” in some server BIOSes in the past. Note that it might have been quite old since my memory associates this to L2 not L3. Also IBM’s POWER8 and above definitely use ECC for L2 & L3.

willy
willy
1 year ago

The link doesn’t work here, it displays “privatebin is a minimalist …”.

willy
willy
1 year ago

Tried already, and again, but same result. It’s no big deal though, don’t worry 🙂

Eversor
Eversor
1 year ago

Hi there,

This is not normal. One error once in a couple months due to cosmic rays, that is normal.

You are being warned because at that rate there is something wrong with the hardware or too much EMI from another device.

tkaiser
tkaiser
1 year ago

> at that rate there is something wrong with the hardware

So far only ECCed (corrected) single bit flips in L2 cache. Harms slightly performance and will only be a real issue once two bits flip at the same time. While I wouldn’t trust such a CPU that much for the average use case a Gemini Lake box is taken (media center, desktop) this shouldn’t matter that much.

David Willmore
David Willmore
1 year ago

Physics is physics. You’re going to get bit flips in caches. Beta particles from C-14 decomposition. Beta particles from the Si itself, etc. Then there’s cosmic rays that you just can’t avoid as they’re everywhere. As transistors get smaller and smaller, Johnson–Nyquist noise becomes a bigger issue. The error may have even occured due to a transmission error on the chip–the right value may have been stored, but it got corrupted between the storage element and the ECC block. As long as the ECC is catching single bit errors at a relatively low rate, there’s noting to worry about. If… Read more »

willy
willy
1 year ago

Sometimes there are 4 in the same second, this is far too much, something is busted in this machine, and its ability to recover from all the events you enumerated is affected by this existing one. The probability of two-bit *unrelated* errors remains very low. But if the hardware is defective, the probability of two-bit errors in the same cache line cannot be dismissed.

Eversor
Eversor
1 year ago

That is usually coil whine and is not the issue.

I’d try setting a lower maximum clock and see if the problem goes away. I’d want to check if power regulation/supply isn’t deficient.

If the problem is the hardware, a torture test like Prime 95 will also make it much worse.

David Willmore
David Willmore
1 year ago

I don’t use an x86 MB/memory combo without first doing at least 24 hours of memtest86 and then Prime95 on torture test. That helps validate memroy, processor, power delivery, etc.

tkaiser
tkaiser
1 year ago

> I’ve gone to the BIOS and changed two settings

And you lost around 1/3 of the CPU performance 🙂

Smells a bit like unstable high DVFS OPP…

Eversor
Eversor
1 year ago

Yeah, any of the modes be the cause of this unless they’re bugged. I agree with Kaiser it’s DVFS bug probably. Could also be something on the hardware design – like capacitors – which can’t handle higher current swings.

Eversor
Eversor
1 year ago

BTW, I once had an issue with a motherboard BIOS where it would have correctable and uncorrectable L1/L2 errors when doing light loads.

The board was initially K8 only but eventually supported Phenom 1 and 2.

Somewhere something broke, as the motherboard induced errors if I enabled frequency scaling on the K8 but worked fine with a Phenom II – which used a different version of C&Q.

K8 CPU worked fine if set at any fixed speed, just couldn’t change on demand. Prob was changing clocks too fast, without letting the VRM stabilize at the higher voltages first.

theguyuk
theguyuk
1 year ago

Have Odroid replied yet?

willy
willy
1 year ago

We could have hoped better, like “given that yours shows problems we cannot reproduce it definitely indicates a hardware issue and a risk of accelerated aging, we’re going to replace it”. Their response is a bit disappointing.

willy
willy
1 year ago

OK that’s understandable then. Still they’d better make some statements like “oh we know our early samples were not perfect” than let the doubt exist about their hardware.

Eversor
Eversor
1 year ago

I agree with Willy – most manufacturers would want the board back for further testing. Doesn’t give much confidence that they will do proper Q/A.

Avra
Avra
1 year ago

Thank you for reporting MCE problem on Gemini Lake. It helped me not to buy it. I already have MCE errors on Apollo Lake Asrock J3455-ITX and thought that problem will not show on Gemini Lake. Unfortunately it does not seam to be the case… [email protected]:~$ uname -a Linux falcon 4.19.0-0.bpo.2-amd64 #1 SMP Debian 4.19.16-1~bpo9+1 (2019-02-07) x86_64 GNU/Linux [email protected]:~$ dmesg | grep microcode dmesg: read kernel buffer failed: Operation not permitted [email protected]:~$ sudo dmesg | grep microcode [sudo] password for avra: [ 0.961770] mce: [Hardware Error]: PROCESSOR 0:506c9 TIME 1564253320 SOCKET 0 APIC 0 microcode 1e [ 3.731218] microcode: sig=0x506c9,… Read more »

Hugh
Hugh
1 year ago

This is an error in the processor. It could be induced by bad power or inappropriate clock speed. Other than that, it is likely a busted processor chip.

If you are just using the machine for benchmarking, who cares.
Otherwise, I would discard the board.
(I’m assuming that the processor is not socketted.)

The right way of thinking about ECC is as a life preserver. Your boat is sinking, but you can float long enough to get to another boat. You don’t use it to try to keep the old boat floating.

Advertisements