An Attempt to Benchmark Entry-level x86 Boards against RK3399 & Exynos Arm Boards

Some Arm boards have become quite powerful, while hardware based on low power Intel processor has generally become cheaper with both architectures somewhat converging in terms of performance and price.

Piotr Maliński got interested and purchased some low cost (<$150) Intel hardware to compare to mid-range Arm boards, throwing a Raspberry Pi 3 B+ into the mix as well for comparison.

Those are the Intel test boards / computers:

  • Qotom motherboard with Intel Atom Z3735F Bay Trail processor, 2GB RAM, 32GB flash – $74 + shipping on Aliexpress
  • Piesia nano ITX board with Intel Celeron N2806 Bay Trail processor, DDR3 SO-DIMM socket, SATA / mSATA interfaces – Piotr found it for around $85 on Aliexpress, but the price now jumped to over $150 plus shipping, which does not make it very attractive
  • Generic thin mini ITX motherboard based on Celeron N3160 “Braswell” processor, DDR3 SO-DIMM socket, SATA / mSATA interfaces. $62.68 shipped on Aliexpress.
  • MSI E350DM-E33 motherboard with “old” AMD E-350 dual core processor, 2x DDR3 SO-DIMM sockets, 6x SATA. Bought second hand for around $25-$30
  • Asrock J5005-ITX motherboard with Intel Pentium J5005 Gemini Lake processor, 2x DDR4 SO-DIMM sockets, 4x SATA III interface. Bought for around $135

Those have be pitted against three Arm SBCs:

  • RockPro64 with Rockchip RK3399 processor, 4GB LPDDR4, micro SD card or eMMC flash module storage – Around $95.
  • ODROID-XU4 with Samsung Exynos 5422, 2GB RAM, micro SD card or eMMC flash module storage – Around $95, which should include shipping and EU taxes.
  • Raspberry Pi 3 B+ – Around $45

He installed Linux on all those platforms and ran several Phoronix benchmarks and GLMark for graphics. The test did not run properly on all platform, which explains why not all 8 boards are not always represented in results.

Arm vs Intel kernel compile timeThe Gemini Lake motherboard is bound to outperform all other competitors in those test since it’s the most recent and expensive. So it’s not a surprise that it comes on top by a wide margin. As expected RockPro64 is quite faster than the Raspberry Pi 3 board. However, there must be a serious problem with the Celeron N3160 motherboard, as it should be faster than Bay Trail processors, and almost as fast as an Atom x7-Z8700 processor, but performs really poorly in all benchmarks run by Piotr. Sp it’s probably a good idea to ignore results about this board for now.

Arm vs Intel C-RayC-Ray floating-point benchmark actually looks quite good on Arm platforms, with the Pentium J5005 “only” about twice as fast. It’s also surprising to see ODROID-XU4 almost equal Pine64’s RK3399 board.

OpenSSL Arm vs IntelWhen benchmarking it’s always important to pay attention to details, and it’s especially the case with OpenSSL benchmarks, as specific instructions are often used for crypto operations like AES. Not all processors support those, sometimes they are not implemented in the code, or compiler flags used do not enable them. For more insights about this, read the insightful comments thread about Arm vs Intel OpenSSL benchmarks in a previous post. Nevertheless, OpenSSL benchmark results for the top three boards look very much like the same as for C-Ray, just ODROID-XU4 is slightly ahead of RockPro64.

OpenGL Arm vs IntelPiotr also attempted to test graphics performance, but it’s particularly hard to compare Arm and x86 SoC because the latter normally rely on OpenGL ES that is a subset of OpenGL, and OpenGL benchmarks may not run on Arm, and if they do they may rely on emulation for missing commands. The latter possibly explains why RockPro64 compares so poorly, beside the faster GPU in the Gemini Lake processor.

GLmark2 ODROID-XU4 RockPro64Comparison of Arm boards with GLMark2 may be more interesting here, as again ODROID-XU4 surprises by outperforming RockPro64. But there’s again something very odd about the scores, as other have reported Raspberry Pi 3 B+ achieving 101 points. Testing graphics performance get even more complicated since graphics “frameworks” differ e.g. framebuffer, X11, Wayland, etc…, and in some cases offscreen benchmarks come into play.

The Linux distributions names and kernel versions, and the compiler flags are important to analyze the results since bugs get fixed, and different compiler flags may yield to fairly different results. You should be able to find those in OpenBenchmarking.org with the full benchmarks results.

Beside the benchmarks, details about each board including photos taken with a thermal camera can be found on Piotr’s blog.

Support CNX Software - Donate via PayPal or become a Patron on Patreon

22
Leave a Reply

avatar
6 Comment threads
16 Thread replies
1 Followers
 
Most reacted comment
Hottest comment thread
11 Comment authors
crashoverridetkaiserLeo LiMemeka Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
blu
Guest
blu

While cpu testing is simple in comparison, gpu testing is a very tricky business — one *really* needs to know what they’re testing. For instance, it’s quite often that a GL(ES) stack might be bottlenecked by a (cpu-based) copy-full framebuffer presentation stage, taking away cycles from the game logic code (if it’s a game, or anything that does sizeable work on the cpu). Alas, the gpu-stack stituation in arm land has been a wild west. One of the reasons my daily arm drive is a chromebook is the predictability and streamlining of the GLES stack.

Member

N3160 performance is low but it could be something with the board having custom TDP set for it. With passive cooling they could crippled it. As of GLMark2 – GPUs those two ARM SoC have according to notebookcheck align like in my test – Exynos has a stronger GPU than RK3399. Plus there had been some artifacts in this test – running OpenGL test won’t be the best case scenario. Better of with Android testing both boards.

blu
Guest
blu

Re gpus, that’s the point, though — they’re significantly more complex than cpus, and are utilized via *way* more complex and abstract APIs, so ‘according to this or that paper spec (read: raw figures)’ means nothing unless you know exactly where the bottlenecks in your benchmarks sit. IFF the bottleneck is in *compute*, and the test aligns well with the older (2nd-gen Midgard) T628MP6 architecture, rather than with the newer (4th-gen Midgard) T860MP4, then *perhaps* the older Midgard could outperform the newer Midgard by some good margin. Otherwise I have gpu benchmarks here where the rk3399 outperforms the exynos5422 by 30% and then some, under a well-tuned presentation stack (wayland). So that 2x glmark2 advantage of the exynos5422 that you’ve registered is not truly indicative of the delta between the two parts.

Memeka
Guest
Memeka

You can always do off-screen tests, they will show you the GPU performance in the absolute – but it will be useless because what good use is a powerful gpu that is crippled by a buggy display server?

blu
Guest
blu

True. And even when doing offscreen tests, desktop GL (less so GLES) allows client code to do some quite stupid things that could tax the cpu beyond the necessart and/or impede cpu/gpu parallelism. Of course one day we’ll all be doing vulkan, but today is not that day. BTW, once again, thank you for your XU4 gnome-wayland image — its’ been quite useful to me.

Mookid
Guest
Mookid

I have no idea about broblems with this board, but support for intel atom class power saving features can seriously cripple the performance. Usually i start testing this type of boards by disabling all power saving features to get a reliable reference point.

crashoverride
Guest
crashoverride

Why not use glmark2-es2 since OpenGL-ES2 is supported natively on all systems?

I also second @Memeka’s comment that “off-screen” tests are useless since an expected primary use of a GPU is to actually see the rendering.

Member

Was limited with what test cases Phoronix had. If there is an ES version I’ll look into it separately.

Tony mac
Guest
Tony mac

The RK3399 and XU4 aren’t surprising, the XU4 has more “fast” cores, and more cores overall.

Member

If you are going to spend more than $100 always pick Intel/AMD since those CPUs are much faster than any ARM offerings. This does assume that the x86 boards are working correctly. Under $100 Intel/AMD can’ compete since they don’t really offer anything in that price category other that older chips.

My two cents, it you want a desktop/laptop with GPU just spend over $100 and get an x86. ARM wins by far for embedded systems that don’t need a huge CPU and battery powered mobile (phones/tablets). Servers are currently a battleground. x86 dominates but there are serious attempts by the ARM people to build competitive chips.

I also don’t expect this landscape to change much in the future. Sure ARM can design chips that are competitive with x86 desktop CPUs. But those chips are going to be just as big and complex as the Intel ones so I don’t except them to be significantly cheaper.

Lemmy
Guest
Lemmy

The Benchmark is not true, Kernel compile time on Raspberry Pi 3 is 6 Hours (21000 seconds, Raspbian)

Member

And why so? Note that phoronix benchmark has a specific kernel it builds. It doesn’t build a fully working kernel for given platform or distro. Just compile the same amount of code wherever it is run.

tkaiser
Guest
tkaiser

> And why so?

Since kernel compilation is also an (random) IO benchmark (same with some ‘DB benchmarks’). Kitchen-sink benchmarks do not differentiate between IO and CPU and thereby produce only numbers without meaning.

Did you check at which clockspeeds the various CPUs were running? Did you check for memory bandwidth and latency?

Member

It’s not really without meaning as it can be re-run to get the same result and thus showcasing system performance for one of possible workloads.

tkaiser
Guest
tkaiser

IMO you need to test individually especially on system where random IO performance can be slow as hell (SD card). That’s the main culprit with all these low-end system comparisons: Apple vs. Oranges since storage is not taken into account (this even heavily influences tasks like ‘running Android’ or ‘Desktop Linux’ — slow random IO write performance and overall performance sucks regardless of any CPU and GPU benchmark results).

ODROID-XU4 with fast eMMC vs. RockPro64 with slow SD card: XU4 wins ‘kernel compilation’ competition easily. Run RockPro64 off a fast NVMe SSD and it’s the other way around. So you need to carefully test the subsystems and then look at the workload or the use case to find a suitable system.

My use case is server workloads so I ended up with one single benchmark that somehow represents this workload: 7-zip –> https://github.com/ThomasKaiser/sbc-bench (the only real advance of running sbc-bench and not the tools individually is the excessive amount of background monitoring to really get an idea what’s going on — if you benchmark a throttled system what’s the meaning of the generated numbers afterwards?)

And wrt IO performance (both sequential and the way more important random IO): https://forum.armbian.com/topic/1925-some-storage-benchmarks-on-sbcs/?do=findComment&comment=51350 — it all depends on the storage capabilities exposed. And with RK3399 and if the four Gen2 PCIe lanes are exposed you’re at the same storage performance level compared to entry-level x86 (at least the Atom based thingies also only have Gen2 PCIe).

Member

Yes, it’s good to know each subsystem capabilities and performance, however at a first glance board consumer will want to know if and how well it can run software he/she wants it to. Can it run a database?, a desktop fluently?, can it run Skyrim? some sort of imperfect software… after that it can be beneficial to go down to a lower level to see what’s the limit or where performance can be gained.

I still have those boards and I’m waiting for eMMC module plus I already have PCIe/NVMe adapter that can be used with RockPro64. I will be testing them a lot 🙂 I used a SD card in this case as it’s the cheapest and most commonly used solution (while also trying to pick a better one from SanDisk cards known for better than average random I/O, especially a A2 rated one).

tkaiser
Guest
tkaiser

> SanDisk … especially a A2 rated one

Based on my experiences not the best idea currently since A2 SanDisk cards are slower than their older A1 siblings: https://github.com/ThomasKaiser/Knowledge/blob/master/articles/A1_and_A2_rated_SD_cards.md

Leo Li
Guest
Leo Li

If performance of XU4 are nearly same as RK3399, then what will be the result of ODROID-N2 ? Other than hardware accelerated video decoding, it is pointless to buy RK3399?

tkaiser
Guest
tkaiser

You should really read the article above and the link to comments section where it’s discussed how meaningless some (most) of the Phoronix ‘benchmarks’ are (especially true for the OpenSSL ‘benchmark’).

Then you should always keep in mind that you test software and not hardware even if kitchen-sink benchmarks like Phoronix try to create another impression.

Numbers vary based on settings and versions and even ‘events’ (see Spectre/Meltdown mitigations and their performance impacts). Essentially you benchmark software and need a lot of more work to get an idea how that relates to the underlying hardware capabilities.

In case of RockPro64 vs. ODROID-XU4 you compare software not yet optimized with a mature software stack that has also been tuned for performance already. Apples and Oranges.