Home > Allwinner H-Series, Debian, Intel Atom, Intel Celeron, Linux, Rockchip RK33xx, Testing, Ubuntu > Linux Benchmarks – Intel J3455 Apollo Lake vs Z3735F Bay Trail vs RK3399 and Other ARM Platforms

Linux Benchmarks – Intel J3455 Apollo Lake vs Z3735F Bay Trail vs RK3399 and Other ARM Platforms

Since I’ve just installed Ubuntu 17.10 on MeLE PCG35 Apo, I decided I should also run some benchmarks comparing with other ARM and x86 Linux platforms I’ve tested in the past.I was particularly interested to compare the performance of Intel Apollo Lake processors (Celeron J3455 in this case) against higher end ARM processors like Rockchip RK3399 (2x A72, 4x A53) since systems have a similar price (~$150+), as well as against the older Bay Trail processor to see the progress achieved over the last 2 to 3 years.

To do so, I used Phoronix Benchmark Suite against Videostrong VS-RK3399 results (RK3399 development board):


The benchmark first issued a warning about “powersave” governor, but I still went ahead, and once completed I change it to “performance” governor:


…and ran the tests again. All results are available on OpenBenchmarking.

Let’s address the governor results first. cpufreq-info reports that powersave governor can also switch between 800 MHz and 2.30 GHz (turbo freq).


As we’ll see from the results below pitting “MeLE PCG35 Apo – Ubuntu 17.10” (with powersave) and “MeLE PCG35 Apo- Ubuntu 17.10 Performance” that the governor settings did not matter one bit on the results, at least for the six benchmarks I ran.

Note that “MeUbuntu 14.04.3” represents MeLE PCG02U TV stick running Ubuntu 14.04.3. Every platform runs a different OS and kernel, so keep in mind the results may differ slightly (up or down) with different version. But as we’ll see the differences in performance are large enough that it likely does not matter that much.
John the Ripper password cracker, a multi-threaded benchmark, shows the Apollo Lake processor is clearly ahead of Rockchip RK3399 hexa-core processor, and the fastest ARM platform, Banana Pi M3, is equipped with an Allwinner A83T octa-core Cortex A7 processor @ 2.0 GHz. The Bay Trail system is over  twice as slow as the Apollo Lake one, also note the larg-ish standard deviation (+/- 83.72) due to some cooling problem in the small form factor.

C-Ray is another multi-threaded benchmark, and here Rockchip RK3399 SoC does fairly well, but still but quite as well as the Celeron J3455.

Smallpt, yet another multi-threaded benchmark, does not really change the order with MeLE PCG35 Apo well ahead.

Himeno, a linear solver of pressure Poisson, must be using some x86 specific instructions or optimizations, as Intel platforms are well ahead, with Celeron J3455 about 2.5x faster than Rockchip RK3399 board.

OpenSSL is the domain of Intel platforms likely benefiting from Advanced Encryption Standard instruction set (AES-NI). Performance improvement between Bay Trail and Apollo Lake is also impressive here. You’d need 10 Raspberry Pi 3 to match MeLE PCG35 Apo in this particular test.


Intel is normally better with SIMD accelerated multimedia application, and FLAC audio encoding (single threaded) confirms that.

I was expecting a close fight between Rockchip RK3399 and Celeron J3455, but RK3399 only has two fast Cortex A72 cores against four x86 cores in the Intel Apollo Lake SoC.

 

  1. tkaiser
    October 26th, 2017 at 22:16 | #1

    Two small remarks:

    1) On Intel systems with intel_pstate driver running (Sandy Bridge or above and obviously a somewhat recent kernel) powersave and performance are expected to give similar results in synthetic benchmarks (since ramping up CPU clockspeeds fast enough). But on other platforms depending on kernel version powersave vs performance can result in huge performance differences (since powersave can lead to the CPU cores remaining on lowest allowed clockspeed all the time while performance chooses the maximum as on Intel)

    2) OpenSSL performance and AES-NI:if we’re talking about AES here then ARMv8 SoCs with AES crypto extensions (like RK3399 that is missing above — why?) should be able to compete. At least when we (Armbian community) did some tests a while ago single threaded Banana Pi M3 ‘performance’ at 1.8 GHz (Cortex-A7) was ~30 times lower compared to a Cortex-A53 at 1.3 GHz with ARMv8 crypto extensions enabled.

  2. willy
    October 26th, 2017 at 22:28 | #2

    For now I’ve been quite disappointed by RK3399’s performance. I managed to make my build farm run on the H96max at 1.8 GHz (A72) + 1.4 GHz (A53), and at 1.8 GHz, the A72 shows slightly lower performance than the RK3288 on the MiQi (ie 16.2 sec instead of 16.0 for a build). And that’s the best I can take out of it, it’s in 32-bit (armv7t or armv8l). In 64-bit (armv8), it’s about 10-15% slower for the same task. It was said that the A72’s architecture was very close to the A17, the former with 3-decode and 5-issue, the latter with 2-decode and 5-issue, but I suspect we’re caping out on the ALU after the instructions are decoded so the A72 isn’t faster in the end. By the way the test using JtR above shows the 4-core MiQi 35% faster than the 6-core RK3399, so that tends to confirm that there’s definitely a corner case where it doesn’t shine unfortunately.

  3. blu
    October 26th, 2017 at 23:37 | #3

    @willy
    Those settop boxes traditionally have notoriously bad thermal design, and a 28nm A72 can get really hot at near-2GHz — have you monitored the throttling?

  4. Manuel
    October 26th, 2017 at 23:51 | #4

    Why not add a comparative about power consumption? It would be interesting…

    Thanks.

  5. theguyuk
    October 27th, 2017 at 01:25 | #5

    Unless it is like for like hardware the results have no place in the real world.

    Most TV boxes watch TV and comparing a £30 less board against a £130 + Intel J3455 Apollo Lake. Is like racing a child’s peddle bike against a F1 Racer. Funny but no value.

    How about benchmarking a Xbox One against a Intel J3455 Apollo Lake? It would have just as much real world use.

  6. willy
    October 27th, 2017 at 01:27 | #6

    @blu
    Yes absolutely, and it’s really at 1.8 GHz while running such tests, and doesn’t throttle. BTW the advertised 1.5 and 2.0 GHz frequencies are not even part of the frequencies list at all, that’s the beauty of “up to”, it probably means that it can go “up to 2.0 GHz if you hack your DTB”.

    Honestly the perf is not bad at all and it heats less than the 3288 at the same performance level, it just doesn’t have enough cores to compete with it. The RK3288 has 4 strong cores, but the 3399 has only two, and the 4 remaining A53 are just jokes compared to them. Basically you have less than an odroid C2 sharing the cache and RAM bandwidth with the two A72. It’s fine to run SSH or probably to run background music on a tablet but that’s almost all, as they’re less than half of the performance of the large ones.

    The A75 is said to be a real performer. We’ll see when it’s available. For now I’m sticking to the A17.

  7. bob
    October 27th, 2017 at 03:09 | #7

    ARM is not power efficient, it’s just cheap in some TV box :

    50$ A53 x4 cpu burn = 5Watt
    200$ intel I3 5005U cpu burn = 15-20W
    Intel is 10x more powerfull in benchmark, so 3X more efficient benchmark pt / watt.
    Conclusion expensive Arm is a no go

  8. tkaiser
    October 27th, 2017 at 03:38 | #8

    @willy
    I don’t get it. Your workload scales horizontally? And 2 x A72 are as fast as 4 x A17?

  9. theguyuk
    October 27th, 2017 at 04:09 | #9

    @bob
    The real fact is most Arm SoC spend most of the time not running at the claimed top speed, they can only run very sort bursts.

    Even over designed cooler cannot help.

  10. blu
    October 27th, 2017 at 04:44 | #10

    @bob
    There are no quad-core A53 that a 5005U would be 10x as powerful — the average IPC difference between Broadwell and A53 is approx 2-2.5 in favor of the former. There might be isolated cases where AVX2-centric scenarios can produce 10x difference, but how common are they?

    btw, 410c is a severely underclocked SBC — its normal ‘stock-cooling’ clock is ~600MHz IIRC.

  11. bob
    October 27th, 2017 at 06:49 | #11

    @theguyuk
    Right about idle/low use cpu need, for that i don’t understand strategy of arm don’t provide hw encoder/decoder driver for a good power efficiency…it’s a shame for their business

    And maybe we miss a compilation souce code benchmarck, because i don’t think my little H3 compile 10X slower than my intel cpu…

  12. Jonathan
    October 27th, 2017 at 06:57 | #12

    What are the RAM speeds and configurations on these boxes? I suspect that may tell as much or more of the story, especially if the benchmark is blowing out the icache on the ARMs.

  13. October 27th, 2017 at 09:16 | #13

    @theguyuk
    As mentioned in the post RK3399 and Apollo Lake devices are in the same price range.

  14. October 27th, 2017 at 09:18 | #14

    @tkaiser
    OpenSSL benchmark is missing on RK3399 because the test failed. If I remember correctly, the download failed.

  15. tkaiser
    October 27th, 2017 at 13:32 | #15

    @Jonathan
    The Phoronix test suite as well as the results on openbenchmarking.org should be considered useless garbage especially when trying to compare different platforms in the way it’s done here. Most of the ‘benchmarks’ use questionable or no optimization settings at build time so results vary a lot with the OS used and the platform it’s running on (compiler shipped and default settings).

    I’ve seen some of these ‘benchmarks’ performing 3 times faster on the same hardware just by exchanging the OS and thereby default compiler and settings (switching from GCC 4.x to 6.x for example).

  16. RooTer
    October 27th, 2017 at 14:44 | #16

    @theguyuk
    when thinking about (relatively) low power server I actually do think whenever should I go with arm and x86 and such benchmark did made sense for me.
    You can argue like other commenters the metric used was not the best, but the idea itself? no. Look at the scores, they are clearly comparable, especially if you think about the price involved for the performance.

  17. tkaiser
    October 27th, 2017 at 14:54 | #17

    RooTer :
    @theguyuk
    when thinking about (relatively) low power server I actually do think whenever should I go with arm and x86 and such benchmark did made sense for me.

    Which kind of server task(s) could be represented by the collection of synthetic benchmarks above? Really curious 🙂

  18. Eversor
    October 27th, 2017 at 15:30 | #18

    @bob
    That may be true for these cheap implementations of A53 cores. Have a look at Nvidia Jetson TX2 benchmarks. Phoronix did some a good while back. Same is true for Apple’s cores.

  19. m][sko
    October 27th, 2017 at 15:52 | #19

    My results for Hikey 960
    https://pastebin.com/twarU4eC

  20. October 27th, 2017 at 16:07 | #20

    @m][sko
    Himeno Benchmark 3.0 -> 1033.08
    That’s really a big gap compared to other ARM platforms (5x faster than RK3399), and even faster than the Apollo Lake processor.
    I wonder what may have caused the jump beside just the faster cores.

  21. tkaiser
    October 27th, 2017 at 16:17 | #21

    @m][sko
    Can you please provide output from

  22. blu
    October 27th, 2017 at 18:02 | #22

    @cnxsoft
    Those benchmarks are extremely sensitive to compiler version and flags. One wrong flag and/or compiler version can make all the difference. It’s essential when doing synthetic benchmarking to know very well what a given compiler produces from the sources (e.g. ‘Hey, it’s not generating AES instructions!’, ‘Hey, it produced this snafu in the inner-most loop!’, etc)

  23. raul
    October 27th, 2017 at 19:31 | #23

    Arm boards for the general use case have lost their initial promise. Lack of support from Arm, lack of proper software support from vendors, and the throttling and performance has left the promise in tatters.

    And the cost of higher performing boards, even the 3399, takes one into low power lower priced Intel platforms at which point its game over as the software support on Intel just cannot be compared.

    But for boards like the Pi or Odroid with better software support, at least things like a low power HTPC makes sense. And if the SOC provider decides to provide support, sometimes even a NAS or router can become possible. But on the whole the ecosystem showed early promise but years later its not really gone anywhere,

  24. willmore
    October 27th, 2017 at 20:30 | #24

    @raul
    I would say that there are vendors who don’t suffer the problems you mention. ODROUD from Hard Kernel for one provides boards without cooling problems. They support their boards with software.

  25. tkaiser
    October 27th, 2017 at 20:36 | #25

    blu :
    Those benchmarks are extremely sensitive to compiler version and flags. One wrong flag and/or compiler version can make all the difference.

    And that’s the main reason no one with a brain in his head should use this terrible Phoronix crap. I used Jean-Luc’s last results to rely on and tested on a ROCK64: https://openbenchmarking.org/result/1710279-TY-1710254TY63

    According to the Phoronix results ROCK64 shows just 41% OpenSSL performance compared to the octa core Banana Pi M3 and less than 9 percent compared to the Apollo Lake box (obviously the latter making use of AES-NI here). So obviously the Phoronix ‘benchmark’ uses weird compiler flags and makes no use of ARMv8 crypto extensions. Let’s have a closer look and compare the openssl binary Phoronix built from sources with the one the distro contains:

    So Phoronix managed to build the binary that strangely that it performs 2.4 times lower than the distro package every real-world task will use. But that’s still not talking about ARMv8 crypto extensions, so let’s have a look too (full command output for both tests — check especially build options!):

    Hey, the Ubuntu package that every real-world task on this device will use is just 4 times faster with really small chunks of data and up to 20 times with 8K data blocks. Srsly? Why do we use a ‘benchmark’ suite that is well known since years to only produce numbers without meaning especially when trying to compare different architectures?

    @cnxsoft
    In case you have the Apollo Lake box still running with Ubuntu… care to repeat just those 4 OpenSSL runs (one time using the distro’s openssl binary and one time the one Phoronix built. I’m especially interested in the build options reported since I’m wondering why AES-NI is used on Intel for the sign/verify benchmark)

  26. October 27th, 2017 at 20:42 | #26

    @tkaiser
    OpenSSL testes on MeLE PCG35 Apo:

    Not sure how to do that with Phoronix Suite yet. Going to bed soon.

  27. October 27th, 2017 at 20:47 | #27

    Found out how (.phoronix directory). Openssl Phoronix:

  28. tkaiser
    October 27th, 2017 at 21:01 | #28

    @cnxsoft
    Thank you. So on x64 we’re talking about:

    With both tests the PTS binary scores even better than Ubuntu’s (but on Intel totally different compiler flags are used and those make the real difference) while on ARM the PTS binaries score 2.4 to 20 times lower than the Ubuntu distro package. So when switching from stupid passive benchmarking mode to active benchmarking an interesting approach would be to analyze compiler flags above, then rebuild then openssl binary and only stop when results with active ARMv8 crypto extensions outperform the AES-NI numbers.

    At least it should be obvious that this Phoronix crap is not an option to compare performance of different architectures since it ignores everything that’s important and is only able to provide a bunch of numbers without any meaning.

  29. blu
    October 27th, 2017 at 21:16 | #29

    @tkaiser
    Nice catch. Yes, that’s part of what I was talking about — in this particular case what phoronix has produced is literally apples to shotshells.

  30. theguyuk
    October 28th, 2017 at 00:09 | #30

    I know lots will think I am nuts for suggesting it, but if you want TV boxes and ARM SBC to improve, people need to be encouraged to play more 3D games on them.

    Just look how demanding 3D games have pushed Game Console and Gaming PC hardware. The same market can push ARM hardware.

    Does the average office really need a i7, i5 to write letters etc, no.

    Network gear is a different market same as financial data, Movie editing etc

  31. willy
    October 28th, 2017 at 01:06 | #31

    @tkaiser
    No Thomas, I meant 2*A72 are exactly as fast as 2*A17, or about half as fast as 4*A17. Yes the build workload scales reasonably well (provided the memory bandwidth is there and the number of files to build is high enough and these files are about the same size). So for this workload, the RK3288 is almost twice as fast as the RK3399 just because it has twice the number of really usable cores.

  32. willy
    October 28th, 2017 at 01:13 | #32

    @bob
    Power efficiency doesn’t compare like this because it decreases with peak performance. You need to compare the power drawn by a device capable of achieving a given peak performance. For example, my RK3288, while less power efficient than my Atom 8350, is significantly faster (15-20%). These two are the only ones in this thermal envelope capable of delivering more or less comparable performance. If I want higher performance on x86, I have to seek a significantly larger design (a much faster one) which will eat more power than the RK3288. It might be a bit more power efficient but the reality is that this efficiency isn’t interesting anymore due to extra cooling solutions.

    Also the price per task is important in certain environments, just like the price per peak MIPS in other ones. In my case, for a build farm, I have to find a sweet spot between the power envelope, the price and the performance. With a very high price I can find a high frequency core i7 in a small thermal envelope being 3 times faster than my RK3288 but it will cost 20 times the price and will need more cooling.

  33. theguyuk
    October 28th, 2017 at 04:53 | #33

    I always understood that the main trick Intel use, is to do more things in memory, and have as much of the data as you can, as close to the CPU cores as possible.

    Bigger faster cache and faster memory?

  34. willy
    October 28th, 2017 at 10:16 | #34

    @theguyuk
    That’s true for high end CPUs, when the cache algorithm are very advanced, allowing almost instant access to any memory location. But having compared memory performance between a few Atoms and the RK3288 in dual-channel configuration shows a significant difference in favor of the latter on in-cache and RAM patterns.

    One feature helping x86 CPUs is the trace cache replacing the instruction cache. The principle is to store decoded and fused instructions, saving a few pipeline stages for hot code paths. This also allows some old CISC instructions (eg: “rep movs” for memcpy()) to be expanded to very efficient sequences that even possibly give hints to the cache about what lines to prefetch next, and when to flush the written ones, which is hardly possible using more RISC-like instruction sets.

    The branch prediction on modern x86 is very efficient as well, inheriting from certain history mechanisms that once made the Alpha shine. These are typically the stuff that you observe on high performance calculations, crypto and such stuff.

  35. wzyy2
    October 28th, 2017 at 11:27 | #35

    RK3399 is more like Celeron N series, since Celeron J series usually have a higher power consumption.

  36. tkaiser
    October 28th, 2017 at 14:54 | #36

    @blu
    Well, almost all of those Phoronix ‘benchmarks’ are totally broken since how they’re built and executed is not with providing useful numbers in mind but only focusing on ‘portability/compatibility’ so that PTS users being tricked into believing they would do serious benchmarking get all those tests compiled on their hosts regardless of platform, OS and environment.

    Isn’t it really ‘funny’ when I’m interested in encryption performance of an inexpensive ARM thingie and compare with a way more expensive Intel box that Phoronix is telling me the ARM thing would be 11 times slower since the ‘test suite’ only produces BS numbers while using OpenSSL’s own benchmark mode avoiding ‘PTS crippled’ binaries ARMv8 crypto extensions seem to outperform AES-NI:

    Whether/how these synthetic benchmarks relate with real-world encryption workloads like VPN endpoints/gateways or full disk encryption is another story — but if I’m told the ARM platform would be 11 times slower than the Intel box (while the official benchmark with normal openssl binaries shows quite the opposite)… who would start to even look into details?

    While I fully understand that clueless people prefering data over information and like staring at nice looking graphs made of numbers without meaning truly love the Phoronix test suite I don’t understand why serious blogs like this here rely on the useless garbage numbers this tool creates, especially in situations where the PTS fails most: comparing different CPU architectures.

  37. October 28th, 2017 at 15:14 | #37

    @tkaiser
    Phoronix Suite shows the CFLAGS / CXXFLAGS used for compilation. For example for OpenSSL:

    I’d assume all the rest is default. That would mean in the case of some of the results above that AES instructions are enabled by default for x86, but disabled for ARM (maybe -march=armv8-a is required to enable those?).

  38. blu
    October 28th, 2017 at 19:07 | #38

    @willy
    Re x86 i-cache — Atoms, at least up to Silvermont, don’t have either uop cache or trace cache (latter was something found in Netburst; former appeared in SNB). The only i-cache optimisation employed by Atoms (read: Silvermonts) is op boundary marks in the i-cache, and thus Atoms are often decode-bound (one of the reasons the Cortexes fair so well against them).

  39. tkaiser
    October 28th, 2017 at 19:12 | #39

    @cnxsoft
    Yes, PTS is using $some defaults in ‘fire and forget’ mode. Sometimes build settings are hard encoded in the makefile, sometimes relying on what the environment defines (so if for example EXTRAOPTFLAGS=’-O3′ in /etc/profile some ‘benchmarks’ will show magnitudes better scores later, eg. the infamous Smallpt stuff) and sometimes just using what the operating system’s default compiler does for whatever reasons.

    Please think about again: what PTS reports as ‘OpenSSL’ performance is RK3328 being 11 times slower than the J3455. While with another benchmark and when using a sanely built openssl binary the ARM SoC easily outperforms the Intel Celeron. Sorry, but this is not serious benchmarking. This is just what the PTS is well known for: a collection of numbers without meaning.

    And this does not only apply to architecture comparisons (eg. ARM vs. Intel) but also to ‘benchmarks’ comparing membes of the same architecture. Your comparison above of Apollo Lake vs. Bay Trail is more importantly a comparison of Ubuntu 14.04 with 17.10 –> GCC 4.8.4 vs. GCC 7.2 — it is well known how different some of the Phoronix ‘benchmarks’ score depending on GCC version. Publishing such numbers as above definitely does not help getting an idea about ‘hardware performance’. Not with the way Phoronix creates numbers.

  40. October 28th, 2017 at 19:17 | #40

    If you are interested: This is how it looks on a ODROID XU4 ( Exynos5422 Cortex™-A15 2Ghz and Cortex™-A7 Octa core CPU ) on a Ubuntu 16.04.3 LTS – Kernel 4.13.0

    # openssl speed -elapsed -evp aes-128-cbc
    You have chosen to measure elapsed time instead of user CPU time.
    Doing aes-128-cbc for 3s on 16 size blocks: 14966032 aes-128-cbc’s in 3.00s
    Doing aes-128-cbc for 3s on 64 size blocks: 4189907 aes-128-cbc’s in 3.00s
    Doing aes-128-cbc for 3s on 256 size blocks: 1104424 aes-128-cbc’s in 3.00s
    Doing aes-128-cbc for 3s on 1024 size blocks: 279763 aes-128-cbc’s in 3.00s
    Doing aes-128-cbc for 3s on 8192 size blocks: 35038 aes-128-cbc’s in 3.00s
    OpenSSL 1.0.2g 1 Mar 2016
    built on: reproducible build, date unspecified
    options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr)
    compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,–noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
    The ‘numbers’ are in 1000s of bytes per second processed.
    type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
    aes-128-cbc 79818.84k 89384.68k 94244.18k 95492.44k 95677.10k

    # openssl speed rsa4096 -multi 4
    OpenSSL 1.0.2g 1 Mar 2016
    built on: reproducible build, date unspecified
    options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr)
    compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,–noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
    sign verify sign/s verify/s
    rsa 4096 bits 0.014487s 0.000225s 69.0 4438.8

    Since it’s a 8 Core system, I also add the relevant 8 thread result:

    # openssl speed rsa4096 -multi 8

    sign verify sign/s verify/s
    rsa 4096 bits 0.009657s 0.000149s 103.6 6698.3

  41. nobitakun
    October 28th, 2017 at 19:58 | #41

    I’m sorry but talking about efficiency comparing two different fabs is nonsense, each platform is designed for what they are, everything said.

  42. tkaiser
    October 28th, 2017 at 20:40 | #42

    Danand :
    This is how it looks on a ODROID XU4

    Same numbers as in the referenced link in my first comment above (we tried to collect numbers from A7, A9, A15, A17, A53 — with and without ARMv8 crypto stuff — and A72. Currently A17 and A72 still missing).

    With RSA signing the Exynos doesn’t look that bad but when it comes to AES encryption the SoC is far behind compared to those SoCs with ARMv8 crypto extensions, both considering ‘raw performance’ and especially ‘performance per Watt’. An ODROID XU4/HC1 doing AES stuff on the CPU cores compared to an ARMv8 SoC with crypto extensions enabled is ~40 times as inefficient if we keep consumption increase also in mind (details in the aforementioned link to Armbian forum):

  43. blu
    October 29th, 2017 at 12:44 | #43

    @tkaiser
    Let me know if and what A72 benchmarking you need – I have one macchiatobin idling here.

  44. tkaiser
    October 29th, 2017 at 15:38 | #44

    @blu
    Would be great to get full output from

    Compiler and version the binaries were built with are important of course. And if you can results from github.com/ssvb/tinymembench would also be great.

  45. willy
    October 29th, 2017 at 22:24 | #45

    @tkaiser
    On H96-Max, I get this on the 2 [email protected] GHz :

    # openssl version -a
    OpenSSL 1.0.2k 26 Jan 2017
    built on: reproducible build, date unspecified
    platform: linux-aarch64
    options: bn(64,64) rc4(ptr,char) des(idx,cisc,16,int) idea(int) blowfish(ptr)
    compiler: aarch64-gcc47l_glibc218-linux-gnueabi-gcc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -O3 -Wall -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM
    OPENSSLDIR: “/usr/share/openssl”
    (built on linaro gcc-4.7.4, not rebuilt since).

    # taskset -c 4,5 openssl speed rsa4096 -multi 2
    sign verify sign/s verify/s
    rsa 4096 bits 0.019186s 0.000257s 52.1 3887.3

    And this on the 4 A53 cores at 1.4 GHz :
    # taskset -c 0-3 openssl speed rsa4096 -multi 4
    sign verify sign/s verify/s
    rsa 4096 bits 0.022229s 0.000310s 45.0 3229.1

    For AES, the numbers are pretty good :
    A72:
    type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
    aes-128-cbc 465434.04k 938648.81k 1221223.85k 1300021.93k 1349861.38k

    A53:
    type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
    aes-128-cbc 170812.96k 464463.87k 783981.14k 982121.81k 1059921.92k

    On the MiQi (at 1.8 GHz as well) :
    # openssl speed rsa4096 -multi 4
    rsa 4096 bits 0.016024s 0.000230s 62.4 4342.0
    # openssl speed rsa4096 -multi 2
    rsa 4096 bits 0.031990s 0.000463s 31.3 2159.8

    Here the 64-bit architecture definitely helps!

  46. blu
    October 29th, 2017 at 23:53 | #46

    Model: MACCHIATOBin-8040
    Clock: CPU 1300 [MHz]
    DDR 800 [MHz]
    FABRIC 800 [MHz]
    MSS 200 [MHz]

    macchiato:~$ ./gov.sh # columns: core governor clock
    0 performance 1300000
    1 performance 1300000
    2 performance 1300000
    3 performance 1300000

    tinymembench: https://pastebin.com/bg1Wha41
    openssl: https://pastebin.com/WRnVa07L
    7z: https://pastebin.com/Z6RFZLkh

  47. dogs breath
    October 30th, 2017 at 05:01 | #47

    bench mark we want to see is huawei fastest versus apple latest

  48. tkaiser
    October 30th, 2017 at 14:30 | #48

    @blu
    Thank you! This is DDR4, right? As a quick reference Jean-Luc’s tinymembench numbers for RK3399 from few weeks ago. And the 7-zip numbers are the only ‘generic CPU’ benchmark resuilts I need for my use cases (server stuff).

    A72 with ARMv8 crypto extensions performs really impressive, this time also with very small chunks of data (the A53 implementation suffers here somewhat).

    @willy
    I put your AES encryption numbers and the Celeron results together:

    When running on the A72 cores RK3399 outperforms J3455 always and is already 2 times faster with 128 byte chunks. Really nice to see these crypto extensions working so well with AES. Now curious how PCIe performance with this SoC looks like.

  49. blu
    October 30th, 2017 at 16:43 | #49

    @tkaiser
    Yep, DDR4 at the lowest rate (1600) — board and DIMM are capable of 2400, but I keep mine at the lowest setting for minimal active cooling (side-mounted 80mm silent fan).

    On an unrelated note, Teres-A64 acquired! ; ]

  50. Anonymous
    October 31st, 2017 at 22:42 | #50

    @tkaiser

    Maybe file a bug against PTS instead of just complaining on a random article on another site?

    PTS is mostly used/developed on x86_64 so they probably aren’t even aware it doesn’t work properly (much slower than distro defaults) on arm.

  51. willmore
    November 2nd, 2017 at 23:48 | #51

    @Anonymous
    He’s commented over there on a number of articles when this topic has come up. Nothing has come from it. Michael shows no interest in fixing PTS.

    So, instead @tkaiser is addressing it where it’s used so that people there can be aware of what junk it is.

  52. tkaiser
    November 3rd, 2017 at 01:06 | #52

    willmore :
    Michael shows no interest in fixing PTS.

    And it really helps to think about Phoronix Media’s business model to understand why this will never change. 🙂

    PTS is great in a lot of areas (especially presenting/visualizing and comparing numbers) but unfortunately the first step to collect useful data and not just garbage is totally flawed. Which is understandable having PTS target audience in mind (people preferring data over information and happy looking at nice graphs representing numbers without meaning).

    IMO most important lesson to learn: Switch from passive to active benchmarking!

  53. November 3rd, 2017 at 09:41 | #53
  54. November 3rd, 2017 at 09:53 | #54

    @vamrs
    For whatever reason, John the ripper seems to be very sensitive to CPU frequency.
    Were the Cortex A72 cores clocked at 1.8 GHz or higher in your boards?

  55. November 3rd, 2017 at 10:47 | #55

    With regards to AES on OpenSSL, the important flags appear to be -DAES_ASM -DVPAES_ASM -DBSAES_ASM for hardware crypto, or SIMD accelerations
    The implementations are shown @ https://github.com/openssl/openssl/tree/master/crypto/aes/asm
    Even 32-bit ARM (ARMv7) would be accelerated, as while there’s no AES instructions, it would still be accelerated with NEON if the SoC supports it.

    ARMv8 does not mean AES crypto instructions are enabled, as at least one Cortex A53 based SoC does not have those.

  56. tkaiser
    November 3rd, 2017 at 14:04 | #56

    cnxsoft :
    @vamrs
    For whatever reason, John the ripper seems to be very sensitive to CPU frequency.

    Nope, the problem is that for whatever reasons people try to use a tool that graphs numbers without meaning (PTS) as performance ‘benchmark’.

    These are the settings your binaries have been built with

    And these are @vamrs settings:

    The PTS is known to produce only numbers without meaning in the most stupid ‘fire and forget’ mode possible. Why do people care about those wrong numbers produced?

    If you compare the settings above it’s easy to spot why ‘performance’ differed (you ‘optimized’ for armv7-a while on ROCK960 aarch64 was set as target architecture — the PTS is all about (missing) compiler optimizations and if used correctly could be used to demonstrate how important it is to use ARMv8 64-bit settings on ARMv8 64-bit platforms. But instead the PTS is mis-used to draw conclusions about hardware performance)

  57. blu
    November 3rd, 2017 at 14:50 | #57

    cnxsoft :
    ARMv8 does not mean AES crypto instructions are enabled, as at least one Cortex A53 based SoC does not have those.

    It seems Amlogic are the greatest ‘offender’ here, as even Allwinner’s A53 have the crypto extensions.

    Do we know exactly which SoCs by Amlogic lack the crypto extensions, besides the S905?

  58. November 3rd, 2017 at 14:54 | #58

    @blu
    Don’t know, I just saw that in the code. They did not name names, just:

    Reason for undertaken effort is that there is at least one popular SoC based on Cortex-A53 that doesn’t have crypto extensions.

    Maybe S905 (and variants?) SoC is the only one.

  59. November 3rd, 2017 at 14:59 | #59

    @cnxsoft
    Now I can see S905 has it own crypto module according to: https://forum.odroid.com/viewtopic.php?t=22236#p151854

  60. tkaiser
    November 3rd, 2017 at 15:02 | #60

    cnxsoft :
    ARMv8 does not mean AES crypto instructions are enabled, as at least one Cortex A53 based SoC does not have those.

    Yes, the crypto stuff is optional with A53 (BTW: it’s not only AES). Samsung/Nexell S5P6818 (A53) comes without, same with BCM2837 on RPi 3 and Amlogic S905 (while S905X is said to have crypto extensions enabled/licensed).

  61. tkaiser
    November 3rd, 2017 at 15:15 | #61

    cnxsoft :
    @cnxsoft
    Now I can see S905 has it own crypto module

    The problem with those vendor proprietary crypto modules is how to get them working and how do they perform (especially with small chunk sizes that might be typical for encryption/decryption of VPN traffic). If you check Banana Pi R2 AES performance in the referenced link in my first comment above you see that MediaTek’s own crypto module performs very poorly with small chunk sizes. Also it’s PITA to get this stuff working since you would need to rebuild the kernel to make use of cryptodev, then build OpenSSL from sources, then start to realize that you also have to take care about IRQ affinity since mtk-aes interrupts ending up all on cpu0 will bottleneck crypto stuff running in parallel and so on.

    While with the ARMv8 crypto extensions it just works as long as you use a somewhat decent ARMv8/aarch64 userland and the kernel has been built correctly… and avoid the Phoronix way to build unoptimised binaries of course 😉

  62. tkaiser
    November 3rd, 2017 at 17:11 | #62

    AES performance and A53 without crypto extensions (Raspberry Pi 3 / BCM 2837):

    Two times tested, first time with RPi foundation software (32-bit kernel ‘4.9.59-v7+ #1047 SMP Sun Oct 29’) and Raspbian Stretch userland (build ‘optimization’ for ARMv6!), then with a true 64-bit OS (64-bit kernel ‘4.11.12-pi64+ #1 SMP PREEMPT Sun Jul 30’ and original Debian Stretch arm64 userland). OpenSSL distro package (OpenSSL 1.1.0f 25 May 2017) used with ‘openssl speed -elapsed -evp aes-128-cbc’

    When using the useless Phoronix test suite we would see same high differences like with RK3399 above (Videostrong VS-RK3399 vs. ROCK960, one time tested with Debian armhf distro, the other time with Debian arm64 userland) and also as expected other benchmark numbers differ, the best example for ‘benchmarking gone wrong’ as usual the stupid sysbench pseudo benchmark (sysbench –test=cpu –cpu-max-prime=20000 run –num-threads=4):

    Same hardware and the ‘renowned’ sysbench ‘benchmark’ running 11 times faster with an arm64 userland compared to Raspbian’s armhf/ARMv6 (BTW: when using sysbench from Jessie then Raspbian ARMv6 number is 120 sec — that’s just the difference between GCC 4.9 vs. GCC 6.3) while 7-zip performs slower with the Debian arm64 variant (due to Raspberry Pi folks using pretty good compiler settings for their ARMv6 Raspbian).

  63. willmore
    November 3rd, 2017 at 17:31 | #63

    @cnxsoft
    The S905 in the ODROID C2 is one such AARCH64 processor without AES. If you need numbers from one, I would be glad to provide them if you don’t have one.

  64. tkaiser
    November 3rd, 2017 at 18:18 | #64

    @willmore
    I use your C2 OpenSSL numbers below you posted to Armbian forum a while back. All results done with ‘openssl speed -elapsed -evp aes-128-cbc’ using arm64 Binaries on 64-bit platforms (since otherwise numbers are lower, see RPi 3 example above):

    BPi R2 and M3 are Cortex-A7, Clearfog A9, ODROID XU4 is A15, MiQi A17. Raspberry Pi 3, ODROID-C2 and NanoPi M3 use A53 without ARMv8 crypto extensions. Orange Pi PC2, EspressoBin, Pinebook and ROCK64 is A53 with crypto extensions and both MacchiatoBin and H96-Max use Cortex-A72.

    Obviously ARMv8 crypto extensions simply outperform everything else. Especially the A72 implementation rocks even with very small data chunks. If we compare Cortex-A72 with A7 (Banana Pi M3 running at the same clockspeed) then we get 13 times better performance with small chunks and up to 30 times better results with large chunks (use cases: VPN vs. full disk encryption). A53 with crypto extensions performs not that great with small chunks but still outperforms easily all ARMv7 cores.

    A53 without crypto extensions performs at Cortex-A9 level but will be outperformed by both A15 and A17 (comparing identical clockspeeds).

  65. tkaiser
    November 3rd, 2017 at 19:09 | #65

    blu :
    Do we know exactly which SoCs by Amlogic lack the crypto extensions, besides the S905?

    S905 lacks them while S905X and S912 have them enabled.

    It’s easy to check using the results searchable at browser.geekbench.com of another flawed benchmark collecting also mostly numbers without meaning: Geekbench. Forget about the ‘total scores’ since total BS but check detailed results and there AES numbers. AML-S905X-CC AKA ‘Le Potato’ gets a multi-core AES score of 1648 points (1.24 GB/sec), a Beelink GT1 2739 points (2.06 GB/sec) while a Raspberry Pi 3 for example scores just 79 points (61.0 MB/sec). The difference in results between S905X/S912 and BCM2837 is of course questionable but at least it’s easy so spot where ARMv8 crypto extensions are available and where not.

    Le Potato single-core AES score is 392 points (302.5 MB/sec) so obviously this number is too low compared to quad-core result of 1648 (it can’t be more than 4 times more)

    For Beelink GT1 single-core result is 448 points (345.5 MB/sec) which looks somewhat realistic given that the SoC has 8 cores and multi-core number is 6 times higher.

  66. blu
    November 3rd, 2017 at 21:47 | #66

    tkaiser :
    S905 lacks them while S905X and S912 have them enabled.

    I still can’t figure out what thought process goes in the head of vendors who decide that:
    1. Their SoC will target crypto applications, yet
    2. They won’t license the proven and easilly accessible aarch64 crypto extensions, instead they’ll provide some proprietary engine.

  67. tkaiser
    November 4th, 2017 at 06:31 | #67

    @blu
    Well, at least Amlogic corrected this later and both S905X and S912 can be considered being the fastest AES/SHA crypto performers with A53 cores (since 1.5GHz clockspeed). Some new results (I partially fail to understand): https://forum.armbian.com/topic/5570-some-basic-benchmarks-for-le-potato/?tab=comments#comment-42725

    Wrt proprietary crypto modules it depends. Marvell’s CESA for example is able to increase the above AES performance numbers for Clearfog (Cortex-A9) by 3 to 4 times. It’s also said that more recent ARMv8 Marvell SoCs like Armada 37xx (EspressoBin) contain a new ‘Marvell 5Gbps Security Engine’ (do a web search for) outperforming both CESA and the already licensed ARMv8 crypto extensions with mainline kernel support landing next year.

    On the other hand the preliminary results we collected eg. for Banana Pi R2 show clearly that proprietary crypto modules might have massive downsides since currently performance pretty much sucks with MTK’s AES implementation as long as the data chunks aren’t rather large:

  68. willmore
    November 4th, 2017 at 09:45 | #68

    @blu
    They probably added that crypto module back in ARMv5 or so and there’s that *one* customer who uses it, so they just carry it along and that customer is happy.

    Crypto customers are either silly middle managers who see ‘crypto’ in the list of features and check a box saying “yep, has crypto” or the legacy people mentioned above.

  1. No trackbacks yet.