Many new phones now come with Silicon-on-Chips (SoC) featuring a Neural Processing Unit (NPU) or similar IP block(s) aiming to accelerate A.I workloads such as facial recognition or object detection. Some silicon vendors will report performance in terms of TOPS (Trillion of Operation per Seconds) with for example 2.25 TOPS for the latest Mediatek Helio P90 processor, or 4.20 TOPS for Hisilicon Kirin 980 SoC.
However, those numbers can be deceptive, and nothing beats actual tests. However, last time I checked Android benchmarks such as Antutu do not include tests specific to artificial intelligence just yet. Luckily there’s already an app for that called AI-Benchmark which you’ll find on Google Play.
The benchmark includes 9 tasks:
- Object Recognition / Classification with MobileNet – V2
- Object Recognition / Classification with Inception – V3
- Face Recognition with Inception ResNet V1
- Image Deblurring with SRCNN 9-5-5 neural network
- Image Super-Resolution with VGG – 19 neural network
- Image Super-Resolution with SRGAN neural network(CPU only)
- Semantic Image Segmentation with ICNet, 2X parallel (CPU only)
- Photo Enhancement with ResNet – 12
- Memory Limits with SRCNN 9-5-5
Unless otherwise noted (CPU only) all tests make use of the CPU, NPU and/or DSP when available. More details on each test can be found on AI Benchmark website.
I decided to give it a try on Xiaomi Mi A2 smartphone running Android 9.0. The phone is powered by a Qualcomm Snapdragon 660 mid-range processor which was not advertised with any kind of neural network acceleration at the time of launch. However, the processor features an Hexagon 680 DSP which may have been used, and may explain why the performance may not be as bad as I expected against phones with SoC equipped with an NPU like Google Pixel 3 XL (Qualcomm Snapdragon 845 with Pixel Visual Core) or Huawei P20 Pro (HiSilicon Kirin 970 with NPU). Xiaomi Mi A2 gets 2,755 points, while the latter achieves 7,833 points.
However, the comparison results in the app do not use the top scores achieved in AI Benchmark, so instead, you may want to go to the website where the gap is much more significant, and top devices currently based on Kirin 980 get over 14,000 points.
If we filter by processor instead, the upcoming Qualcomm Snapdragon 855 and Mediatek Helio P90 takes the lead with the latter taking the top step with 18,231 points.
Thanks to Jon for the tip.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress
I found the associated paper, at https://arxiv.org/abs/1810.01109 a great read. It has a very easy to follow overview of the whole Android situation around AI / NN accelerator chips and what sort of capabilities are exposed to programmers and via which SDKs and interfaces.
It really is a good, quick read to get a handle on what’s out there or will be possible to create beyond the “bokeh effect” manufacturer-provided camera apps that seem to often be the only app that can use the AI accelerator on a new phone.
You need to read the footnotes on those benchmarks. Some of the NPUs are fixed point only, some are floating point only, some are both. The benchmarks contains both fixed and FP tests. The Kirin 970 wins because it accelerates both fixed and FP. Most of the other processor only do one or the other.
If you are looking for a place to train networks in the cloud, the Google AI cloud is fairly low cost compared to Amazon.
https://cloud.google.com/tpu/docs/pricing
Table of training costs at various cloud providers.
https://dawn.cs.stanford.edu/benchmark/
For development training I am using an Nvidia graphics card in my desktop. The cloud systems like Google can be 100x faster than my desktop.
For a laugh and a test I ran this three times on a non AI platform, I used my s905x Khadas Vim ( which Khadas gave me ). The AI programme ran but crashed near the end ( to be expected ) and said I had low speed results. Not that sure the photos used are presented random, as it felt like they ran in the same order of photo every time.