Test AI Performance in Android with AI Benchmark App

Many new phones now come with Silicon-on-Chips (SoC) featuring a Neural Processing Unit (NPU) or similar IP block(s) aiming to accelerate A.I workloads such as facial recognition or object detection. Some silicon vendors will report performance in terms of TOPS (Trillion of Operation per Seconds) with for example 2.25 TOPS for the latest Mediatek Helio P90 processor, or 4.20 TOPS for Hisilicon Kirin 980 SoC.

However, those numbers can be deceptive, and nothing beats actual tests. However, last time I checked Android benchmarks such as Antutu do not include tests specific to artificial intelligence just yet. Luckily there’s already an app for that called AI-Benchmark which you’ll find on Google Play.

AI Benchmark App
Click to Enlarge

The benchmark includes 9 tasks:

  1. Object Recognition / Classification with MobileNet – V2
  2. Object Recognition / Classification with Inception – V3
  3. Face Recognition with Inception ResNet V1
  4. Image Deblurring with SRCNN 9-5-5 neural network
  5. Image Super-Resolution with VGG – 19 neural network
  6. Image Super-Resolution with SRGAN neural network(CPU only)
  7. Semantic Image Segmentation with ICNet, 2X parallel  (CPU only)
  8. Photo Enhancement with ResNet – 12
  9. Memory Limits with SRCNN 9-5-5

Unless otherwise noted (CPU only) all tests make use of the CPU, NPU and/or DSP when available. More details on each test can be found on AI Benchmark website.

Xiaomi Mi A2 AI Benchmark
Click to Enlarge

I decided to give it a try on Xiaomi Mi A2 smartphone running Android 9.0. The phone is powered by a Qualcomm Snapdragon 660 mid-range processor which was not advertised with any kind of neural network acceleration at the time of launch. However, the processor features an Hexagon 680 DSP which may have been used, and may explain why the performance may not be as bad as I expected against phones with SoC equipped with an NPU like Google Pixel 3 XL (Qualcomm Snapdragon 845 with Pixel Visual Core) or Huawei P20 Pro (HiSilicon Kirin 970 with NPU).  Xiaomi Mi A2 gets 2,755 points, while the latter achieves 7,833 points.

However, the comparison results in the app do not use the top scores achieved in AI Benchmark, so instead, you may want to go to the website where the gap is much more significant, and top devices currently based on Kirin 980 get over 14,000 points.

Fastest AI ProcessorsIf we filter by processor instead, the upcoming Qualcomm Snapdragon 855 and Mediatek Helio P90 takes the lead with the latter taking the top step with  18,231 points.

Thanks to Jon for the tip.

Share this:

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK 5 ITX RK3588 mini-ITX motherboard
Subscribe
Notify of
guest
The comment form collects your name, email and content to allow us keep track of the comments placed on the website. Please read and accept our website Terms and Privacy Policy to post a comment.
4 Comments
oldest
newest
Jeff S.
Jeff S.
5 years ago

I found the associated paper, at https://arxiv.org/abs/1810.01109 a great read. It has a very easy to follow overview of the whole Android situation around AI / NN accelerator chips and what sort of capabilities are exposed to programmers and via which SDKs and interfaces.

It really is a good, quick read to get a handle on what’s out there or will be possible to create beyond the “bokeh effect” manufacturer-provided camera apps that seem to often be the only app that can use the AI accelerator on a new phone.

Jon Smirl
5 years ago

You need to read the footnotes on those benchmarks. Some of the NPUs are fixed point only, some are floating point only, some are both. The benchmarks contains both fixed and FP tests. The Kirin 970 wins because it accelerates both fixed and FP. Most of the other processor only do one or the other.

Jon Smirl
5 years ago

If you are looking for a place to train networks in the cloud, the Google AI cloud is fairly low cost compared to Amazon.
https://cloud.google.com/tpu/docs/pricing

Table of training costs at various cloud providers.
https://dawn.cs.stanford.edu/benchmark/

For development training I am using an Nvidia graphics card in my desktop. The cloud systems like Google can be 100x faster than my desktop.

theyguyuk
theyguyuk
5 years ago

For a laugh and a test I ran this three times on a non AI platform, I used my s905x Khadas Vim ( which Khadas gave me ). The AI programme ran but crashed near the end ( to be expected ) and said I had low speed results. Not that sure the photos used are presented random, as it felt like they ran in the same order of photo every time.

Khadas VIM4 SBC