Rockchip RK3588’s NPU open-source driver performs object detection at 30 FPS

Tomeu Vizoso has been working on an open-source driver for NPU (Neural Processing Unit) found in Rockchip RK3588 SoC in the last couple of months, and the project has nicely progressed with object detection working fine at 30 fps using the SSDLite MobileDet model and just one of the three cores from the AI accelerator.

Many recent processors include AI accelerators that work with closed-source drivers, but we had already seen reverse-engineering works on the Allwinner V831’s NPU a few years ago, and earlier this year, we noted that Tomeu Vizoso released the Etvaniv open-source driver that works on Amlogic A311D’s Vivante NPU. Tomeu has now also started working on porting his Teflon TensorFlow Lite driver to the Rockchip RK3588 NPU which is closely based on NVIDIA’s NVDLA open-source IP.

Rockchip RK3588 open source NPU driver

He started his work in March leveraging the reverse-engineering work already done by Pierre-Hugues Husson and Jasbir Matharu and was quickly able to run TensorFLow Lite’s Conv2D and DepthwiseConv2D operations. Only two weeks later, MobileNetv1 model could run on the Pine64 QuartzPro64 SBC with the same performance level as the blob (closed-source binary).

Work was much easier than on the Verisilicon Vivante NPU because lots of the reverse-engineering work was done, and NVDLA is open-source so at least some documentation was available, which was not the case for the Vivante NPU. Nevertheless, it took only four weeks (not full-time) to have the object detection shown below work on the Rockchip RK3588’s NPU at 30 FPS.

YouTube video player

You’ll find the source code for the Teflon project on Freedesktop website, and you can also the status of the project on Tomeu’s blog. Next up, Tomeu plans to write a kernel driver for Linux mainline in the drivers/accel subsystem. There’s still much work to be done and it’s unclear how long it will take, especially since he is working on different NPUs and will split his time between each implementation unless additional contributors join the project(s).

Share this:
FacebookTwitterHacker NewsSlashdotRedditLinkedInPinterestFlipboardMeWeLineEmailShare

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK 5 ITX RK3588 mini-ITX motherboard

15 Replies to “Rockchip RK3588’s NPU open-source driver performs object detection at 30 FPS”

  1. Minor correction: according to Tomeu’s blog post, that 30fps is when running on just one of the three cores!

    1. How so? Great to see you here Megi! Looking forward to the next update of your xnux blog

  2. Exciting! Hardware wise these RK3588 boards are perfect for NVRs. It’s nice to see the software progress toward taking advantage of that on mainline.

  3. It’s just incredible to mesure the amount of work, patience, talent and so on for developing such an open source project. Starting from close to NIL without any reliable doc on a quiet advanced and new topic, this is remarkable.

  4. Excellent work!
    Do you know which available cameras work with the RK3588? Last time i checked not a lot were supported by the libcamera framework, and this limits the useability quite a lot.

Leave a Reply

Your email address will not be published. Required fields are marked *

Khadas VIM4 SBC
Khadas VIM4 SBC