Back in 2024, we noted that the open-source driver for RK3588’s NPU by Tomeu Vizoso delivered decent performance for object detection. Since then, good progress has been made, and it’s now known as the Rocket driver in mainline Linux.
However, the Rockchip RK3576 SoC is not supported by the Rocket driver in mainline yet, despite sharing a similar NPU. The good news is that Ga Hing Woo has been working on mainline Linux support for the Rockchip RK3576 NPU using the Rocket driver stack and has tested the updates on the Radxa ROCK 4D running Linux 7.1-rc5. This required some serious reverse-engineering work due to hidden registers not documented in the TRM.
Ga Hing Woo (also written as Jiaxing Hu) sent us an email explaining some of the challenges:
Single int8 convolution now comes out byte-exact against a CPU reference on the open stack. Getting there meant reverse-engineering the compute registers, which the TRM leaves out completely (searching “CNA”, “CMAC” or “convolution” gives zero hits across all 4381 pages).
Chained layers still stall, and I spent the time to work out why rather than guess. The CNA/CORE/DPU executer bit 16 never engages on chained tasks, which traces down to a sub-MMIO sequencer state the driver can’t see. So it’s a clean negative result, not a fix I’m claiming.
Another developer, MidG971, ran into the same wall independently on the RK3568, coming at it from a different chip. That’s really what pushed me to write it up instead of sitting on it.
The writing in question is a highly technical research paper entitled “Bringing Up the RK3576 NPU on Mainline Linux: A Byte-Exact Single-Task Path and a Falsification-First Negative Result on Chained-Task Compute”. Every number in the paper is measured, not projected. It will probably only be understood by people who have already worked on this type of NPUs.
The source code and instructions for the mainline kernel bring-up for the RK3576 NPU on Radxa ROCK 4D can be found on GitHub. He also packaged the RKNPU/RKLLM runtime to run local LLMs and vision on a mainline kernel (Kiln) to show that the kernel patches successfully enable the vendor driver even though the open-source Rocket driver is not yet fully ready for the RK3576. Results on the ROCK 4D SBC: Llama-3.2-1B at about 13 tok/s, Qwen2.5-1.5B at about 9 tok/s, and MobileNet at about 169 fps.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress. We also use affiliate links in articles to earn commissions if you make a purchase after clicking on those links.
Not that I had understood alot of what has been done, but its a sad story still things need reverse engineering to make it work…
Thanks to all the devs keeping our hobby going.
Even reading the Abstract was too hard to understand for me.
So I asked Google Notebook LM. See below. So: only succesful for single-task, not for chained tasks?
This technical preprint details an end-to-end investigation into bringing up the Rockchip RK3576 NPU using the mainline Linux “rocket” driver and Mesa Teflon delegate. While the author successfully achieved byte-exact results for single-task operations by correcting a requantization error, the primary contribution is a negative result regarding chained tasks. Despite rigorous testing through three independent dispatch mechanisms, multi-layer neural networks failed to compute more than one task per power session. The study refutes all software-observable causes—including clock/power settings, IOMMU state, and firmware—eventually localizing the failure to an undocumented on-chip compute arm that lacks a register footprint. By comparing these findings to the RK3588 and RK3568, the researcher concludes that the open-source stack currently lacks the specific hardware triggers required to re-arm the NPU for sequential tasks. The work ultimately provides a falsification ledger and upstreamable bug fixes to assist future developers in overcoming this architectural barrier.
And tools used:
Instrumentation and Debugging Tools
ftrace: A primary tool for tracing writel and readl operations, regmap_reg_write, clock settings (clk), power domains (genpd), and IOMMU behavior
Replay Harness: A custom-built tool developed by the author to reproduce and test hardware behaviors
AI Assistant: The author explicitly notes using an AI assistant for “mechanical steps,” such as diffing register and configuration dumps and performing literature research on public materials like datasheets and mailing lists
i really really really hope the next Rockchip NPU in RK3688 and sub variants have a better support for LLMs rather than only video like these
with 6 TOPS … would it be relevant?
Or do you mean with new NPU’s with higher TOPS? Then I follow my rule of thumb: if not open source, don’t buy the hardware. And not the other way around: buy the hardware, and then hope & pray someone else will do reverse engineering.
well TOPS end up being a marketing scheme most of the times in these little devices and the new methods of quantization used, let alone the lack of support for transformer operations in this ancient NVDLA architecture in the RK3576/3588…. Regarding the openess, i think the rule of buying and praying only applies to rockchip which is the “major” designer of the chinese ones.
RK3688 NPU is only 18 TOPS according to the presentation, that does not sound much BUT they already have experience designing their own accelerators for LLMs such as RK182X & the second gen which are around 20TOPS and capable of running small qwen models. Now, what i think is going to happen is that the design of the npu is going to get very old when the product hits the market and that is going to end up making the chip pretty useless. I pray that the chip is still in validation and dont end up like the garbage cix we got
Nice work from Ga Hing Woo 🙂 For those curious about the RK3568 bring up efforts check out my repo