DeGirum ORCA M.2 and USB Edge AI accelerators support Tensorflow Lite and ONNX model formats

I’ve just come across an Atom-based Edge AI server offered with a range of AI accelerator modules namely the Hailo-8, Blaize P1600, Degirum ORCA, and MemryX MX3. I had never heard about the last two, and we may cover the MemryX module a little later, but today, I’ll have a closer at the Degirum ORCA chip and M.2 PCIe module.

The DeGirum ORCA is offered as an ASIC, an M.2 2242 or 2280 PCIe module, or (soon) a USB module and supports TensorFlow Lite and ONNX model formats and INT8 and Float32 ML precision. They were announced in September 2023, and have already been tested in a range of mini PCs and embedded box PCs from Intel (NUC), AAEON, GIGABYTE, BESSTAR, and Seeed Studio (reComputer).

DeGirum ORCA specifications:

• Supported ML Model Formats – ONNX, TFLite
• Supported ML Model Precision – Float32, Int8
• DRAM Interface – Optional 1GB, 2GB, or 4GB 32-bit LPDDR4X
• Host interfaces
  • PCIe 2×4 (Gen 3, Root Port, & Endpoint)
  • USB – USB 3.1 Gen2, USB 2.0
• Other I/Os – QSPI master/slave, I2C, UART, 32x GPIO
• Misc – Scalable architecture, meaning several chips can be connected to linearly increase the performance
• Power Consumption – < 3.5W
• Package – 15x15mm BGA 484 Ball MAF-FCCSP
• Temperature Range – 0°C to 70°C

Degirum sells two M.2 PCIe modules: one M.2 2280 module with DRAM that consumes under 4.5 Watts, and an M.2 2242 module without DRAM that consumes less than 4 Watts. ORCA USB Dongles are also listed (with no detail) and they are only shown for pre-order right now. The advantage of having RAM on the module is explained as follows:

Having DRAM access support in our AI accelerator offers significant advantages for users. With the ability to access DRAM directly, our AI accelerator can provide faster data transfer rates, which translates into improved performance and reduced latency. In addition to providing faster data transfer rates, DRAM access support in our AI accelerator also allows for quick and seamless switching of neural network (NN) models. With this capability, our customers can easily switch between different NN models without the need for time-consuming data transfers, reducing downtime and increasing productivity. This feature is particularly valuable for applications that require frequent model changes, such as image or speech recognition, where different models may be needed to handle varying data sets or specific tasks. By enabling rapid model switching directly from DRAM, our AI accelerator provides users with greater flexibility and efficiency in their AI workflows.

I could not find any (more or less useful) TOPS numbers, but the company provides some machine learning performance metrics with the “DeGirum YOLO_V5s with input size 512×512” processed at 120 FPS (Dense) or 180 FPS (Prune) with a latency of 8.3ms (Dense) and 5.5ms (Pruned). You’ll find additional benchmarks in the PySDK examples repository that comes with various Python samples (object detection, sound classification,  license plate recognition, etc..) using the Degirum SDK and models such as MobileNet v2/1, Yolov5, and resnet50. The SDK is supported in Linux, but the company also says Windows and Mac support can be provided on demand. Technical documentation can be found on the company’s website.

The M.2 PCIe cards can be ordered now, but only after providing your company name and expected order quantity.  The ORCA USB 3.1 Gen2 Dongle is only available for pre-order through a similar process, and I could not find pricing. It’s however possible to evaluate the ORCA hardware through the company’s cloud platform which includes a free plan. More details may be found on the product page.

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress