Archive

Posts Tagged ‘computer vision’

Arm’s Project Trillium Combines Machine Learning and Object Detection Processors with Neural Network Software

February 14th, 2018 No comments

We’ve already seen Neural Processing Units (NPU) added to Arm processors such as Huawei Kirin 970 or Rockchip RK3399Pro in order to handle the tasks required by machine learning & artificial intelligence in a faster or more power efficient way.

Arm has now announced their Project Trillium offering two A.I. processors, with one ML (Machine Learning) processor and one OD (Object Detection) processor, as well as open source Arm NN (Neural Network) software to leverage the ML processor, as well as Arm CPUs and GPUs.

Arm ML processor key features and performance:

  • Fixed function engine for the best performance & efficiency for current solutions
  • Programmable layer engine for futureproofing the design
  • Tuned for advance geometry implementations.
  • On-board memory to reduce external memory traffic.
  • Performance / Efficiency – 4.6 TOP/s with an efficiency of 3 TOPs/W for mobile devices and smart IP cameras
  • Scalable design usable for lower requirements IoT (20 GOPS) and Mobile (2 to 5 TOPS) applications up to more demanding server and networking loads (up to 150 TOPS)

Arm OD processor main features:

  • Detects objects in real time running with Full HD (1920×1080) at 60fps (no dropped frames)
  • Object sizes – 50×60 pixels to full screen
  • Virtually unlimited objects per frame
  • Image Input – Raw input from camera or from ISP
  • Latency – 4 frames.
  • Can be combined with CPUs, GPUs or the Arm ML processor for additional local processing, reducing overall compute requirement

The company provides drivers, detailed people model with rich metadata allowing the detection of direction, trajectory, pose and gesture, an object tracking library and sample applications. I could not find any of those online, and it appears you have to contact the company for details if you plan to use this into your chip.

Click to Enlarge

Arm will be more open about their Arm NN software development kit which bridges existing neural network frameworks like TensorFlow, Caffe(2), or MxNet – to the underlying processing hardware including Arm Cortex CPUs, Arm Mali GPUs, or the new ML processor. Arm NN will focus on inference, which happens on the target device (aka at the edge), rather than training which happens on powerful servers.

Arm NN utilizes the Compute Library to target  Cortex-A CPUs, Mali GPUs & Arm ML processor, and CMSIS-NN for Cortex-M CPUs. The first release will support Caffe with downloads, resources, and documentation becoming available in March 2018, with TensorFlow coming next, and subsequently other neural network frameworks will be added to the SDK. There’s also seems to be an Arm NN release for Android 8.1 (NNAPI) that currently works with CPU and GPU.

Visit Arm’s machine learning page for more details about Project Trillum, ML & OD processors, and Arm NN SDK. You may also be interested in a few related blog posts on Arm community.

A Day at Chiang Mai Maker Party 4.0

December 6th, 2017 6 comments

The Chiang Mai Maker Party 4.0 is now taking place until December 9, and I went there today, as I was especially interested in the scheduled NB-IoT talk and workshop to find out what was the status about LPWA in Thailand. But there are many other activities planned, and if you happen to be in Chiang Main in the next few days, you may want to check out the schedule on the event page or Facebook.

I’m going to go though what I’ve done today to give you a better idea about the event, or even the maker movement in Thailand.

Click to Enlarge

Booth and activity area should be the same over the 4 days, but the talks, open activity, and workshop will be different each day. Today, people could learn how to solder in the activity area.
The even was not really big with manufacturers/sellers like ThaiEasyElec, INEX, or Gravitech closer to the entrance…


… and slighter higher up in a different zone, companies and makers were showcasing their products or projects. I still managed to spent 5 interesting hours at the event attending to talks and checking out the various projects.

I started my day with a talk entitled “Maker Movement in South East Asia” presented by William Hooi, previously a teacher, who found One Maker Group and setup the first MakerSpace in Singapore, as well as helped introduce the Maker Faire in Singapore in 2012 onwards.


There was three parts to talk with an history of the Maker movement (worldwide), the maker movement in Singapore, and whether Making should be integrated into school curriculum.
He explained at first the government who not know about makers, so it was difficult to get funding, but eventually they jump on the bandwagon, and are now puring money on maker initiative. One thing that surprised me in the talk is that before makers were hidden their hobby, for fear of being mocked by other, for one for one person doing an LED jacket, and another working on an Iron Man suit. The people around them would not understand why they would waste their time on such endeavors, but the Maker Space and Faire helped finding like minded people. Some of the micro:bit boards apparently ended in Singapore, and when I say some, I mean 100,000 units. Another thing that I learned is the concept of “digital retreat for kids” where parents send kids to make things with their hands – for example soldering -, and not use smartphone or tablets at all, since they are already so accustomed to those devices.

One I was done with the talk, I walked around, so I’ll report about some of the interesting project I came across. I may write more detailed posts for some of the items lateron.

Click to Enlarge

Falling object detection demo using OpenCV on the software side, a webcam connected to…

Click to Enlarge

ASUS Tinker board to handle fall detection, and an NVIDIA Jetson board for artificial intelligence. If fall is detection an alert to send to the tablet, and the system also interfaces with Xiaomi Mi band 2.

Katunyou has also made a more compact product, still based on Tinker Board, for nursing home, or private home where an elderly may live alone. The person at the stand also organizes Raspberry Pi 3 workshops in Chiang Mai.

I found yet another product based on Raspberry Pi 3 board. SRAN is a network security device made by Global Tech that report threats from devices accessing your network using machine learning.

Click to Enlarge

Nordic Technology House showcased a magic mirror based on Raspberry Pi 3, and a webcam to detect your dance move, but their actual product shown above is a real-time indoor air monitoring system that report temperature, humidity, CO2 level, and PM 2.5 levels, and come sent alerts via LINE if thresholds are exceeded.

One booth had some drones including the larger one above spraying insecticides for the agriculture market.

Click to Enlarge

There was also a large about sewing machines, including some smarter one where you can design embroidery in a table before sewing.

There were also a few custom ESP8266 or ESP32 boards, but I forgot to take photos.

The Maker Party is also a good place to go with your want to buy some board or smart home devices.

Click to Enlarge

Beside Raspberry Pi Zero W / 3, ESP8266 boards and Asus Tinker board seem to be popular items in Thailand. I could also spot Sonoff wireless switch, and an Amazon Dot, although I could confirm only English is supported, no Thai language.

BBC Micro:bit board and accessories can also be bought at the event.


M5Stack modules, and Raspberry Pi 3 Voice Kit were also for sale.

Click to Enlarge

Books are also available for ESP32, Raspberry Pi 3, IoT, etc… in Thai language.

Click to Enlarge

But if you can’t read Thai there was also a choice of book in English about RPi, Arduino, Linux for Makers, IoT and so on. I then attended the second talk of the day: “NB-IoT” by AIS, one of the top telco company in Thailand. Speakers included Phuchong Charoensub, IoT Marketing Specialist, and Pornsak Hanvoravongchai, Device Innovation Manager, among others. They went through various part include a presentation of AIS current M2M business, what IoT will change (e.g. brings in statups and makers), some technical details about NB-IoT, and the company offering for makers.

I’ll go into more details in a separate post tomorrow, but if you want to get started the good news is that it’s now possible to pre-order a 1,990 THB Arduino Shield ($61) between December 6-9, and get it shipped on February 14, 2018. NB-IoT connectivity is free for one year, and will then cost 350 Baht (around $10) per year per device. However, there’s a cost to enable NB-IoT on LTE base stations, so AIS will only enable NB-IoT at some universities, and maker spaces, meaning for example, I would most certainly be able to use such kit from home. An AIS representative told me their no roadmap for deployment, it will depend on the business demand for such services.

If you are lucky you may even spot one or two dancing dinosaurs at the event.

$45 AIY Vision Kit Adds Accelerated Computer Vision to Raspberry Pi Zero W Board

December 1st, 2017 2 comments

AIY Projects is an initiative launched by Google that aims to bring do-it yourself artificial intelligence to the maker community by providing affordable development kits to get started with the technology. The first project was AIY Projects Voice Kit, that basically transformed Raspberry Pi 3 board into a Google Home device by adding the necessary hardware to support Google Assistant SDK, and an enclosure.

The company has now launched another maker kit with AIY Project Vision Kit that adds a HAT board powered by Intel/Movidius Myriad 2 VPU to Raspberry Pi Zero W, in order to accelerate image & objects recognition using TensorFlow’s machine learning models.

Click to Enlarge

The kit includes the following items:

  • Vision Bonnet accessory board powered by Myriad 2 VPU (MA2450)
  • 2x 11mm plastic standoffs
  • 24mm RGB arcade button and nut
  • 1x Privacy LED
  • 1x LED bezel
  • 1x 1/4/20 flanged nut
  • Lens, lens washer, and lens magnet
  • 50 mil ribbon cable
  • Pi0 camera flat flex cable
  • MIPI flat flex cable
  • Piezo buzzer
  • External cardboard box and internal cardboard frame

Vision Bonnet Board – Click to Enlarge

Not that the accessory board features the same Movidius VPU as Intel Neural Compute Stick, which has been used with Raspberry Pi 3, and shown to deliver about 3 times the performance compared to a GPGPU implementation leveraging VideoCore IV GPU.

Back to the kit. You’ll need to add your own Raspberry Pi Zero W, Raspberry Pi camera 2, and blank SD card (at least 4 GB) to complete the kit. Follow the assembly guide, and the final results should look like this:

 

Once this is done flash the Vision Kit SD image (available soon) to your micro SD card, insert it into your Raspberry Pi Zero W, and connect the power. The software image will include three neural network models:

  • A model based on MobileNets that can recognize a thousand common objects.
  • A model for face detection capable of detecting faces and facial expressions (sadness, joy, etc…)
  • A model for discerning between cats, dogs and people.

The system will be able to run at speeds of up to 30 fps, providing near real-time performance. TensorFlow code and a compiler will also be included for people wanting to have their own models. A Python API will be provided to customize the RGB button colors, piezo element sounds, and (4x) GPIO pins.

AIY Vision Kit are up for pre-order for $44.99 at Micro Center with shipping planned for earlier December. Just like AIY Voice Kit, we should eventually expect international availability via other websites such as Piromini or Seeed Studio. The complete kit with RPi board and camera, and accessories should cost around $90.

AWS DeepLens is a $249 Deep Learning Video Camera for Developers

November 30th, 2017 4 comments

Amazon Web Services (AWS) has launched Deeplens, the “world’s first deep learning enabled video camera for developers”. Powered by an Intel Atom X5 processor with 8GB, and featuring a 4MP (1080p) camera, the fully programmable system runs Ubuntu 16.04, and is designed expand deep learning skills of developers, with Amazon providing tutorials, code, and pre-trained models.

Click to Enlarge

AWS Deeplens specifications:

  • Camera – 4MP (1080p) camera using MJPEG, H.264 encoding
  • Video Output – micro HDMI port
  • Audio – 3.5mm audio jack, and HDMI audio
  • Connectivity – Dual band WiFi
  • USB – 2x USB 2.0 ports
  • Misc – Power button; camera, WiFi and power status LEDs; reset pinhole
  • Power Supply – TBD
  • Dimensions – 168 x 94 x 47 mm
  • Weight – 296.5 grams

The camera can not only do inference, but also train deep learning models using Amazon infrastructure. Performance wise, the camera can infer 14 images/second on AlexNet, and 5 images/second on ResNet 50 for batch size of 1.

Six projects samples are currently available: object detection, hot dog not hot dog, cat and dog,  activity detection, and face detection. Read that blog post to see how to get started.

But if you want to make your own project, a typical workflow would be as follows:

  • Train a deep learning model using Amazon SageMaker
  • Optimize the trained model to run on the AWS DeepLens edge device
  • Develop an AWS Lambda function to load the model and use to run inference on the video stream
  • Deploy the AWS Lambda function to the AWS DeepLens device using AWS Greengrass
  • Wire the edge AWS Lambda function to the cloud to send commands and receive inference output

This steps are explained in details on Amazon blog.

Click to Enlarge

Intel also published a press release explaining how they are involved in the project:

DeepLens uses Intel-optimized deep learning software tools and libraries (including the Intel Compute Library for Deep Neural Networks, Intel clDNN) to run real-time computer vision models directly on the device for reduced cost and real-time responsiveness.

Developers can start designing and creating AI and machine learning products in a matter of minutes using the preconfigured frameworks already on the device. Apache MXNet is supported today, and Tensorflow and Caffe2 will be supported in 2018’s first quarter.

AWS DeepLens can be pre-ordered today for $249 by US customers only (or those using a forwarding service) with shipping expected on April 14, 2018. Visit the product page on AWS for more details.

Cheap Evil Tech – WiFi Deauther V2.0 Board and Autonomous Mini Killer Drones

November 24th, 2017 10 comments

Most technological advances usually improve life of people, and with the costs coming down dramatically over the years, available to more people. But technology can be used for bad, for example by governments and some hackers. Today, I’ve come across two cheap hardware devices that could be considered evil. The first one is actually pretty harmless and can be use for education, but disconnects you from your WiFi, which may bring severe physiological trauma to some people, but should not be life threatening, while the other is downright scary with cheap targeted killing machines.

WiFi Deauther V2.0 board

Click to Enlarge

Specifications for this naughty little board:

  • Wireless Module based on ESP8266 WiSoC
  • USB – 1x Micro USB type changed, more stable.
  • Expansion – 17-pin header with 1x ADC, 10x GPIOs, power pins
  • Misc – 1x power switch,  battery status LEDs
  • Power Supply
    • 5 to 12V via micro USB port
    • Support for 18650 battery with charging circuit (Over-charge protection, over-discharge protection)
  • Dimensions – 10×3 cm

The board is pre-flashed with the open source ESP8266 deauther firmware, which allows you to perform a deauth attack with an ESP8266 against selected networks. You can select target IP address, and the board will then disconnect that node constantly, either blocking the connection or slowing it down. You don’t need to be connect to the access point or know the password for it to work. You’ll find more details on how it works on the aforelinked Github page. Note: The project is a proof of concept for testing and educational purposes.

WiFi Deauther V2.0 board can be purchased on Tindie or Aliexpress for $10.80 plus shipping.

A.I. Powered Mini Killer Drones

The good news is that those do not exist yet (at least not for civilians), but the video shows what could happen once people, companies, or governments weaponize face recognition, and drone technology to design mini drones capable of targeted killings. You could fit the palm-sized drones with a few grams of explosives (or lethal poison), tell them who to target, and once they’d find it, land on the skull of the victim, and trigger the explosive for an instant kill. Organizations or governments could also have army of those drones for killing based on metadata obtained from phone records, social media posts, etc… The fictional video shows how those drones could work, and what may happen to society as a consequence.

Technology is already here for such devices. Currently you could probably get $400+ DJI Spark drone to handle face recognition, but considering inexpensive $20+ miniature drones and $50 smart cameras are available (but not quite good enough right now), sub $100 drones with face recognition should be up for sale in a couple of years. The explosive and triggering mechanism would still need to be added, but I’m not privy to the costs… Nevertheless, it should be technically possible to manufacture such machine, even for individuals, for a few hundreds dollars. Link to fictitious StratoEnergetics company.

The Future of Life Institute has published an open letter against autonomous weapons that has been signed by various A.I. and robotics researchers, and other notable endorsers, stating that “starting a military AI arms race is a bad idea, and should be prevented by a ban on offensive autonomous weapons beyond meaningful human control”, but it will likely be tough to keep the genie in the bottle.

Hisilicon Hi3559A V100ES is an 8K Camera SoC with a Neural Network Accelerator

November 22nd, 2017 3 comments

Earlier today, I published a review of JeVois-A33 machine vision camera, noting that processing is handled by the four Cortex A7 cores of Allwinner A33 processor, but in the future we can expect such type of camera to support acceleration with OpenCL/Vulkan capable GPUs, or better, Neural network accelerators (NNA) such Imagination Tech PowerVR Series 2NX.

HiSilicon already launched Kirin 970 SoC with such similarIP, except they call it an NPU (Neural-network Processing Unit). However, while looking for camera SoC with NNA, I found a list of deep learning processors, including the ones that go into powerful servers and autonomous vehicles, that also included a 8K Camera SoC with a dual core CNN (Convolutional Neural Network) acceleration engine made by Hisilicon: Hi3559A V100ES.

Click to Enlarge

Hisilicon Hi3559A V100ES specifications:

  • Processor Cores
    • 2x ARM Cortex A73 @ 2 GHz, 32 KB I cache, 64KB D cache or 512 KB L2 cache
    • 2x ARM Cortex A53 @ 1 GHz, 32 KB I cache, 32KB D cache or 256 KB L2 cache
    • 1x single core ARM Cortex A53 @ 1 GHz, 32 KB I cache, 32KB D cache /128 KB L2 cache
    • Neon acceleration and integrated FPU
  • GPU – Triple core ARM Mali-G71 [email protected] 900 MHz with 256KB cache, support for OpenCL 1.1/1.2/2.0, and OpenGL ES 3.0/3.1/3.2
  • Sensor Hub
    • ARM Cortex M7 @200 MHz
    • PMC, which supports only external reset, internal POR
    • General peripheral IPs (UART, SPI, I2C, PWM, GPIO,and LSADC)
    • 3-channel LSADC, 5x UART interfaces, and 8x PWM interfaces
  • Memory Interface – 32-/64-bit DDR4 up to 8GB
  • Storage Interfaces – SPI NOR flash up to 512MB, NAND flash, eMMC 5.1 up to 2TB, UFS 2.1 up to 512GB
  • Video Encoding – H.264 BP/MP/HP, and H.265 Main Profile/Main 10 Profile up to 7680 x [email protected] [email protected] fps+7680 x [email protected] fps snapshot
  • Video Decoding – H.264 BP/MP/HP, H.265 MP/Main 10/High Tier up to [email protected] fps or H.264/H.265 [email protected] fps
  • Intelligent Video Analysis
    • Integrated intelligent analysis and acceleration engine, allowing customers to develop intelligent applications targeted for mobile camera products
    • Dual-core DSP @ 700 MHz, 32 KB I cache, 32 KB IRAM, or 512 KB DRAM
    • Dual-core CNN @ 700 MHz neural network acceleration engine
  • Video and Graphics Processing
    • 3DNR, image enhancement, and DCI
    • Anti-flicker for output videos and graphics
    • 1/15.5x to 16x video & graphics scaling
    • Horizontal seamless stitching of 2-channel videos, and 360° or 720° panoramic stitching of up to 6-channel videos
    • OSD overlaying of eight regions before encoding
    • Video graphics overlaying of two layers (video layer and graphics layer)
  • 2- channel ISP
    • Adjustable 3A functions (AE, AWB, and AF)
    • FPN removal
    • Highlight suppression, backlight compensation, gamma correction, and color enhancement
    • DPC, NR, and 6-DOF DIS
    • Anti-fog
    • LDC and fisheye correction
    • Picture rotation by 90° or 270°;  Picture mirror and flip
    • HDR10, BT.2020 WCG
    • Sensor built-in WDR, 4F/3F/2F frame-based/line-based
    • WDR and local tone mapping
    • ISP tuning tools for the PC
  • Audio Encoding and Decoding
    • Voice encoding/decoding complying with multiple protocols by using software
    • MP3, AAC, and other audio encoding formats
    • Audio 3A functions (AEC, ANR, and ALC)
  • Security Engine
    • AES, DES, and 3DES encryption and decryption algorithms implemented by using hardware
    • RSA1024/2048/3072/4096 signature verification algorithm implemented by using hardware
    • SHA1/224/256/384/512 of the HASH and HMAC_SHA1/224/256/384/512 tamper proofing algorithms implemented by using hardware
    • Integrated 32-kbit OTP storage space and hardware random number generator
  • Video Interfaces
    • Input
      • Multiple sensor inputs. The maximum resolution is 32 megapixels (7680 x 4320).
      • 8-/10-/12-/14-bit RGB Bayer DC timing VI, up to 150 MHz clock frequency
      • BT.601, BT.656, and BT.1120 VI interfaces
      • Maximum 16-lane MIPI/LVDS/sub-LVDS/HiSPi/SLVS-EC interface for the serial sensor inputs
      • Maximum 6-channel video inputs for the serial sensor inputs, supporting various working modes such as 1×16-lane/2×8-lane/4×4-lane/2×4-lane+4×2-lane
    • Output
      • HDMI 2.0, supporting maximum [email protected] fps output
      • 8-/16-/24-bit RGB digital LCD output, supporting maximum 1920 x [email protected] fps output
      • 4-lane MIPI DSI output, supporting maximum 2.5 Gbit/s per lane frequency
  • Audio Interfaces
    • Integrated audio codec, supporting 16-bit audio inputs and outputs
    • I2S interface for connecting to the external audio codec
    • Dual-channel differential MIC inputs for reducing background noises
  • Peripherals
    • POR, external reset input,
    • Internal RTC
    • Integrated 2-channel LSADC
    • 5x UART interfaces
    • IR interface, I2C interface, SSP main interface, and GPIO interface
    • Integrated GMAC, supporting  RGMII and RMII
    • 2x PWM interfaces
    • 2x SD 3.0/SDIO 3.0 interfaces, supporting SDXC
    • 2x USB 3.0/USB 2.0 host/device ports
    • 2-lane PCIe 2.0 RC/EP mode
  • Operating Voltages – 0.8V core voltage, 1.8V I/O voltage, 1.2V DDR4 voltage
  • Power Consumption – 2.6 Watts
  • Package – 15 x 15 mm with 0.4 mm pitch

Boy, that’s a monster… They should have called it MOACSoC (Mother of All Camera SoCs) 🙂 The main ARM cores are said to run Linux+Huawei LiteOS AMP heterogeneous dual systems, and the company provide a dedicated SDK for the consumer mobile camera, cient for the iOS and Android mobile phones, and a high-
performance H.265 decoding library. The SDK might be in the wild as “Hi3559AV100ES_SDK_V2.0.2.0” but I did not find a download link. I got all information above from Hi3359A V100ES ultra-HD Mobile Camera SoC product brief.

Mobile Camera and Professional Camera Solution Block Diagram

Based on the block diagram above, some mobile camera and professional camera will start taking SSD drives beside the boring SD card and USB 2.0/3.0 storage devices.

Hi3559A V100ES will also be found in drone cameras, 3D/VR cameras, and 4K/8K network-based EDR. I have no idea what the latter stands for, but the photo in the document looks like a car dashboard camera with display. Anyway, this should allows for some interesting use cases with near real-time object recognition.

Hisilicon showcased a dynamic object categorization and identification system at CPSE2017 in Shenzhen earlier this month. The company did not mention Hi3559A V100, but made clear an 8K solution was used.

Click to Enlarge

If we are to believe one person on Tencent’s ncnn Github repo, performance is really good with a 10ms lag for GoogleNet, and 89ms for VGG-SSD. We’ll have to wait a little to get more details, and Hisilicon did not post any product info on their website about their new 8K SoC, only about the earlier Hi3559 2K/4K SoC.

JeVois-A33 Linux Computer Vision Camera Review – Part 2: Setup, Guided Tour, Documentation & Customization

November 22nd, 2017 4 comments

Computer Vision, Artificial Intelligence, Machine Learning, etc.. are all terms we hear frequently those days. JeVois-A33 smart machine vision camera powered by Allwinner A33 quad core processor was launched last year on Indiegogo to bring such capabilities in a low power small form factor devices for example to use in robotics project.

The company improved the software since the launch of the project, and has now sent me their tiny Linux camera developer kit for review, and I’ve already checked  out the hardware and accessories in the first post. I’ve now had time to test the camera, and I’ll explained how to set it up, test some of the key features via the provided guided tour, and show how it’s possible to customize the camera to your needs with one example.

Getting Started with JeVois-A33

In theory, you could just get started by inserting the micro SD card provided with the camera, connect it to your computer via the USB cable, and follow the other instructions on the website. But to make sure you have the latest features and bug fixed, you’d better download the latest firmware (jevois-image-latest-8G.zip), and flash it to the micro SD card with the multi-platform Etcher tool.

You could also use your own micro SD card, as long as it has 8GB or more capacity. Once this is done, insert the micro SD card into the camera with the fan of the camera and the golden contact of the micro SD card both facing upwards. Connect the camera to your computer with the provided mini USB to USB cable. I also added the USB power meter to monitor the power consumption for the different use cases, and USB serial cable to checkout output from the console. At least that was the plan, but I got no lights from the camera, and voltage was reported to be only 4V. Then I read the guide a little better, and found out I had to use a USB 3.0 port, or two USB 2.0 ports for power.

Once I switched to using two USB 2.0 ports from a powered USB 2.0 hub, I could see output from the serial console…

and both green and orange/red LEDs were lit. The instructions to use JeVois camera are mostly OS agnostic, except for the video capture software. If you are using Windows you can use the free OBS Studio or AMCap programs, and on Mac, select either PhotoBooth or OBS Studio. I’m a Ubuntu user, so instead I installed guvcview:

and ran it use 640×360 resolution and YUYV format as instructed in the getting started guide:

But then I got no output at all in the app:

The last line above would repeat in a loop. The kernel log (dmesg) also reported a crash linked to guvcview:

Another person had the same problem a few months ago, and it was suggested it may be a USB problem. So I connect the camera to directly to two of the USB ports on my tower, and it worked…

Click to Enlarge

The important part of the settings are in the Video Controls tab, where we can change resolution and frame rate to switch between camera modes as we’ll see later on.

But since my tower is under the desk, the USB cable is a bit too short, and the program crashed with the same error message a few minutes later. So I went with my Ubuntu 16.04 laptop instead. Powering the camera via the USB 3.0 port worked until I started deep learning modes, where the camera would stop, causing guvcview to gray out. Finally, I connected the camera to both my USB 3.0 port, and the power bank part of the kit, and the system was then much more stable.

Click to Enlarge

I contacted the company about the issues I had, but they replied this problem was not often reported:

… we have only received very few reports like that but we were able to confirm here using front panel ports on one machine. On my desktop I have a hub too, but usb3 and rated for fast charging (60W power supply for 7+2 ports) and it works ok with jevois. A single usb3 port on my mac laptop is also ok.

So maybe it’s just me with all my cheap devices and accessories…

So three main points to get started:

  1. Update the firmware
  2. Install the camera software
  3. Check power in case of issues / crashes (Both LEDs should be on if the camera is working)

JeVois-A33 Guided Tour

Now we have the camera running, we can try the different features, and the best way to do so is to download Jevois Guided Tour (PDF) that will give you an overview of the camera and how it works, as well as examples.

Click to Enlarge

As shown above, the PDF includes information for each module with the name, link to documentation, introduction, display explanation, and on the top right the resolution/framerate that can be used to launch a given module. On following pages, there will be example pictures that you can point to with the camera.

Some of modules include:

  • Visual attention – finding interesting things
  • Face and handwritten digit recognition
  • QR-codes and other tags
  • Road detection
  • Object matching
  • Object recognition with deep neural networks
  • Color-based object tracking
  • Moving object detection
  • Record video to the microSD card inside JeVois
  • Motion flow detection
  • Eye tracking
  • and more…

You could print the guide with a color printer, but the easiest way is problem to use two screens, once with the PDF guide open, and the other running the camera application (guvcview, OBS Studio…). I’ve gone through some of the example in the guided tour in the video below, with PDF shown on a TV box, and the camera application output shown on the laptop screen on the top bottom corner.

That’s lot of fun, and everything works pretty well most of the time. Some of the tests are quite demanding for such low power device, as for example Darknet based “Deep neural scene analysis” using 1280×480 @ 15 fps with the ability to recognize multiple object types would only refresh the results every 2.7 seconds or so.

Documentation & Customization of Salient SURF Module

If you’ve gone through the guide tour, you should now have a good understanding of what the camera is capable of. So now, let’s take one of the modules, and try to adjust it to our needs. I picked SaliencySURF module with the documentation available here for this section of the review. Introduction for the module:

Trained by default on blue iLab logo, point JeVois to it and adjust distance so it fits in an attention box.
Can easily add training images by just copying them to microSD card.
Can tune number and size of salient regions, can save regions to microSD to create a training set

So let’s take a few other images (Tux logo), copy them to the micro SD card in the camera, and tune some of the settings.

Ideally the camera should also be detected, as a storage device, so that we can easily copy files and edit parameters, and in my computer it was shown as a UVC camera, a USB ACM device, and USB storage device when I connect it:

But for some reasons, I could not see the /dev/sdb storage after that:

[Update: We can use use jevois-usbsd script to access the camera storage from the host computer / board:

]

So instead I had to take out the micro SD card from the camera, and copy the files in /modules/JeVois/SaliencySURF/images/ directory in JEVOIS partition.

The module will process those photo when we start it, and return the name of the file when detected.

We can go back to SaliencySURF directory to edit params.cfg file, and change some parameters to determine how strict a match should be, taking into account that a stricter matching may mean the object was not be detected, and looser matching that we get some false positives. But this is where it gets a little more complicated, as we’ll see from a subset of the list of parameters.

Click to Enlarge

I cannot understand what half of the parameters are supposed to do. That’s where you can click on the SaliencySURF / Saliency links to access the base documentation. and find out how the module works, find out more about each parameter, and easily access the source code for the functions used by the module. That type of documentation is available for all modules used in JeVois C++ framework, and it’s a very good learning tool for people wanting to know more about computer vision. You’ll have to be familiar with C++ to understand the code, and what it really does, beside learning jargon and acronyms specific to computer vision or machine learning.

By default params.cfg file includes just two lines:

Those are the parameters for ObjectMatcher module, with goodpts corresponding to the number range of good matches considered, and distthresh being the maximum distance for a match to be considered good.

I’ve set looser settings in params.cfg:

I saved the file, put the micro SD card back into the camera, and launch guvcview with 320×288 @ 30 fps resolution/framerate to enter SaliencySURF mode.

Click to Enlarge

Oops, it’s seeing Tux logos everywhere, even when there are none whatsoever, so our settings are clearly too loose. So I went back to the default settings, but the rsults was still similar, so since the distance was shown to be 0.30 in my first attempt, I reduced distthresh to 0.2. False positive are now mostly gone, except for very short period od time, and it’s now detecting CNX Tux logo accuractely. Note that Green square is for object detection, and the white squares for saliency zones.

However, it struggles to detect my third Tux logo repeatedly, often following back to CNX Tux logo.

But as you could see with the green square, the detection was done on the left flap of the penguin. That’s because SaliencySURF detection is done in a fixed size zone (64×64 pixels per detault), so camera distance, or size of the zone matter. You can change the size of the salient regions with SaliencySURF rsiz parameter which defined the height and length of the quare in pixel. When I did the test I first tried to detected if from the list of Tux images from DuckDuckGo search ut it was not small or blurry. After switchign to a bigger photo, the cable was too short to cover the logo, so instead I copied to gimp and resized it so that it could fit in the 64×64 square while using the camera, and in this case detection worked resaonably well.

The more you use the camera, the better you’ll be at understanding how it works, and leverage its capabilities.

Final Words

JeVois-A33 camera is an inexpensive way to get started with computer vision and deep learning, with excellent documentation, and if you put the efforts, you’ll even understand how it works at the source code level. It’s also fun to use with many different modules to try. I have not tried it n this review due to time limitations, but you could also connect the camera to an Arduino board controlling a robot (Cat chasing bot anyone?) via the serial interface.

The main challegenges you may face while getting started ar:

  1. Potential crashes due to power issues, but that’s solvable, and a power issues troubleshooting guide has even been published
  2. For robotics projects, you have to keep in mind there will be some lag for some modules, for example from 500ms (single object) to 3 seconds (YOLO test with multiple object types) for deep learning algorithms. Other modules such as ArUco marker are close to real-time performance however.

Bear in mind all processing is done by the Allwinner A33 CPU cores, as the Mali-400MP GPU is not suitable for GPGPU. As more affordable SoC with OpenCL/Vulkan capable GPU (e.g. Mali-T720) are launched, and in some cases even NNA (Neural Network Accelerator), we’ll be able to get similar low power smart cameras, but with much better computer vision performance.

JeVois-A33 can be purchased for $49, but to avoid wasting time with power issues, and give you more options, I’d recommend to go with JeVois-A33 Developer/Robotics Kit reviewed here, going for $99.99 on Amazon, RobotShop, or JeVois Store.

JeVois Smart Machine Vision Camera Review – Part 1: Developer / Robotics Kit Unboxing

October 24th, 2017 No comments

JeVois-A33 computer vision camera was unveiled at the end of last year through a Kickstarter campaign. Powered by an Allwinner A33 quad core Cortex A7 processor, and a 1.3MP camera sensor, the system could detect motion, track faces and eyes, detect & decode ArUco makers & QR codes, follow lines for autonomous cars, etc.. thanks to JeVois framework.

Most rewards from KickStarter shipped in April of this year, so it’s quite possible some of the regular readers of this blog are already familiar the camera. But the developer (Laurent Itti) re-contacted me recently, explaining they add improves the software with Python support, and new features such as the capability of running deep neural networks directly on the processor inside the smart camera. He also wanted to send a review sample, which I received today, but I got a bit more than I expected, so I’ll start the review with an unboxing of they call the “Developer / Robotics Kit”.

I got the kit in a white package, so I’ll skip the photo, and checking out directly the content.

Click to Enlarge

I was really expecting to receive a tiny camera, and not much else. So my first reaction was: “what!?” 🙂

You’ll find 5 mini USB cables inside (from top left to bottom middle):

Power Bank Info

  • USB to micro serial adapter cable, 1m long, to access the serial console in the camera when running in debug mode, or while troubleshooting Arduino code
  • mini USB + micro USB splitter cable, 15cm long, to power both the camera and Arduino board from the power bank
  • mini USB Y cable, 80cm long, to power the board via two USB 2.0 ports or to one USB 3.0 port on your host computer
  • mini USB cable, 23cm long, to power the camera from a USB port or power bank.
  • mini USB cable, 75cm long, to connect the camera to one USB 3.0 port or power bank.

The kit also includes an 8GB micro SD card pre-loaded with JeVois software, an SD adapter, a micro SD card reader, a 5V USB tester compatible with QuickCharge 2.0 to monitor the power consumption of the camera with your chosen algorithm, a 2,600 mAh power bank (large enough to power the camera for several hours), an Arduino compatible Pro mini board based on Microchip Atmel Atmega 32U4 MCU, and a business card providing useful information such as a link to a Quick Start Guide.

Oh… I almost forgot. Can you see the “fan” in the middle of photo above? That’s the actual JeVois-A33 camera. I knew it was small, but once you put it into your hands, you realize how tiny it is. The cable on the left of the camera is a micro serial cable to connect to an MCU board.

Click to Enlarge

The back of the camera features all the ports and connectors with a micro SD slot, a mini USB port, the micro serial port connector (which looks like a battery connector), and a dual color LED on left of the micro serial connector that indicates power and camera status.

Click to Enlarge

The bottom reveals an opening to cool down AXP223 PMIC.

Click to Enlarge

If you’re interested in the exact developer/robotics kit I’ve received, you can purchase it for $99.99 on JeVois, Amazon, or RobotShop (with locations in US, Canada, Japan, and France). But if you just want the camera without all cable and accessories, $49.99 will do.