Archive

Posts Tagged ‘camera’

Haven Open Source App Transforms Your Old Android Smartphone into a Smart Security Camera

December 23rd, 2017 3 comments

About two years ago, I wrote a post asking what to do with old devices instead of throwing them away. My own proposals included giving them away, reselling them on eBay, recycling them for other purpose like servers or download clients, or scavenging some parts. Other people also comments what they did with theirs, for example setting up a Linux cluster with old TV boxes.

Another way to recycling an old (Android) smartphone – albeit you could always buy an inexpensive one – is to install and run Haven, an open source app that transforms your phone into some sort of smart security camera, but instead of only using the camera from the phone, the app also logs audio events using its microphone (array), as well as data reported by sensors.

Click to Enlarge

One of you first reaction might be: “cool! somebody may an app that would allow hackers or government to make spying on your ever easier”. But actually, the app was initially intended to protect journalists against raids, or more exactly record their occurrence (as proof), and is released by the Guardian Project that aims to “create secure apps, and open-source software libraries that can be used around the world by any person looking to protect their communications and personal data from unjust intrusion, interception and monitoring”.  Haven can also be used to monitor anything you care about, or even as a baby monitor for instance.

While audio and video is continuously monitored, the app only logs the data inside the phone if “thresholds” are exceeded (e.g. motion sensing, audio level…). If you decide to enable notifications, it does not transform your smartphone into another IoT device that relies on the cloud, but instead leverages Signal secure communication app, and the Tor network via Orbot app. A SIM card is not needed, unless you plan to use the optional (and less secure) SMS options.

Commercial Products vs Haven – Click to Enlarge

The app only runs in Android, but iPhone users can still receive notifications via Signal + Tor, they’d just need to buy a cheap Android phone acting as the “camera”. You may want to check out the presentation slides for a quick overview, and visit the app page for more details.

The app can be downloaded from the Google Play Store, F-Droid, or as an apk, and the source code can be cloned from Github.

Apertus AXIOM Beta Open Source Professional Digital Cinema is Built around MicroZed Board

December 20th, 2017 No comments

Apertus AXIOM Beta is a professional digital cinema camera built around FOSS (Free and Open Source Software) and open hardware licenses. The project started around year 2011 with AXIOM Alpha camera, and AXIOM Beta is the latest iteration powered by MicroZed development board based on Xilinx Zynq 7020 ARM + FPGA SoC, and running Arch Linux ARM.

Developer Kit – Click to Enlarge

AXIOM Beta developer kit (planned) hardware specifications:

  • “Linux” Board – Xilinx Zynq 7020 based MicroZed board
  • Beta Main Board – Hosts two external medium-speed shield connectors and two high-speed plugin module slot connectors.
  • Image Sensor – 12MP CMV12000 (Used for research and development) via CMV12K ZIF Sensor Board
  • Lens Mount Passive E-mount
  • Ports – USB / USB UART / JTAG / Gigabit Ethernet
  • Modules and Shields
    • Single HDMI Full HD (4:4:4) output at up to 60 FPS
    • Dual 6G SDI output (in development)
    • 3x PMOD debug module
    • LED matrix debug module
    • Genlock, Trigger, Timecode, LANC shields (in development)
    • 4K Displayport/HDMI (in development)
  • Power Supply – 5V/5A via power adapter board; Other voltages provided via Beta Power Board
  • Dimensions -111.76 x 74 x 65.1 mm (devkit)
  • Weight – 319 grams (devkit)

There’s also a Beta Interface dummy board that acts as a bridge between the image sensor board and the rest of the camera.


The camera will run Arch Linux ARM on MicroZed board, support common network protocols (SSH/FTP/SCP/etc), and be configurable via a web interface. Features will include global shutter capture, output of 4K RAW experimental HDMI/Displayport outputs over 1080p60, remote control, WiFi connectivity, support for  motion tracking via various sensors for image stabilization, as well as image processing with Look-Up-Tables (LUTs), matrix color conversion, dead pixel compensation, and so on. However, audio recording is not currently supported. Many more software and hardware details can be found in the Wiki.

Click to Enlarge

A compact enclosure is also planned for the production camera that will be called AXIOM Beta Compact, and while the timeline is unclear, you can register your interest for what is planned to be a 5,990 Euros camera. Some of the development kits pictured above have been shipped last summer, and can still be ordered for 3,990 Euros ex. VAT. FYI, the image sensor represents around 2,000 Euros out of the total cost. Visit the product page for more details.

Rendering of Expect Final Product (AXIOM Beta Compact)

Qualcomm Snapdragon 845 Octa Core Kryo 385 SoC to Power Premium Smartphones, XR Headsets, Windows Laptops

December 7th, 2017 9 comments

Qualcomm Snapdragon 845 processor was expected since May 2017 with four custom Cortex A75 cores, four Cortex A53 cores, Adreno 630 GPU, and X20 LTE modem. with the launch planned for Q1 2018. At least, that what the leaks said.

Qualcomm has now formally launched Snapdragon 845 Mobile Platform and rumors were mostly right, as the the octa-core processor comes with four Kryo 385 Gold cores (custom Cortex A75), four Kryo 385 Silver cores (custom Cortex A55) leveraging DynamIQ technology, an Adreno 630 “Visual Processing System”, and Snapdragon X20 modem supporting LTE Cat18/13.

The processor is said to use more advanced artificial intelligence (AI) allowing what the company calls “extended reality (XR)” applications, and will soon be found in flagship smartphones, XR headsets, mobile PCs, and more.

Qualcomm Snapdragon 845 (SDM845) specifications:

  • Processor
    • 4x Kryo 385 Gold performance cores @ up to 2.80 GHz (custom ARM Cortex A75 cores)
    • 4x Kryo 385 Silver efficiency cores @ up to 1.80 GHz (custom ARM Cortex A55 cores)
    • DynamIQ technology
  • GPU (Visual Processing Subsystem) – Adreno 630 supporting OpenGL ES 3.2, OpenCL 2.0,Vulkan 1.x, DxNext
  • DSP
    • Hexagon 685 with 3rd Gen Vector Extensions, Qualcomm All-Ways Aware Sensor Hub.
    • Supports Snapdragon Neural Processing Engine (NPE) SDK, Caffe, Caffe2, and Tensorflow
  • Memory I/F – LPDDR4x, 4×16 bit up to 1866MHz, 8GB RAM
  • Storage I/F – TBD (Likely UFS 2.1, but maybe UFS 3.0?)
  • Display
    • Up to 4K Ultra HD, 60 FPS, or dual 2400×2400 @ 120 FPS (VR); 10-bit color depth
    • DisplayPort and USB Type-C support
  • Audio
    • Qualcomm Aqstic audio codec and speaker amplifier
    • Qualcomm aptX audio playback with support for aptX Classic and HD
    • Native DSD support, PCM up to 384kHz/32bit
  • Camera
    • Spectra 280 ISP with dual 14-bit ISPs
    • Up to 16 MP dual camera, up to 32 MP single camera
    • Support for 16MP image sensor operating up to 60 frames per second
    • Hybrid Autofocus, Zero Shutter Lag, Multi-frame Noise Reduction (MFNR)
    • Video Capture – Up to 4K @ 60fps HDR (H.265), up to 720p @ 480fps (slow motion)
  • Connectivity
    • Cellular Modem – Snapdragon X20 with peak download speed: 1.2 Gbps (LTE Cat 18), peak upload speed: 150 Mbps (LTE Cat 13)
    • Qualcomm Wi-Fi 802.11ad Multi-gigabit, integrated 802.11ac 2×2 with MU-MIMO, 2.4 GHz, 5 GHz and 60 GHz
    • Qualcomm TrueWireless Bluetooth 5
  • Location – Support for 6 satellite systems: GPS, GLONASS, Beidou, Galileo, QZSS, SBAS; low power geofencing and tracking, sensor-assisted navigation
  • Security – Qualcomm Secure Processing Unit (SPU), Qualcomm Processor Security, Qualcomm Mobile Security, Qualcomm Content Protection
  • Charging – Qualcomm Quick Charge 4/4+ technology
  • Process – 10nm LPP

The company will provide support for Android and Windows operating systems. eXtended Reality (XR) is enabled with features such as room-scale 6DoF with simultaneous localization and mapping (SLAM), advanced visual inertial odometry (VIO), and Adreno Foveation. Maybe I don’t follow the phone market closely enough, but I can’t remember seeing odometry implemented in any other phones, and Adreon Foveation is not quite self-explaining, so the company explains it combines graphics rendering with eye tracking, and directs the highest graphics resources to where you’re physically looking, while using less resources for rendering other areas. This improves the experience, performance, and lower power consumption.

 

Click to Enlarge

Compared to Snapdragon 835, the new processor is said to be around 25 to 30% faster, the Spectra camera and Adreno graphics architectures are claimed to boost power efficiency by up to 30 percent, and the LTE modem is a bit faster (1.2 Gbps/150Mbps vs 1.0 Gbps/150Mbps). Quick Charge 4+ technology should deliver up  to 50 percent charge in 15 minutes. Earlier this year when SD835 was officially launched, there was virtually no mention of artificial intelligence support in mobile APs, but now NNA (Neural Network Accelerator) or NPE (Neural Processing Engine) are part of most high-end mobile processors, which in SD845 appears to be done though the Hexagon 685 DSP. High Dynamic Range (HDR) for video playback and capture is also a novelty in the new Snapdragon processor.

One of the first device powered by Snapdragon 845 will be Xiaomi Mi 7 smartphone, and according to leaks it will come with a 6.1″ display, up to 8GB RAM, dual camera, 3D facial recognition, and more. Further details about the phone are expected for Mobile World Congress 2018. Considering the first Windows 10 laptop based on Snapdragon 835 processor are expected in H1 2018, we may have to wait until the second part of the year for the launch of Snapdragon 845 mobile PCs.

More details may be found on Qualcomm Snapdragon 845 mobile platform product page.

AWS DeepLens is a $249 Deep Learning Video Camera for Developers

November 30th, 2017 4 comments

Amazon Web Services (AWS) has launched Deeplens, the “world’s first deep learning enabled video camera for developers”. Powered by an Intel Atom X5 processor with 8GB, and featuring a 4MP (1080p) camera, the fully programmable system runs Ubuntu 16.04, and is designed expand deep learning skills of developers, with Amazon providing tutorials, code, and pre-trained models.

Click to Enlarge

AWS Deeplens specifications:

  • Camera – 4MP (1080p) camera using MJPEG, H.264 encoding
  • Video Output – micro HDMI port
  • Audio – 3.5mm audio jack, and HDMI audio
  • Connectivity – Dual band WiFi
  • USB – 2x USB 2.0 ports
  • Misc – Power button; camera, WiFi and power status LEDs; reset pinhole
  • Power Supply – TBD
  • Dimensions – 168 x 94 x 47 mm
  • Weight – 296.5 grams

The camera can not only do inference, but also train deep learning models using Amazon infrastructure. Performance wise, the camera can infer 14 images/second on AlexNet, and 5 images/second on ResNet 50 for batch size of 1.

Six projects samples are currently available: object detection, hot dog not hot dog, cat and dog,  activity detection, and face detection. Read that blog post to see how to get started.

But if you want to make your own project, a typical workflow would be as follows:

  • Train a deep learning model using Amazon SageMaker
  • Optimize the trained model to run on the AWS DeepLens edge device
  • Develop an AWS Lambda function to load the model and use to run inference on the video stream
  • Deploy the AWS Lambda function to the AWS DeepLens device using AWS Greengrass
  • Wire the edge AWS Lambda function to the cloud to send commands and receive inference output

This steps are explained in details on Amazon blog.

Click to Enlarge

Intel also published a press release explaining how they are involved in the project:

DeepLens uses Intel-optimized deep learning software tools and libraries (including the Intel Compute Library for Deep Neural Networks, Intel clDNN) to run real-time computer vision models directly on the device for reduced cost and real-time responsiveness.

Developers can start designing and creating AI and machine learning products in a matter of minutes using the preconfigured frameworks already on the device. Apache MXNet is supported today, and Tensorflow and Caffe2 will be supported in 2018’s first quarter.

AWS DeepLens can be pre-ordered today for $249 by US customers only (or those using a forwarding service) with shipping expected on April 14, 2018. Visit the product page on AWS for more details.

Hisilicon Hi3559A V100ES is an 8K Camera SoC with a Neural Network Accelerator

November 22nd, 2017 3 comments

Earlier today, I published a review of JeVois-A33 machine vision camera, noting that processing is handled by the four Cortex A7 cores of Allwinner A33 processor, but in the future we can expect such type of camera to support acceleration with OpenCL/Vulkan capable GPUs, or better, Neural network accelerators (NNA) such Imagination Tech PowerVR Series 2NX.

HiSilicon already launched Kirin 970 SoC with such similarIP, except they call it an NPU (Neural-network Processing Unit). However, while looking for camera SoC with NNA, I found a list of deep learning processors, including the ones that go into powerful servers and autonomous vehicles, that also included a 8K Camera SoC with a dual core CNN (Convolutional Neural Network) acceleration engine made by Hisilicon: Hi3559A V100ES.

Click to Enlarge

Hisilicon Hi3559A V100ES specifications:

  • Processor Cores
    • 2x ARM Cortex A73 @ 2 GHz, 32 KB I cache, 64KB D cache or 512 KB L2 cache
    • 2x ARM Cortex A53 @ 1 GHz, 32 KB I cache, 32KB D cache or 256 KB L2 cache
    • 1x single core ARM Cortex A53 @ 1 GHz, 32 KB I cache, 32KB D cache /128 KB L2 cache
    • Neon acceleration and integrated FPU
  • GPU – Triple core ARM Mali-G71 [email protected] 900 MHz with 256KB cache, support for OpenCL 1.1/1.2/2.0, and OpenGL ES 3.0/3.1/3.2
  • Sensor Hub
    • ARM Cortex M7 @200 MHz
    • PMC, which supports only external reset, internal POR
    • General peripheral IPs (UART, SPI, I2C, PWM, GPIO,and LSADC)
    • 3-channel LSADC, 5x UART interfaces, and 8x PWM interfaces
  • Memory Interface – 32-/64-bit DDR4 up to 8GB
  • Storage Interfaces – SPI NOR flash up to 512MB, NAND flash, eMMC 5.1 up to 2TB, UFS 2.1 up to 512GB
  • Video Encoding – H.264 BP/MP/HP, and H.265 Main Profile/Main 10 Profile up to 7680 x [email protected] [email protected] fps+7680 x [email protected] fps snapshot
  • Video Decoding – H.264 BP/MP/HP, H.265 MP/Main 10/High Tier up to [email protected] fps or H.264/H.265 [email protected] fps
  • Intelligent Video Analysis
    • Integrated intelligent analysis and acceleration engine, allowing customers to develop intelligent applications targeted for mobile camera products
    • Dual-core DSP @ 700 MHz, 32 KB I cache, 32 KB IRAM, or 512 KB DRAM
    • Dual-core CNN @ 700 MHz neural network acceleration engine
  • Video and Graphics Processing
    • 3DNR, image enhancement, and DCI
    • Anti-flicker for output videos and graphics
    • 1/15.5x to 16x video & graphics scaling
    • Horizontal seamless stitching of 2-channel videos, and 360° or 720° panoramic stitching of up to 6-channel videos
    • OSD overlaying of eight regions before encoding
    • Video graphics overlaying of two layers (video layer and graphics layer)
  • 2- channel ISP
    • Adjustable 3A functions (AE, AWB, and AF)
    • FPN removal
    • Highlight suppression, backlight compensation, gamma correction, and color enhancement
    • DPC, NR, and 6-DOF DIS
    • Anti-fog
    • LDC and fisheye correction
    • Picture rotation by 90° or 270°;  Picture mirror and flip
    • HDR10, BT.2020 WCG
    • Sensor built-in WDR, 4F/3F/2F frame-based/line-based
    • WDR and local tone mapping
    • ISP tuning tools for the PC
  • Audio Encoding and Decoding
    • Voice encoding/decoding complying with multiple protocols by using software
    • MP3, AAC, and other audio encoding formats
    • Audio 3A functions (AEC, ANR, and ALC)
  • Security Engine
    • AES, DES, and 3DES encryption and decryption algorithms implemented by using hardware
    • RSA1024/2048/3072/4096 signature verification algorithm implemented by using hardware
    • SHA1/224/256/384/512 of the HASH and HMAC_SHA1/224/256/384/512 tamper proofing algorithms implemented by using hardware
    • Integrated 32-kbit OTP storage space and hardware random number generator
  • Video Interfaces
    • Input
      • Multiple sensor inputs. The maximum resolution is 32 megapixels (7680 x 4320).
      • 8-/10-/12-/14-bit RGB Bayer DC timing VI, up to 150 MHz clock frequency
      • BT.601, BT.656, and BT.1120 VI interfaces
      • Maximum 16-lane MIPI/LVDS/sub-LVDS/HiSPi/SLVS-EC interface for the serial sensor inputs
      • Maximum 6-channel video inputs for the serial sensor inputs, supporting various working modes such as 1×16-lane/2×8-lane/4×4-lane/2×4-lane+4×2-lane
    • Output
      • HDMI 2.0, supporting maximum [email protected] fps output
      • 8-/16-/24-bit RGB digital LCD output, supporting maximum 1920 x [email protected] fps output
      • 4-lane MIPI DSI output, supporting maximum 2.5 Gbit/s per lane frequency
  • Audio Interfaces
    • Integrated audio codec, supporting 16-bit audio inputs and outputs
    • I2S interface for connecting to the external audio codec
    • Dual-channel differential MIC inputs for reducing background noises
  • Peripherals
    • POR, external reset input,
    • Internal RTC
    • Integrated 2-channel LSADC
    • 5x UART interfaces
    • IR interface, I2C interface, SSP main interface, and GPIO interface
    • Integrated GMAC, supporting  RGMII and RMII
    • 2x PWM interfaces
    • 2x SD 3.0/SDIO 3.0 interfaces, supporting SDXC
    • 2x USB 3.0/USB 2.0 host/device ports
    • 2-lane PCIe 2.0 RC/EP mode
  • Operating Voltages – 0.8V core voltage, 1.8V I/O voltage, 1.2V DDR4 voltage
  • Power Consumption – 2.6 Watts
  • Package – 15 x 15 mm with 0.4 mm pitch

Boy, that’s a monster… They should have called it MOACSoC (Mother of All Camera SoCs) 🙂 The main ARM cores are said to run Linux+Huawei LiteOS AMP heterogeneous dual systems, and the company provide a dedicated SDK for the consumer mobile camera, cient for the iOS and Android mobile phones, and a high-
performance H.265 decoding library. The SDK might be in the wild as “Hi3559AV100ES_SDK_V2.0.2.0” but I did not find a download link. I got all information above from Hi3359A V100ES ultra-HD Mobile Camera SoC product brief.

Mobile Camera and Professional Camera Solution Block Diagram

Based on the block diagram above, some mobile camera and professional camera will start taking SSD drives beside the boring SD card and USB 2.0/3.0 storage devices.

Hi3559A V100ES will also be found in drone cameras, 3D/VR cameras, and 4K/8K network-based EDR. I have no idea what the latter stands for, but the photo in the document looks like a car dashboard camera with display. Anyway, this should allows for some interesting use cases with near real-time object recognition.

Hisilicon showcased a dynamic object categorization and identification system at CPSE2017 in Shenzhen earlier this month. The company did not mention Hi3559A V100, but made clear an 8K solution was used.

Click to Enlarge

If we are to believe one person on Tencent’s ncnn Github repo, performance is really good with a 10ms lag for GoogleNet, and 89ms for VGG-SSD. We’ll have to wait a little to get more details, and Hisilicon did not post any product info on their website about their new 8K SoC, only about the earlier Hi3559 2K/4K SoC.

JeVois-A33 Linux Computer Vision Camera Review – Part 2: Setup, Guided Tour, Documentation & Customization

November 22nd, 2017 4 comments

Computer Vision, Artificial Intelligence, Machine Learning, etc.. are all terms we hear frequently those days. JeVois-A33 smart machine vision camera powered by Allwinner A33 quad core processor was launched last year on Indiegogo to bring such capabilities in a low power small form factor devices for example to use in robotics project.

The company improved the software since the launch of the project, and has now sent me their tiny Linux camera developer kit for review, and I’ve already checked  out the hardware and accessories in the first post. I’ve now had time to test the camera, and I’ll explained how to set it up, test some of the key features via the provided guided tour, and show how it’s possible to customize the camera to your needs with one example.

Getting Started with JeVois-A33

In theory, you could just get started by inserting the micro SD card provided with the camera, connect it to your computer via the USB cable, and follow the other instructions on the website. But to make sure you have the latest features and bug fixed, you’d better download the latest firmware (jevois-image-latest-8G.zip), and flash it to the micro SD card with the multi-platform Etcher tool.

You could also use your own micro SD card, as long as it has 8GB or more capacity. Once this is done, insert the micro SD card into the camera with the fan of the camera and the golden contact of the micro SD card both facing upwards. Connect the camera to your computer with the provided mini USB to USB cable. I also added the USB power meter to monitor the power consumption for the different use cases, and USB serial cable to checkout output from the console. At least that was the plan, but I got no lights from the camera, and voltage was reported to be only 4V. Then I read the guide a little better, and found out I had to use a USB 3.0 port, or two USB 2.0 ports for power.

Once I switched to using two USB 2.0 ports from a powered USB 2.0 hub, I could see output from the serial console…

and both green and orange/red LEDs were lit. The instructions to use JeVois camera are mostly OS agnostic, except for the video capture software. If you are using Windows you can use the free OBS Studio or AMCap programs, and on Mac, select either PhotoBooth or OBS Studio. I’m a Ubuntu user, so instead I installed guvcview:

and ran it use 640×360 resolution and YUYV format as instructed in the getting started guide:

But then I got no output at all in the app:

The last line above would repeat in a loop. The kernel log (dmesg) also reported a crash linked to guvcview:

Another person had the same problem a few months ago, and it was suggested it may be a USB problem. So I connect the camera to directly to two of the USB ports on my tower, and it worked…

Click to Enlarge

The important part of the settings are in the Video Controls tab, where we can change resolution and frame rate to switch between camera modes as we’ll see later on.

But since my tower is under the desk, the USB cable is a bit too short, and the program crashed with the same error message a few minutes later. So I went with my Ubuntu 16.04 laptop instead. Powering the camera via the USB 3.0 port worked until I started deep learning modes, where the camera would stop, causing guvcview to gray out. Finally, I connected the camera to both my USB 3.0 port, and the power bank part of the kit, and the system was then much more stable.

Click to Enlarge

I contacted the company about the issues I had, but they replied this problem was not often reported:

… we have only received very few reports like that but we were able to confirm here using front panel ports on one machine. On my desktop I have a hub too, but usb3 and rated for fast charging (60W power supply for 7+2 ports) and it works ok with jevois. A single usb3 port on my mac laptop is also ok.

So maybe it’s just me with all my cheap devices and accessories…

So three main points to get started:

  1. Update the firmware
  2. Install the camera software
  3. Check power in case of issues / crashes (Both LEDs should be on if the camera is working)

JeVois-A33 Guided Tour

Now we have the camera running, we can try the different features, and the best way to do so is to download Jevois Guided Tour (PDF) that will give you an overview of the camera and how it works, as well as examples.

Click to Enlarge

As shown above, the PDF includes information for each module with the name, link to documentation, introduction, display explanation, and on the top right the resolution/framerate that can be used to launch a given module. On following pages, there will be example pictures that you can point to with the camera.

Some of modules include:

  • Visual attention – finding interesting things
  • Face and handwritten digit recognition
  • QR-codes and other tags
  • Road detection
  • Object matching
  • Object recognition with deep neural networks
  • Color-based object tracking
  • Moving object detection
  • Record video to the microSD card inside JeVois
  • Motion flow detection
  • Eye tracking
  • and more…

You could print the guide with a color printer, but the easiest way is problem to use two screens, once with the PDF guide open, and the other running the camera application (guvcview, OBS Studio…). I’ve gone through some of the example in the guided tour in the video below, with PDF shown on a TV box, and the camera application output shown on the laptop screen on the top bottom corner.

That’s lot of fun, and everything works pretty well most of the time. Some of the tests are quite demanding for such low power device, as for example Darknet based “Deep neural scene analysis” using 1280×480 @ 15 fps with the ability to recognize multiple object types would only refresh the results every 2.7 seconds or so.

Documentation & Customization of Salient SURF Module

If you’ve gone through the guide tour, you should now have a good understanding of what the camera is capable of. So now, let’s take one of the modules, and try to adjust it to our needs. I picked SaliencySURF module with the documentation available here for this section of the review. Introduction for the module:

Trained by default on blue iLab logo, point JeVois to it and adjust distance so it fits in an attention box.
Can easily add training images by just copying them to microSD card.
Can tune number and size of salient regions, can save regions to microSD to create a training set

So let’s take a few other images (Tux logo), copy them to the micro SD card in the camera, and tune some of the settings.

Ideally the camera should also be detected, as a storage device, so that we can easily copy files and edit parameters, and in my computer it was shown as a UVC camera, a USB ACM device, and USB storage device when I connect it:

But for some reasons, I could not see the /dev/sdb storage after that:

[Update: We can use use jevois-usbsd script to access the camera storage from the host computer / board:

]

So instead I had to take out the micro SD card from the camera, and copy the files in /modules/JeVois/SaliencySURF/images/ directory in JEVOIS partition.

The module will process those photo when we start it, and return the name of the file when detected.

We can go back to SaliencySURF directory to edit params.cfg file, and change some parameters to determine how strict a match should be, taking into account that a stricter matching may mean the object was not be detected, and looser matching that we get some false positives. But this is where it gets a little more complicated, as we’ll see from a subset of the list of parameters.

Click to Enlarge

I cannot understand what half of the parameters are supposed to do. That’s where you can click on the SaliencySURF / Saliency links to access the base documentation. and find out how the module works, find out more about each parameter, and easily access the source code for the functions used by the module. That type of documentation is available for all modules used in JeVois C++ framework, and it’s a very good learning tool for people wanting to know more about computer vision. You’ll have to be familiar with C++ to understand the code, and what it really does, beside learning jargon and acronyms specific to computer vision or machine learning.

By default params.cfg file includes just two lines:

Those are the parameters for ObjectMatcher module, with goodpts corresponding to the number range of good matches considered, and distthresh being the maximum distance for a match to be considered good.

I’ve set looser settings in params.cfg:

I saved the file, put the micro SD card back into the camera, and launch guvcview with 320×288 @ 30 fps resolution/framerate to enter SaliencySURF mode.

Click to Enlarge

Oops, it’s seeing Tux logos everywhere, even when there are none whatsoever, so our settings are clearly too loose. So I went back to the default settings, but the rsults was still similar, so since the distance was shown to be 0.30 in my first attempt, I reduced distthresh to 0.2. False positive are now mostly gone, except for very short period od time, and it’s now detecting CNX Tux logo accuractely. Note that Green square is for object detection, and the white squares for saliency zones.

However, it struggles to detect my third Tux logo repeatedly, often following back to CNX Tux logo.

But as you could see with the green square, the detection was done on the left flap of the penguin. That’s because SaliencySURF detection is done in a fixed size zone (64×64 pixels per detault), so camera distance, or size of the zone matter. You can change the size of the salient regions with SaliencySURF rsiz parameter which defined the height and length of the quare in pixel. When I did the test I first tried to detected if from the list of Tux images from DuckDuckGo search ut it was not small or blurry. After switchign to a bigger photo, the cable was too short to cover the logo, so instead I copied to gimp and resized it so that it could fit in the 64×64 square while using the camera, and in this case detection worked resaonably well.

The more you use the camera, the better you’ll be at understanding how it works, and leverage its capabilities.

Final Words

JeVois-A33 camera is an inexpensive way to get started with computer vision and deep learning, with excellent documentation, and if you put the efforts, you’ll even understand how it works at the source code level. It’s also fun to use with many different modules to try. I have not tried it n this review due to time limitations, but you could also connect the camera to an Arduino board controlling a robot (Cat chasing bot anyone?) via the serial interface.

The main challegenges you may face while getting started ar:

  1. Potential crashes due to power issues, but that’s solvable, and a power issues troubleshooting guide has even been published
  2. For robotics projects, you have to keep in mind there will be some lag for some modules, for example from 500ms (single object) to 3 seconds (YOLO test with multiple object types) for deep learning algorithms. Other modules such as ArUco marker are close to real-time performance however.

Bear in mind all processing is done by the Allwinner A33 CPU cores, as the Mali-400MP GPU is not suitable for GPGPU. As more affordable SoC with OpenCL/Vulkan capable GPU (e.g. Mali-T720) are launched, and in some cases even NNA (Neural Network Accelerator), we’ll be able to get similar low power smart cameras, but with much better computer vision performance.

JeVois-A33 can be purchased for $49, but to avoid wasting time with power issues, and give you more options, I’d recommend to go with JeVois-A33 Developer/Robotics Kit reviewed here, going for $99.99 on Amazon, RobotShop, or JeVois Store.

Xiaomi Mi A1 Smartphone Review – Part 2: Android 7.1.2 Firmware

November 15th, 2017 12 comments

Google recently announced several Android One smartphones, which are supposed to get 2 years of firmware updates, including to the latest version of Android, such as HTC U11 Life and Android One Moto X4. Many of those phones are limited to some specific countries, but Xiaomi Mi A1 will be launched in over 40 countries, and thanks to Chinese online shops is really available worldwide. GearBest sent me the latter last month, and in the first part of Xiaomi Mi A1 review I simply went through unboxing, booted the phone, perform an OTA update, and ran Antutu 6.x on the phone for a quick estimate of performance.

Since then, I’ve had around four weeks to play with the smartphone running Android 7.1.2 (still), so I’m ready to report my experience in the second part of the review.

Click to Enlarge

General Impressions

In the past year, I used Vernee Apollo Lite smartphone powered by Mediatek Helio X20 deca-core SoC, which in theory is quite faster than the Qualcomm Snapdragon 625 used in Mi A1, but in practise, I did not feel much difference in performance for example while browsing the web or checking email, and in some games, performance of Xiaomi Mi A1 was actually much better than on Apollo Lite, as I reported in the post entitled “Mediatek Helio X20 vs Qualcomm Snapdragon 625 – 3D Graphics Benchmarks and CSR 2 Game“.

Some of my wishes in Vernee Apollo Lite included a better camera, and improved GPS accuracy, and Mi A1 is a big improvement for both as we’ll see in more details later on in the review. The build quality of the phone is good, and the design looks more stylish and thinner than my previous phone. The display is clear, and I like the wide brightness range, that is low enough not to hurt eyes in the dark, and high enough to use the phone in sunlight. It’s quite glossy though, so you’ll have reflect especially with black background, and it’s possibly to use it as a mirror without turning it on… I seldom call with my phone, but the couple of times I made or received actual calls, the sound was loud and clear. I spend most of my time browsing the web, checking emails, watching YouTube video, and playing games (mostly CSR 2) on my phone, and do so over WiFi connection, and the phone just works flawlessly for this with good performance, and no overheating (that I could notice) contrary to Vernee Apollo Lite, which does get hot in some cases, and slows down considerably.

I’m also happy about battery life, and with my use case of hour 4 to 5 hours use a day, I can still get around 30 hours on a charge. One of the downside is the lack of fast charging, so I can’t quickly top of the battery for 5 minutes before going out. A full charge takes around 1h30, so still not too bad, and since the battery lasts more than 24 hours, it would be possible to charge every day at the same time to avoid low battery charge while on the go.

The main selling of the phone is being part of Android One program, as you’ll get security updated once or twice a month, as well as bigger Android version updates for two years hopefully up to Android 9 / P.  You do pay a premium for this, so if regular security/firmware updates are not important to you, you’ll get better value with other smartphones.

Overall, I’m very satisfied with Xiaomi Mi A1 smartphone, I could not find any major flaws so far, so I can safely recommend it especially if having up-to-date firmware (for the next two years) is important to you.

Benchmarks: Antutu, Vellamo, and 3DMarks

Here are Antutu 6.x benchmark results for people who have yet to read the first part of the review.

Click to Enlarge

60,000 points is a decent score for a mid-range phone, but for example quite lower than the 85,840 points I got on Vernee Apollo Lite.

Next up… Vellamo 3.x benchmark. Comparisons are against older phone / Android version, so I should probably drop that benchmark in future reviews…

Click to Enlarge

Note that I could not run Vellamo with Chrome browser, since it would hang during CSS 3D animation. Firefox mostly worked, except for Pixel Bender test timing out… The number are all much lower than my Vellamo results on Vernee Apollo Lite.

So I also ran GeekBench 4. AFAIK, It’s however limited to CPU performance so it does not really give real world indication like Vellamo’s Browser test for example.

Click to Enlarge

We can see the single core performance is quite lower than more powerful Cortex A72 “class” processors, but multicore performance is close enough. You can find the full details here.

I also ran 3DMark Ice Store Extreme for evaluation 3D performance further. Vernee Apollo Lite would max out the test, but Xiaomi Mi A1 scored “only” 8,045 points.

Click to Enlarge

The interested part is that my real-life experience does not match the benchmarks at all, as I found Mi A1 to perform just as well as if not much better in many apps. We’ll find out one potential reason just below.

Storage and Wi-Fi Performance

I ran A1 SD Benchmark app to estimate storage performance of 64 GB eMMC flash, and Xiaomi Mi A1 has by far the best storage I’ve used on any devices.

With sequential read speed of 198.94 MB/s, and a write speed of 192.45 MB/s, the device is in a class of its down. Ideally, random I/O performance should be tested too, but it still gives an indication.

Read & Write Speeds in MB/s – Click to Enlarge

Time for some WiFi testing. I did not have any issues, and felt web pages were always loading fast, and YouTube videos played smoothly even at 1080p. But let’s have some numbers to play using SAMBA file copy (278MB) over 802.11ac WiFi  with ES File Explorer, and placing the phone is the same test location as the other DUTs. Just like many recent devices SAMBA “download” is much faster than “upload”:

  • File copy SAMBA to Flash – 47.5s on average (5.85 MB/s)
  • File copy Flash to SAMBA – 2m10s on average (2.13 MB/s)

When we average both numbers, Xiaomi WiFi SAMBA performance is only slightly above average, but still outperformed by some 802.11n devices.

Throughput in MB/s – Click to Enlarge

Maybe that’s an Android Nougat bug… In order to have raw numbers, I also used  iperf for both upload and download

  • 802.11ac WiFi upload:

  • 802.11ac WiFi download:

Assymetry is gone, and Mi A1 is the best device in that test, but we have less data for comparison…

Throughput in Mbps

The main takeaway is that WiFi is working well, and performance is very good.

Rear and Front Facing Cameras

Beside being part of Android One program, another key feature of Xiaomi Mi A1 smartphone is the dual rear camera with optical zoom.

Rear Camera

So I’ve taken a few shots with the camera, starting with an easy cat shot… The thing that surprised me the most at first was the speed at which the photo is taken. It just happens instantaneously. With older devices, I often had to wait around one second after pressing the button while it was doing the auto-focus and take photos. You can launch the camera app very quickly – without having to unlock your phone – by pressing the power button twice.

“What do you want?” Cat – Click for Original Size

Clear enough for a camera phone. Close up shots are sometimes problematic with phone, but I had pretty good results. The text book shot is close to perfect.

I used to Read that Stuff – Click for Original Size

Development board can be tricky to photograph because the camera can focus on the wrong part (e.g. top of Ethernet/USB connector), But Orange Pi One photo below is fairly good. I had to try a few times to get the right focus.

 

Best.Board.Ever? – Click for Original Size

You can press on the live view to set the focus point. It will help.

Click for Original Size

Click for Original Size

Flower photos were also good with color matching reality.

Local Angel – Click for Original Size

Photos taken with good lighting are always good.

What year is this? Morning Shot – Click for Original Size

The photo above was taken in the morning with the sun in my back.

Dirt Road Genocide at Sunset – Click for Original Size

When it get a little dark, or in shots with different lighting conditions for foreground and background it helps to enable HDR function.

HDR Works in Temples Too – Click for Original Size

Night shots can be a little grainy, but I find they are still pretty good…

Smoking Bear with Pig and Hedgehog overlooked by Confused Panda at Night – Click for Original Size

Now some video testing, starting with the easiest of all 1080p30 day time video.

The video looks fine, but if you’ve watched it with audio, you may have noticed may not be quite right with the microphone/audio.

4K video can be recorded at 30 fps, but it does feel as smooth as the 1080p one while panning.

All videos are recorded using MP4 Quicktime container, H.264 video codec @ 30 fps, and MPEG-4 AAC stereo audio. If you plan to watch 4K videos from the phone on TV, you’ll have to make sure the player supports 4K H.264 @ 30 fps, as some 4K TV boxes are limited to 24 fps.

Slow motion recording is something that I did not have in my previous phone, and it’s working fairly well up to 720p30 (recorded at 120 fps).

Night time videos are the most difficult, and even at 1080p the results are quite poor with the video frame rate at 14 fps, auto focus being seriously confused, and and audio has a metallic component to it, even more than for the video I recorded during day time.

So I tried again to shot a video will taking to myself, and audio was just fine. So I guess the issue may be specific to far field audio or traffic noise.

Font-facing camera

The front-facing camera works pretty well for selfies.

Angel with Bra – Click for Original Size

Golden Necklace Beauty – Click for Original Size

Black “The Boss” – Click for Original Size

I also used it with a one hour long Skype call.

Camera App Settings

Let’s have a look at the camera app interface. In the preview window we have three icons at the top to change flash settings, enable/disable portrait mode (if enabled it will bur the background), and enable/disable HDR.

Click to Enlarge

If we tap on Options we’ll have the choice to play with Panorama mode, adjust timer and audio settings, set manual camera settings for white balance, exposure time, focus, ISO, lens selection (wide/tele), and more. Tapping the Settings icon on the top right corner will bring further camera settings.

Click to Enlarge

If you plan to share photos with strangers you may want to disable “Save location info” as otherwise your GPS location will be embedded into the photos’s EXIF info. Face detection is nice, but you may consider disabling “Age & gender”, as it will automatically detect whether a person is male or female, and estimate their age while taking a photo (although it won’t show on the photo itself). I’ve seen the phone misgender people, and age can always be a contentious subject 🙂

Click to Enlarge

If we switch to video capture we have much fewer options, mostly time-lapse or slow-motion, and we can select video quality (4K, FULL HD, HD, SD).

Battery Life

Xiaomi Mi A1’s ~3,000 mAh baterry provides enough juice for over 30 hours in my use cases (Web browsing / YouTube / Gaming 4 to 5 hours a day). I also like to turn off my phone automatically at night between 22:00 and 7:00, so it adds a little to the battery life too. A typical cycle for charge to charge looks as the one below.

I normally use LAB501 Battery Life app to test battery life from 100% to 15% for browsing, video and gaming cases, with brightness to 50%, WiFi and Cellular (no data) enabled, but for some reasons I cannot explain, the tests would always stop after a few hours – despite several attempts -, not drawing the battery down to 15%.

Click to Enlarge

However, the battery discharge on this phone, and Vernee Apollo Lite looks linear…

…so I’ll use linear approximation to estimate the actual battery life..

  • Browsing (100% to 15%) – 740 minutes (12h20)
  • Video (100% to 15%) –  598 minutes (9h58)
  • Gaming (100% to 15%) –  389 minutes (6h29)

…and compare it to the other battery powered mobile devices I’ve tested so far.

Battery Life in Minutes

Xiaomi Mi A1 wins hands down against the other (older) devices I’ve tested when it comes to battery life. The good news is that battery life seems to improve over the years, as the older devices fare the worse. So a few more years, and we can get a week of charge on our phones?

Charging is not as fast as on Vernee Apollo Lite since there’s no Quick Charge, and it takes 1h30 to 1h50 to fully charge the phone from 15% to 100%. Topping the battery from a low of 8% to 27% took me 23 minutes. For comparison, I could do a full charge in one hour on Apollo Lite with Quick Pump 3.0, and a 20 minute charge would add about 40% to the battery.

Miscellaneous

Bluetooth

No problems here. I could transfer photos between the phone and Zidoo H6 Pro Android TV box over Bluetooth, connect two different Bluetooth headsets to the phone, and pair with, and retrieve data from a fitness tracker using Smart Movement app.

GPS

GPS is also an improvement over all the other Mediatek phones I’ve had. GPS fix is super fast like on Vernee Apollo Lite, but while using Nike+ Running, GPS accuracy is much better on the Xiaomi Mi A1 smartphone, as you can see from the two screenshots below.

Xiaomi Mi A1 (left) vs Vernee Apollo Lite (Right) – Click to Enlarge

I ran two laps with the Xiaomi phone, and they almost exactly overlap. The downside is that I have to run a little longer to achieve the same distance on the app 🙂

Gaming

I tried four games: Candy Crush Saga, Beach Buggy Bleach, Riptide GP2, and CSR Racing 2. All played very smoothly, to my surprise CSR 2 performed much better than on Vernee Apollo Lite, despite the latter having a more powerful ARM Mali-T880 GPU in Helio X20 SoC. As mentioned in a aforelinked post, I can see 3 potential reasons for the difference in that game: more optimization on Qualcomm SoCs than Mediatek SoCs, slightly lower level of details shown in the Qualcomm phone, better cooling for Xiaomi Mi A1 smartphone, which stays cool at all times, contrary to the Vernee phone which may require a cool pack to run smoothly…

IR Transmitter / Remote Control App

An infrared transmitter is built into Xiaomi Mi A1  smartphone, and can be control with Mi Remote app. I tried with LG 4K UHD TV, and it worked well.

Click to Enlarge

Air conditioners are always more challenging. So first I had to go through a process to detect which Haier aircon model I had, pressing poweroff button, and then other buttons, to find the right model among 158 options.

Click to Enlarge

It finally found mine, I gave it a name “Bedroom Haier AC” and realized on some functions would work, and some temperatures are not supported. So not so useful in that case.

Others

Multitouch app reports the touchscreen supports 10 touch points. The smartphone has a single speaker with mediocre quality when listening to music, but that’s not that big of an issue as Bluetooth speakers are now rather inexpensive, and in my daily life I mostly use wired or Bluetooth audio headsets. It’s good to have a 3.5mm audio jack, but I normally prefer when it’s placed on the top of the phone, rather than the bottom left, which can be an issue when using an armband, or while holding the phone.

Video Review

I’ve also shot a video review mostly summarizing the points above, showing the camera in action, playing Riptide GP2, a YouTube video up to 1080p, opening a large PDF files, etc…

Long Term Review / History

Since I’ve very satisfied with the phone, I’m going to retire Vernee Apollo Lite, and make Mi A1 my main phone. Since it’s also supposed to be upgraded for two years, I’ll keep this section to report the history of the phone, like a long term review, and report important events like firmware updates, or if something stops working. I got 3 firmware updates since I received the phone less than a month ago.

  • September 5, 2017 – Xiaomi Mi A1 announcement
  • September 12, 2017 – Official launch in India
  • October 16, 2017 –  Unboxing and September 2017 security update (1059.6 MB), Android 7.1.2 / Linux 3.18.31
  • October 21, 2017 – October 2017 security update (118 MB), Android 7.1.2 / Linux 3.18.31
  • November 3, 2017 – October 2017 security update (75.7 MB), Android 7.1.2 / Linux 3.18.31
  • November 15, 2017 – This review
  • November 22, 2017 – November 2017 security update (466.9 MB), Android 7.1.2 / Linux 3.18.31
  • December 12, 2017 – December 2017 security update (153.0 MB), Android 7.1.2 / Linux 3.18.31
  • December 27, 2017 – December 2017 security update (63.4MB), Android 7.1.2 / Linux 3.18.31
  • December 31, 2017 – December 2017 update (1107.4MB), Android 8.0.0 / Linux 3.18.66

Conclusion

I’m really pleased with my experience with Xiaomi Mi A1 smartphone, and to my surprise it’s an improvement over Vernee Apollo Lake with most features, except for fast charging that’s missing from the phone.

PROS

  • Stable and relatively recent Android 7.1.2 firmware
  • Part of Android One program with promise of regular security and firmware updates for 2 years (including Android 8.x and 9.x).
  • Good & sharp 1920 x 1080 display; wide brightness range
  • Excellent Wi-Fi 802.11ac performance
  • Excellent eMMC flash performance (Best I’ve tested so far)
  • Long battery life (about 30 hours per charge for 4 to 5 hours active use per day)
  • Good front-facing camera and rear dual cameras for depth effect
  • Overall better app performance compared to my previous Helio X20 based smartphone, especially for some games
  • Support forums

CONS

  • Quick Charge (Fast charging) not available
  • Videos shot with the rear camera are not smooth in dark scenes, and audio is poor in some videos (metal sound)
  • SAMBA WiFi performance is average for transfer from phone to server
  • Mi Remote  app (infrared remote) is not working well with my aircon (Haier)
  • Display is quite glossy / reflective
  • Built-in speaker not really good to listen to music
  • Android One support may add about $30 to $40 to the price of the phone
  • GPL source code not released yet, but an article suggests Mi A1 Linux kernel source code may be released within three months.

I’d like to thank GearBest for providing a review sample. Xiaomi Mi A1 (Black) can be purchased on their shop for $219.99 shipped with coupon A1HS. Other shopping options include GeekBuying, Banggood, eBay, and others online shops.

Some people noticed that Xiaomi Redmi Note 4 smartphone has very similar specifications with a Snapdragon 625 processor, 4GB RAM, and 64GB storage, the same 5.5″ Full HD display, but no dual rear camera, and a bigger battery (4,100 mAh). It’s sold for on Aliexpress for about $190 (Black version) and around $180 (Other colors), so if we assume the battery / camera features cancel out (in terms of price) that means Android One support adds about $30 to $40. One way to look at it is that you pay a little less than $2 per month for 2-year support with regular security & firmware updates.

NanoPi Fire2A & Fire3 Boards Released with Samsung/Nexell Quad & Octa Core Processors

November 12th, 2017 34 comments

FriendlyElec previously launched NanoPi 2 Fire board powered by Samsung (Nexell) S5P4418 quad core Cortex A9 SoC, mostly interesting because of its small form factor, camera and LCD interfaces.

The company has now launched two new boards based on S5Pxx18 processors, namely NanoPi Fire2A powered by S5P4418 SoC, and NanoPi Fire3 based on S5P6818 octa-core Cortex-A53 SoC. Both boards share the same form factor, which remains quite similar to the one of NanoPi 2 Fire, except the HDMI connector now makes place for a micro HDMI port, the USB 2.0 has moved into vertical position, and a few other tweaks have been made to positions of buttons and components.

NanoPi Fire2A / Fire3 specifications:

  • SoC
    • Fire2A – Samsung S5P4418 quad core Cortex A9 processor @ up to 1.4GHz, Mali-400MP GPU
    • Fire3 – Samsung S5P6818 octa core Cortex A53 processor @ up to 1.4 GHz, Mali-400MP GPU
  • System Memory
    • Fire2A – 512MB DDR3
    • Fire3 – 1GB DDR3
  • Storage – 1x Micro SD Slot
  • Connectivity – Gigabit Ethernet port
  • Video Output / Display I/F- 1x micro HDMI 1.4a port up to 1080p60, RGB LCD interface
  • Camera – 24-pin DVP interface; 0.5mm pitch
  • USB – 1x USB Host port; 1x micro USB 2.0 OTG port for power and data
  • Expansions Headers – 40-pin Raspberry Pi compatible header with UART, I2C, SPI, GPIOs…
  • Debugging – 4-pin header for serial console
  • Misc – Power and reset buttons, power and system LEDs, RTC battery header
  • Power Supply – 5V/2A via micro USB port; STM32F03 ARM Cortex M0 MCU for power handling (SW power off, sleep , and wakeup function)
  • Dimension: 75 x 40 mm

Other differences with the earlier model: AXP288 PMIC is gone, and replaced by an STM32 Cortex M0 MCU, and the company has now added mounting holes for a heatsink. The company provides FriendlyCore, and Debian firmware images for both hardware, and an extra Android image for Fire3 board. FriendlyCore is based on Ubuntu Core 16.04 with Linux 4.4, Qt 5.9 with OpenGL, and GStreams with VPU acceleration. The good news is the Linux kernel got an upgrade from Linux 3.4 to a more recent Linux 4.4 LTS kernel.

You’ll find download links and instructions to get starting in the Wiki pages here and there. NanoPi Fire2A is sold for $28 plus shipping, while NanoPi Fire3 goes for $35. You may also be interested in compatible accessories and external modules, including S430 4.3″ capacitive touch screen LCD display, X710 7.1″ capacitive touch screen LCD display, HD101 10.1″ touchscreen LCD display, CAM500B 5MP CMOS camera, Matrix GPS module, and others which you can find by browsing in the store.

NanoPi Fire2A/3 Connected to LCD430 Display (Left) and GPS Matrix Module (Right)