Low Latency Visual Inertial Odometry With On-Sensor Accelerated Optical Flow for Resource-Constrained UAVs

Visual inertial odometry (VIO) is the task of estimating the movement trajectory of an agent from an onboard camera stream fused with additional inertial measurement unit (IMU) measurements. A crucial subtask within VIO is the tracking of features, which can be achieved through optical flow (OF). As the calculation of OF is a resource-demanding task in terms of computational load and memory footprint, which needs to be executed at low latency, especially in robotic applications, OF estimation is today performed on powerful CPUs or GPUs. This restricts its use in a broad spectrum of applications where the deployment of such powerful, power-hungry processors is unfeasible due to constraints related to cost, size, and power consumption. On-sensor hardware acceleration is a promising approach to enable low latency VIO even on resource-constrained devices such as nano drones. This article assesses the speed-up in a VIO sensor system exploiting a compact OF sensor consisting of a global shutter camera and an application-specific integrated circuit (ASIC). By replacing the feature tracking logic of the VINS-Mono pipeline with data from this OF camera, we demonstrate a 49.4% reduction in latency and a 53.7% reduction of compute load of the VIO pipeline over the original VINS-Mono implementation, allowing VINS-Mono operation up to 50 FPS instead of 20 FPS on the quad-core ARM Cortex-A72 processor of a Raspberry Pi Compute Module 4.