A Low-Power Neuromorphic System for Real-Time Visual Activity Recognition

We describe a high-accuracy, real-time, neuromorphic method and system for activity recognition in streaming or recorded videos from static and moving platforms that can detect even small objects and activities with high-accuracy. Our system modifies and integrates multiple independent algorithms into an end-to-end system consisting of five primary modules: object detection, object tracking, convolutional neural network image feature extractor, recurrent neural network sequence feature extractor, and an activity classifier. We also integrate neuromorphic principles of foveated detection similar to how the retina works in the human visual system and the use of contextual knowledge about activities to filter the activity recognition results. We mapped the complete activity recognition pipeline to the COTS NVIDIA Tegra TX2 development kit and demonstrate real-time activity recognition from streaming drone videos at less than 10 W power consumption.