Multi-Modal Deep Learning for Vehicle Sensor Data Abstraction and Attack Detection

Mulimodal learning is the process of reasoning with different types of data simultaneously using a neural network. This is no easy feat. Challenges to tackle include mapping features from one modality to another and finding a joint representation. As applied to connected vehicles, we propose a multimodal module that detects manipulations in the image sensor through verifying its consistency with the LiDAR sensor. A state-of-the-art multimodal network, Regnet, is adapted and trained for attack detection. Adaptation includes replacing the input layer and abstracting the multimodal input process. The KITTI dataset is extended to represent two attacks on connected vehicles: inpainting and translation. The end product is a smart multimodal module that abstracts connected-vehicle sensor input and guards against data integrity breaches.