Naturalistic Driving Scenario Recognition with Multimodal Data

Driving Scenario recognition is one fundamental technology of automated driving systems or advanced driver assistance systems. A common practice of driving scenario recognition is to conduct classification tasks with the data collected by in-vehicle data acquisition system or driving simulator. In most existing works, visual data were used since the relevant information of driving scenarios is usually inferable from their visual appearance. However, the non-visual information, e.g. physiological state of the driver, also provide complementary information for the scenario recognition task especially when the visual appearance is insufficient to differentiate similar naturalistic scenarios in some cases. In this paper, we propose a hybrid driving scenario recognition model with multimodal input. The model consists of a convolutional neural network based visual data sub-model, a stacked autoencoder based physiological data sub-model, and a fusion sub-model that combines the extracted features from both visual and physiological sub-models. Besides, a post-processing is adopted to correct the recognition results of some ambiguous scenarios. Experimental results on the trip data collected in the naturalistic driving context demonstrated the effectiveness of the proposed method.