Multimodal feature assessment using multibranch 3D CNN to BI-LSTM for feature level multi-polarization SAR image data fusion and vehicle identification

Deep machine learning computer vision algorithms have been widely explored for the purpose of multisensory data fusion. The ability to combine feature, pixel, and decision level information from multiple sensors in order to enhance accurate assessments and decisions made by the platforms has been a significant point of interest for the remote sensing community. In this paper, we propose a dual branch 3D convolutional neural network (CNN) to bi-long short-term memory network (BILSTM) algorithm that seeks to fuse sparse multiresolution, multi-pose and multimodal VV and HV polarizations of synthetic aperture radar (SAR) vehicle image information to enhance vehicle identification in unfamiliar and uncoherent environments. We cultivated and explored the proposed algorithm using the SDMS CV Data Domes repository of 14,430 augmented images per modality, equally represented over ten vehicle classes under similar and dissimilar vehicle pose augmentations with low to high levels of added testing set noise via zero-mean white Gaussian noise. Our results indicated that the local individual modality 3D convolution fusion of multiple poses and resolutions as well as dual-modality fusion of both polarizations enhanced the developed algorithm's ability to classify SAR vehicle image information in unfamiliar pose, elevation angle and moderate to low noise environments.