Ship detection and classification with terrestrial hyperspectral data based on convolutional neural networks

The detection and classification of maritime objects in a harbour environment or coastal areas using a terrestrial hyperspectral system in combination with a high-resolution RGB sensor is a challenging task since the large number of spectral channels requires a robust analytical method. Recently, deep learning methods have shown a good performance in many computer vision applications. In this paper, we present a general analysis workflow for ship detection and classification based on fused terrestrial RGB and hyperspectral images, which employs a deep learning network for the localization of ships in the high-resolution images and a following convolutional neural network based multi-input model for the classification of each detected object. During a measurement campaign, images of various ship types were collected under distinct weather conditions for the training and evaluation of the network model. In the first part of the workflow, ship candidates were located using the Mask R-CNN framework based on the RGB images. For the following classification process, which was trained to separate different ship type classes, we developed a multi-input convolutional neural network using the RGB and the hyperspectral images as input data. For the pre-processing procedure of the hyperspectral data a principal component analysis was applied to reduce the number of input channels for the network while still maintaining a large fraction of the initial information. For the architecture of the RGB classification branch, the structure and the weights of a pre-trained model were integrated and fine-tuned. Since only limited training data was available, regularization methods and data augmentation were employed. The detection and the multi-input classification network was finally evaluated and showed that the classification performance can be increased when integrating additional information from a hyperspectral sensor.