Semantic Segmentation of Marine Species in an Unconstrained Underwater Environment

A non-invasive Underwater Fish Observatory (UFO) was developed and deployed on the seafloor to perform continuous recording of stereo video and sonar data as well as various oceanic parameters at a high temporal sampling rate. The acquired image data is processed to automatically detect, classify and measure the size of passing aquatic organisms. An important subtask in this processing chain is the semantic segmentation of the previously detected animals. Within this publication, a former segmentation system, that only considered a binary classification of fish and background, is extended to a multi-class segmentation system by including an additional species. Since the images usually contain a lot of background, the semantic segmentation is a problem with a high class imbalance, which demands special care in the choice of loss functions and evaluation metrics. Therefore, three different loss functions, namely Dice loss, Focal loss and Lovasz loss, which are well suited for class-imbalance problems, are investigated and their effect on the final mean intersection-over-union (IoU) on a separate test set is explored. For the given dataset, the model trained with a Focal loss performed best achieving an average, class specific IoU of 0.982 for the background class, 0.828 for the Aurelia aurita and 0.678 for the Gadus morhua.