LRFNet: An Occlusion Robust Fusion Network for Semantic Segmentation with Light Field

Semantic segmentation, aiming to assign a categorical label to each pixel in an image, has drawn a lot of attention and has made significant achievements in recent years. However semantic segmentation remains a challenging problem caused by occlusion due to the lack of view information. In practice, the result of pixel assignment is greatly influenced by the scene information, especially in complex occlusion areas. One way to address the pixel assignment problem of occlusion areas is to train a feature representation that is extracted with enough scene information. To this end, we present light robust features(LRFs) through light field(LF), which contains all the light information of the scene. In addition, LRF consists of two components: 1)light color features(LCFs) and 2)light spatial features(LSFs). On the one hand, LCF is the expert in exploring the comprehensive RGB characteristic of LF images. When training LCF, we adopt a ResNet based feature extraction module. Moreover, a cross-entropy loss is deployed in LCF network. On the other hand, we design a LSF feature extraction module, which has a similar architecture to a robust depth estimation network. The difference between LCF and LSF is that LSF represents the depth characteristic of LF images. Finally, by combining LCF and LSF through pyramid-pooling and conditional random field(CRF) module, LRF can alleviate the influence of the occlusion, even in scenes of multi scales and complex categories. Experiments are conducted on the LF Semantic Segmentation data sets. We show that LRF produces competitive performance compared with state-of-the-art approaches.