Automated Scene Understanding via Fusion of Image and Object Features

Scene understanding is an important component of intelligence, surveillance, and reconnaissance systems as well as autonomous vehicles. Scenes are composed of objects and their surrounding environment, both of which should be useful for a vision system to recognize the type of scene. A fusion system architecture that combines features from a whole image and detected objects, analogous to simultaneous top-down and bottom-up processing, is presented. The bottom-up pipeline uses a deep convolutional neural network to extract visual features from the whole image. The detected objects from the top-down pipeline are converted into a bag-of-words feature space that is combined with the visual feature space. The two streams can be fused either at the feature level or at the class probability level; these two methods are compared. A support vector machine classifier is trained with supervised learning on the combined feature space and used to produce a scene type label. We evaluate the system on an aerial imagery dataset that contains a variety of outdoor scene and object combinations. Applications that require both scene understanding and object detection can benefit from this fused architecture, especially when generalizing trained networks to domains with less available training data.