Training an FCN with Synthetic Images for Component Segmentation with Applications in Orientation Estimation and Image Inpainting

The detection and segmentation of real objects along with their components is still an advanced topic. More detailed segmentation is needed to solve tasks like position and orientation estimation or for doing inpainting of components. In this paper, we specifically focus on cars and present a segmentation of components, such as rims or lights, which requires detailed and accurate training data. A decomposed 3D model is used to render highly detailed images of the car that fit to corresponding ground truth images. The main challenge is to create high quality synthetic datasets that allow reliable and accurate segmentation of real-world footage. Different camera shots, filters, environment maps, shapes, and neural networks are used and their benefits as well as problems are discussed within this paper. The accuracy and reliability of the segmentation depends on the quality and variability of the rendered images. We started with simple 3D models and a real-time renderer and reached accurate and reliable segmentation with almost photorealistic images that are created with a global illumination renderer. These results are then used for replacing components of the car as well as for deriving the position of special points of interest, like the center of a wheel, which is also necessary for subsequent processing such as correctly aligning the 3D model with the real camera stream for Augmented Reality applications. Here, the quality of replacing a component of an object with its rendered 3D counterpart depends on the accuracy of segmentation. Therefore, segmented components are used to determine the position and orientation of the car along with the size of the inpainting area. Then, a matching rendered image of the component is inpainted only into the segmented area. In this regard, we also compare two different approaches for deriving the center points of the components of an object.