Multimodal information fusion based on LSTM for 3D model retrieval

With advances in low-cost 3D model capturing devices and virtual 3D model building software, the acquisition of 3D data has become increasingly easier. The subsequent 3D model retrieval skill has also become essential when we utilize 3D models. Although a large number of methods have been proposed to address this problem, most of them cannot fully utilize the information represented by 3D models. To solve this problem. We present a multimodal feature fusion method based on the LSTM network. First, we placed some cameras evenly around the 3D model at a fixed distance, which was aimed at the centroid of the 3D model to obtain a series of pictures. Second, the skeleton information is extracted from these rendered pictures. Finally, the rendered pictures, along with the skeleton information, were sequentially fed into the LSTM network to obtain the feature of the fusion information. The confusion matrix was completed to evaluate retrieval performance. In the experiment section, datasets named NTU and ModelNet40 were utilized to demonstrate the performance of the proposed method. Many experiments and corresponding experimental results also demonstrated the superiority of our approach.