AR-based deep learning for real-time inspection of cable brackets in aircraft

In the process of aircraft assembly, there exist numerous and ubiquitous cable brackets that shall be installed on frames and subsequently need to be manually verified with CAD models. Such a task is usually performed by special operators, hence is time-consuming, labor-intensive, and error-prone. In order to save the inspection time and increase the reliability of results, many researchers attempt to develop intelligent inspection systems using robotic, AR, or AI technologies. However, there is no comprehensive method to achieve enough portability, intelligence, efficiency, and accuracy while providing intuitive task assistance for inspectors in real time. In this paper, a combined AR+AI system is introduced to assist brackets inspection in a more intelligent yet efficient manner. Especially, AR-based Mask R-CNN is proposed by skillfully integrating markerless AR into deep learning-based instance segmentation to generate more accurate and fewer region proposals, and thus alleviates the computation load of the deep learning program. Based on this, brackets segmentation can be performed robustly and efficiently on mobile devices such as smartphones or tablets. By using the proposed system, CAD model checking can be automatically performed between the segmented physical brackets and the corresponding virtual brackets rendered by AR in real time. Furthermore, the inspection results can be directly projected on the corresponding physical brackets for the convenience of maintenance. To verify the feasibility of the proposed method, experiments are carried out on a full-scale mock-up of C919 aircraft main landing gear cabin. The experimental results indicate that the inspection accuracy is up to 97.1%. Finally, the system has been deployed in the real C919 aircraft final-assembly workshop. The preliminary evaluation reveals that the proposed real-time AR-assisted intelligent inspection approach is effective and promising for large-scale industrial applications.