Deep Learning Models for Content-Based Retrieval of Construction Visual Data

Deep learning (DL) algorithms such as convolutional neural networks (CNNs) can assist in tasks such as content search and retrieval, image tagging and captioning, scene description, motion prediction, and language processing. This paper presents research that aims at designing and validating DL models for automated content-based retrieval of daily construction images and videos. Information retrieval from visual data is key to labor-intensive tasks such as safety inspection, crew activity logging, and work progress documentation. In order to train deep neural networks (DNNs), large repositories of high-quality annotated visual data are needed. However, generating such labeled datasets in construction is non-trivial and resource intensive, and requires specific skillset. To overcome this challenge, we present a methodology for fast object detection and tagging in visual data using DNNs trained with a relatively small dataset. Two state-of-the-art object detection algorithms, i.e., you-only-look-once (YOLO) and mask region-based CNN (a.k.a., Mask R-CNN) are investigated. Training data is obtained via web mining (the Internet) and crowdsourcing. Results show that training on data from both sources yields the best classification accuracy. Testing the model on new data reveals that the fully-tuned model can achieve a minimum mean average precision (mAP) of 79% when tested on different image subsets.