Deep Learning-Based Thermal Image Reconstruction and Object Detection

Recently, thermal cameras are being widely used in various fields, such as intelligent surveillance, biometrics, and health monitoring. However, the high cost of the thermal cameras poses a challenge in terms of purchase. Additionally, thermal images have an issue pertaining to blurring caused by object movement, camera movement, and camera focus settings. There have been very few studies on image restoration centered around thermal images to address such problems. Moreover, it is important to increase the processing speed of image restoration methods to jointly conduct with methods such as action recognition and object tracking that use temporal information from thermal videos. However, no study has been conducted on simultaneously performing super-resolution reconstruction and deblurring using thermal images. Furthermore, existing studies on object detection using thermal images have errors owing to the incapability in distinguishing reflections on the surrounding ground or wall due to the heat radiated from the object. To address such issues, this study proposes a deep learning-based thermal image restoration method that simultaneously performs super-resolution reconstruction and deblurring. According to recent development of deep learning, generative adversarial network (GAN)-based methods which have ability to preserve texture details in images, and yield sharper and more plausible textures than classical feed forward encoders show success in image-to-image translation tasks. Considering the advantages of GAN, we propose a deblur-SRRGAN for thermal image reconstruction. In addition, we propose a light-weighted Mask R-CNN for object detection in the reconstructed thermal image. For the input, we employ an image processing method that converts 1-channel thermal images (often used in the existing studies) into 3-channel images. The results of the experiments conducted using self-collected databases and an open database demonstrate that our method outperforms the state-of-the-art methods.