Limits on transfer learning from photographic image data to X-ray threat detection

BACKGROUND: X-ray imaging is a crucial and ubiquitous tool for detecting threats to transport security, but interpretation of the images presents a logistical bottleneck. Recent advances in Deep Learning image classification offer hope of improving throughput through automation. However, Deep Learning methods require large quantities of labelled training data. While photographic data is cheap and plentiful, comparable training sets are seldom available for the X-ray domain. OBJECTIVE: To determine whether and to what extent it is feasible to exploit the availability of photo data to supplement the training of X-ray threat detectors. METHODS: A new dataset was collected, consisting of 1901 matched pairs of photo & X-ray images of 501 common objects. Of these, 258 pairs were of 69 objects considered threats in the context of aviation. This data was used to test a variety of transfer learning approaches. A simple model of threat cue availability was developed to understand the limits of this transferability. RESULTS: Appearance features learned from photos provide a useful basis for training classifiers. Some transfer from the photo to the X-ray domain is possible as similar to 40% of danger cues are shared between the modalities, but the effectiveness of this transfer is limited since similar to 60% of cues are not. CONCLUSIONS: Transfer learning is beneficial when X-ray data is very scarce-of the order of tens of training images in our experiments-but provides no significant benefit when hundreds or thousands of X-ray images are available.