Deep Learning-based 3D Image Generation Using a Single 2D Projection Image

Due to the inter-fraction and intra-fraction variation of respiratory motion, it is highly desired to provide real-time volumetric images during the treatment delivery of lung stereotactic body radiation therapy (SBRT) for accurate and active motion management. Motivated by this need, in this study we propose a novel generative adversarial network integrated with perceptual supervision to derive instantaneous 3D image from a single 2D kV projection. Our proposed network, named TransNet, consists of three modules, i.e., encoding, transformation, and decoding modules. Rather than only using image distance loss between the generated 3D image and the ground truth 3D CT image to supervise the network, perceptual loss in feature space is integrated into loss function to force the TransNet to yield accurate lung boundary. Adversarial loss is also used to improve the realism of the generated 3D image. We conducted a simulation study on 20 patient cases, who had undergone 4D-CT scan and received lung SBRT treatments in our institution, and evaluated the efficacy and consistency of our method at four different projection angles, i.e., 0 degrees, 30 degrees, 60 degrees and 90 degrees. For each 3D CT image of a breathing phase in the 4D CT image set, we simulated its 2D projections at these two angles. Then for each projection angle, a patient's 3D CT images of 9 phases and the corresponding 2D projection data were used for training, with the remaining phase used for testing. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index metric (S SIM) achieved by our method are 99.5 +/- 13.7HU, 23.4 +/- 2.3dB and 0.949 +/- 0.010, respectively. These results demonstrate the feasibility and efficacy of our method on generating a 3D image from single 2D projection, which provides a potential solution for in-treatment real-time on-board volumetric imaging to guide treatment delivery and ensure the effectiveness of lung SBRT treatment.