Adapting Neural Radiance Fields (NeRF) to the 3D Scene Reconstruction Problem Under Dynamic Illumination Conditions

The problem of new image synthesis with the use Neural Radiance Fields (NeRF) for an environment with dynamic illumination is considered. When training NeRF models, a photometric loss function is used, i.e., a pixel-by-pixel difference between intensity values of scene images and the images generated using NeRF. For reflective surfaces, image intensity depends on the viewing angle, and this effect is accounted for by using the direction of a ray as the NeRF model input parameter. For scenes with dynamic illumination, image intensity depends not only on the position and viewing direction, but also on time. It is shown that illumination change affects the learning of NeRF with a standard photometric loss function and decreases the quality of the obtained images and depth maps. To overcome this problem, we propose to introduce time as an additional NeRF input argument. Experiments performed on the ScanNet dataset demonstrate that NeRF with a modified input outperform the original model version and generate more consistent and coherent 3D structures. The results of this study can be used to improve the quality of training data augmentation for training distance forecasting models (e.g., depth-from-stereo models allowing for depth/distance forecasts based on stereo data) for scenes with non-static illumination.