Multi-View Stereo and Depth Priors Guided NeRF for View Synthesis

In this paper, we present a new framework for view synthesis of novel view based on Neural Radiance Fields(NeRF). We aim to address two main limitations of NeRF. Firstly, we propose to combine multi-view stereo into NeRF to help construct general neural radiance fields across different scenes. Specifically, We build a MVS-Encoding Feature Volume with average group-wise correlation to aggregate the multi-view appearance and geometry feature for every source view. And then we use an MLP to encode neural radiance fields by using the scene-dependent features interpolated from the MVS-Encoding Feature Volumes. This makes our model can be applied to other unseen scenes without any per-scene fine-tuning, and render realistic images with few images. If more training images are provided, our method can be fine-tuned quickly to render more realistic images. In fine-tuning phase, we propose a depth priors guided sampling method, which can make the model represent more accurate geometry for corresponding scenes and so render high-quality images of novel view. We evaluate our method on three common datasets. The experiment results show that our method performs better than other baselines, neither without or with fine-tuning. And the depth priors guided sampling method can be easily applied on other methods based on Neural Radiance Fields to further improve the quality of rendered images.