NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction

CVPR 2022 (Oral)

Large-Scale scene reconstruction and neural rendering from direct inference.

1UC San Diego.    2Adobe Research.   

Abstract

While NeRF has shown great success for neural reconstruction and rendering, its limited MLP capacity and long per-scene optimization times make it challenging to model large-scale indoor scenes. In contrast, classical 3D reconstruction methods can handle large-scale scenes but do not produce realistic renderings. We propose NeRFusion, a method that combines the advantages of NeRF and TSDF-based fusion techniques to achieve efficient large-scale reconstruction and photo-realistic rendering. We process the input image sequence to predict per-frame local radiance fields via direct network inference. These are then fused using a novel recurrent neural network that incrementally reconstructs a global, sparse scene representation in real-time at 22 fps. This volume can be further fine-tuned to boost rendering quality. We demonstrate that NeRFusion achieves state-of-the-art quality on both large-scale indoor and small-scale object scenes, with substantially faster reconstruction speed than NeRF and other recent methods.

overview_image

Results

ScanNet

Bird View Rendering

Extracted Mesh from Fine-Tuned Model

NeRF Synthetic

More results coming

BibTeX


      @article{zhang2022nerfusion,
        author    = {Zhang, Xiaoshuai and Bi, Sai and Sunkavalli, Kalyan and Su, Hao and Xu, Zexiang},
        title     = {NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction},
        journal   = {CVPR},
        year      = {2022},
      }