Den-SOFT: Dense Space-Oriented Light Field Dataset for 6-DOF Immersive Experience

1Tsinghua University Beijing, China, 2Metaverse-AI-Lab

We have utilized a self-designed multi-camera device named "Compound Eye" to capture a dataset of images and videos from multiple scenes, with a particular focus on large-scale outdoor environments. The quality and viewpoint density of this dataset may represent the current state-of-the-art within the public domain as we validated the effectiveness on popular algorithms and integrated it with VR engine.

Please note: Flythrough videos are recorded from reconstructed scenes using vanilla 3D Gaussian Splatting. Encoded in 480p for fast display on web, you can refer to youtube for high quality display and VR showcase.

Abstract

We have built a custom mobile multi-camera large-space dense light field capture system, which provides a series of high-quality and sufficiently dense light field images for various scenarios. Our aim is to contribute to the development of popular 3D scene reconstruction algorithms such as IBRnet, NeRF, and 3D Gaussian splitting. More importantly, the collected dataset, which is much denser than existing datasets, may also inspire space-oriented light field reconstruction, which is potentially different from object-centric 3D reconstruction, for immersive VR/AR experiences. We utilized a total of 40 GoPro 10 cameras, capturing images of 5k resolution. The number of photos captured for each scene is no less than 1000, and the average density (view number within a unit sphere) is 134.68. It is also worth noting that our system is capable of efficiently capturing large outdoor scenes. Addressing the current lack of large-scale scene datasets, we made efforts to include elements such as sky, reflections, and dynamic objects that are of interest to researchers in the field of 3D reconstruction during the data capture process. Finally, we validated the effectiveness of our provided dataset on three popular algorithms and integrated it with the Unity engine, demonstrating the potential of utilizing high-quality captured scenes to enhance the realism of virtual reality (VR) and create feasible interactive spaces.

Capture Process

Training and VR demo

Side-by-Side comparison

Caputure density visualize

BibTeX

      @InProceedings{Den-SOFT,
        author    = {Xiaohang Yu and
                     Zhengxian Yang and
                     Shi Pan and
                     Yuqi Han and
                     Haoxiang Wang and
                     Jun Zhang and
                     Shi Yan and
                     Borong Lin and
                     Lei Yang and
                     Lu Fang and
                     Tao Yu },
        title     = {{Den-SOFT}: Dense Space-Oriented Light Field Dataset for 6-DOF Immersive Experience},
        year      = {2024},
        url       = {https://metaverse-ai-lab-thu.github.io/Den-SOFT/},
      }