Nvidia develops fastest NeRf technology: compositing 2D photos into 3D scenes in seconds

Capturing a 3D world in lifelike 2D images was groundbreaking when Polaroids took their first instant photos 75 years ago, today AI researchers are doing the opposite, converting a set of still pictures into 3D in seconds digital scene.

At a conference at Nvidia's GTC last month, Nvidia demonstrated the latest in artificial intelligence and paid homage to early Polaroid images, showing a video of a man dressed like Andy Warhol holding an old Polaroid camera and quickly converting dozens of 2D photos. Render the scene in 3D.

The process, known as inverse rendering, uses artificial intelligence to simulate how light behaves in the real world, allowing researchers to reconstruct three-dimensional scenes from a small number of 2D images taken at different angles.

Nvidia applied this approach to a popular new technology called Neural Radiation Fields, or NeRF. The tool, called Instant NeRF, was jointly developed by UC Berkeley, UC San Diego, and Google Research in 2020 to generate data by mapping the color and light intensity of different 2D shots, combined with camera position data, and then These images from different locations are concatenated to render a complete 3D scene.

Although Instant NeRF also requires camera angle data for the photos taken, the model can be trained on dozens of still photos in seconds and then render the final 3D scene in tens of milliseconds, the fastest to date. NeRF technology.

"If traditional 3D representations like polygon meshes are similar to vector images, NeRFs are like bitmap images: they densely capture how light radiates in an object or scene," said David Luebke, vice president of graphics research at NVIDIA. That said, Instant NeRF may be as important to 3D as digital cameras and JPEG compression are to 2D photography—greatly increasing the speed, ease of use, and coverage of 3D capture and sharing.”

In the future, Nvidia hopes that Instant NeRF can create scenes for virtual worlds, capture video conference participants and their environments in 3D, reconstruct scenes for 3D digital maps, and be used in areas such as training robots and self-driving cars.

For more details see:

https://blogs.nvidia.com/blog/2022/03/25/instant-nerf-research-3d-ai/

https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf