3D Reconstruction using SfM and NeRF

The goal of this project was to perform 3D reconstruction on ‘in the wild’ images. 3D reconstruction is a technique where we triangulate the location of points using two different images. Since 3D reconstruction is usually performed in controlled conditions, where we know the focal length of the camera’s lens, the resolution of the image, and can also control the lighting conditions the images were taken in, this proved to be a challenging task.

We scraped the web to obtain images of objects in at least two different views, and utilized these images along with a dataset to test our methods. The first method we implemented was Structure from Motion, which reconstructs the given object using the motion of the camera. An optimization based triangulation method was used, and this was relatively simple, since the camera’s parameters remained constant between two shots. The 3D location of the points was optimized with respect to reprojection error in each image. Reprojection error is the error between the position of the original point and its reprojected location.

The second method we implemented was NeRF, or Neural Radiance Field based reconstruction. This method involves simulating a ray of light using neural networks, and tracing it back to its source. The usage of neural networks makes this method highly accurate and fast due to parallel processing. Tracing the ray of light back to the source allows us to reconstruct the object in directions that we may not have images of. NeRF based methods also have an advantage of being highly realistic, when rendered.

My main contributions to this project were scraping the web to obtain the dataset, and helping formulate the optimization strategy for the Structure from Motion methodology.

1 / 7