Computer Vision

3D Reconstruction

ActivePublished: 9 November 2026Updated: 9 November 2026Published

Key innovation

Turning sets of 2D images or depth measurements into a coherent 3D representation of a scene's geometry and appearance without manual modeling.

How it works

A typical passive pipeline begins with detecting and matching features across images; Structure-from-Motion then estimates camera poses and a sparse point cloud (usually refined by bundle adjustment). Multi-View Stereo densifies geometry into a dense point cloud or depth maps, which are fused into a mesh or other representation. Neural methods (NeRF) learn a continuous volumetric function mapping coordinates and view direction to color and density, rendered via volume rendering; 3D Gaussian Splatting represents a scene as a set of anisotropic Gaussians rasterized in real time. Active methods (structured light, LiDAR, ToF) measure depth directly, skipping feature matching.

Problem solved

Manual 3D modeling is labor-intensive and does not scale to real, complex scenes. 3D reconstruction automates the creation of faithful geometry and appearance models directly from images or measurements, enabling large-scale digital capture of the real world.

Components

Structure-from-Motion (SfM)Camera pose and sparse geometry estimation

Recovers camera poses and a sparse 3D point cloud from matched features across multiple images, usually with global bundle adjustment.

Multi-View Stereo (MVS)Dense geometry reconstruction

Given known camera poses, computes dense geometry (depth maps or a dense point cloud) that is fused into a surface.

Depth estimation / stereo matchingPer-pixel depth recovery

Estimates depth for pixels from stereo/multi-view imagery or directly from a single image (monocular depth estimation).

3D scene representationGeometry and appearance encoding

Mesh, point cloud, voxels, SDF, or neural/rasterization representations such as NeRF and 3D Gaussian Splatting.

Official

Implementation

Reference implementations

COLMAP

C++/CUDA/Python · Johannes Schönberger et al.

Official

3D Gaussian Splatting (official)

Python/CUDA · Inria GRAPHDECO

Official

Implementation pitfalls

Textureless and reflective surfacesHigh

Lack of distinctive features breaks SfM/MVS matching, causing holes and artifacts.

Fix:Add more views, active sensors (LiDAR/structured light), or neural priors.

Camera calibration and pose errorsHigh

Inaccurate camera parameters propagate through the whole reconstruction.

Fix:Careful calibration and bundle adjustment.

Scale ambiguity (monocular)Medium

Single-camera reconstruction is defined only up to an unknown scale.

Fix:Use a known reference object, stereo, or a depth sensor.

Overfitting to training views (NeRF/3DGS)Medium

Neural/Gaussian representations may extrapolate poorly beyond observed views.

Fix:Regularization and more diverse views.

Evolution

2000

Multiple-view geometry

Structure-from-Motion and multi-view geometry formalized (Hartley and Zisserman, Multiple View Geometry in Computer Vision).

2016

COLMAP

General-purpose SfM and MVS pipeline (Schönberger and Frahm) that became a de facto standard for image-based reconstruction.

Structure-from-Motion Revisited (paper)

2020

NeRF (Neural Radiance Fields)

Inflection point

Neural Radiance Fields enabled photorealistic novel-view synthesis from a continuous volumetric representation.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (paper)

2023

3D Gaussian Splatting

Inflection point

Rasterizing anisotropic Gaussians delivered radiance-field quality with real-time rendering.

3D Gaussian Splatting for Real-Time Radiance Field Rendering (paper)

Hyperparameters (configurable axes)

Scene representationCritical

Mesh / point cloud / voxels / SDF / NeRF / 3D Gaussian Splatting.

Input modalityHigh

Images, video, stereo, structured light, LiDAR/ToF.

Camera calibrationHigh

Whether camera poses/intrinsics are known or estimated.

Number of viewsHigh

From a single image to hundreds/thousands of views.

Output densityMedium

Sparse vs dense reconstruction.

Parallelism

Parallelism level

Partially parallel

Per-pixel/per-point stages and rasterization are highly parallel (GPU), while Structure-from-Motion and bundle adjustment are iterative and partly sequential.

Scope

Inference