Gaussian Splatting
How it works
Full 3DGS pipeline: (1) Initialisation - initial 3D Gaussians are created from an SfM (Structure from Motion, e.g. COLMAP) point cloud on the input images - each SfM point becomes a small isotropic Gaussian. (2) Optimisation - iterative gradient descent: at each iteration a training image is sampled, a view is rendered via differentiable rasterisation (projecting Gaussians to 2D, back-to-front sorting, alpha blending), L1 + SSIM + LPIPS loss versus ground truth is computed, gradients flow through the rasteriser back to Gaussian parameters (position, covariance, SH, opacity). (3) Densification - every N iterations: Gaussians with large position gradient are cloned (split if large) or duplicated (clone if small) to increase density in under-represented regions. (4) Pruning - Gaussians with opacity <threshold or too large size are removed, keeping their count under control (final models typically have 1-5M Gaussians for a scene). (5) Inference rendering - for a new view: Gaussians are projected, per-tile GPU-sorted, alpha-blended in screen space - fully rasterisation-based, no neural network in the critical path. Key to quality: using Spherical Harmonics (typically up to degree 3 = 16 coefficients per RGB colour = 48 per Gaussian) gives view-dependent colour - reproducing reflections and BRDF-like effects.
Problem solved
NeRF (Mildenhall et al. 2020) set a new standard of photorealism in novel view synthesis, but had a critical flaw: slow rendering. Classic NeRF required many hundreds of neural network queries per pixel - rendering a single 1080p frame took seconds to minutes on a consumer GPU. Optimisations (Instant NGP, Plenoxels, MobileNeRF) shortened that time but still did not achieve real-time 30+ FPS while maintaining SOTA quality. Classic textured 3D meshes deliver real-time, but reproducing complex phenomena (hair, fog, reflections, transparency) was hard or impossible. Gaussian Splatting solves this quality-speed-expressiveness trilemma: an explicit representation (like a mesh) gives real-time rendering, but continuous Gaussians with SH colour reproduce phenomena hard for meshes, while achieving PSNR equal to or better than the best NeRFs.
Components
Basic representation element: an ellipsoidal 'blob' in 3D space. Parametrisation: μ ∈ ℝ³ (position), Σ ∈ ℝ³ˣ³ (covariance decomposed into rotation as a quaternion + scale), c(d) as Spherical Harmonics (view-direction dependent colour), α ∈ [0,1] (opacity). A typical scene contains 1-5M such primitives.
Official
CUDA kernel implementing tile-based rasterisation with back-to-front alpha blending. Critical part - projecting 3D Gaussians to screen space (2D), per-tile sorting, colour accumulation. Supports backward gradient propagation to Gaussian parameters.
Function basis for representing view-direction dependent colour. In 3DGS usually up to degree 3 (16 coefficients per channel, 48 per RGB Gaussian). Enables reflections, BRDF-like effects, and view-dependent shading.
Official
Heuristic procedure run periodically during training: analyses position gradient for each Gaussian and clones/splits when too big, prunes when opacity <ε. Keeps Gaussian count under control (1-5M) and ensures adaptive density.
Official
Structure from Motion pipeline (typically COLMAP) generating from a set of images: camera poses + sparse 3D point cloud. That cloud initialises the Gaussians (each SfM point = one small Gaussian). Critical for convergence - random init does not work.
Official
Implementation
3DGS heavily depends on the COLMAP point cloud quality. Scenes with low texture (white walls), too few images, or bad angles yield poor SfM points and prevent good reconstruction.
Without densification limits, scenes can grow to 50M+ Gaussians (tens of GB VRAM), slowing training and blocking deployment.
Original 3DGS shows visible artefacts (dilation, flickering) when the view significantly deviates from training cameras - Gaussians rendered at a different resolution than trained.
Extracting a triangle mesh from 3DGS is hard - Gaussians are volumes, not surfaces. Naive marching cubes on densities gives a lower-quality mesh than from NeRF.
Classic 3DGS is per-scene optimisation - each new scene needs a separate 30-40 min training. No feed-forward inference from a single image.
Evolution
ECCV 2020 - implicit scene representation as an MLP, photorealistic novel view synthesis but slow rendering. Predecessor to Gaussian Splatting.
SIGGRAPH 2022 Best Paper - NeRF training in seconds instead of hours. Rendering still slow.
SIGGRAPH 2023 Best Paper (arXiv 2308.04079) - real-time 30+ FPS on consumer GPU at SOTA PSNR quality. Explicit representation instead of a neural field.
ICLR 2024 - 2-minute 3D asset generation from text by applying Score Distillation Sampling to 3DGS.
3DV 2024 - extension of 3DGS to time-varying scenes (dynamic objects, motion).
SIGGRAPH 2024 - 2D disks instead of 3D ellipsoids for better surface and normal reconstruction.
CVPR 2024 - solves zoom-in/zoom-out artefacts in the original 3DGS via Gaussian pre-filtering.
CVPR/ECCV 2024 - training amortisation via a feed-forward transformer, Gaussian generation in <1s instead of 30 min.
Gaussian SLAM (Yugay et al.), Photo-SLAM - using 3DGS as a map for robot navigation and autonomous vehicles (Wayve, Waymo, Tesla).
Scene size reduction from ~700 MB to 10-50 MB via discretisation, hierarchies, and learnable neural offsets - critical for web/mobile deployment.
Hyperparameters (configurable axes)
Key scene hyperparameter. Controlled by densification/pruning thresholds.
Controls richness of view-dependent colour. Higher degree = more reflections, but also more memory per Gaussian.
Impacts final PSNR quality and training duration.
Controls how aggressively Gaussians are cloned/split. Lower threshold = more Gaussians, denser scene.
Computational complexity
Time complexity: O(N · P + T · N_pixels). Space complexity: O(#Gaussians · 59).
Compute bottleneck
Main bottleneck: back-to-front per-tile Gaussian sorting (parallel radix sort on GPU) and alpha compositing. Grows linearly with the number of Gaussians projected onto a pixel. Anti-aliasing (Mip-Splatting) adds cost.
Execution paradigm
During pixel-level rendering: only Gaussians visible in a given tile are selected (frustum culling + tile assignment) - sparse in the system sense (not every Gaussian on every pixel), but dense in the weight sense (no parameters are skipped).
No expert routing. Each Gaussian is trained independently, all visible ones are active in every render pass.
Parallelism
Training and inference are massively parallel on GPU. Tile-based rendering exploits full SM pipeline width.
Hardware requirements
3DGS is designed for CUDA. Key ops (tile-based rasterisation, radix sort, alpha blending, backprop) are optimised for modern GPUs. RTX 3090+ / A100+ / H100 deliver real-time 30-100+ FPS at 1080p.
AMD RDNA3+ and Intel Arc can run 3DGS via ports (rocm, vulkan) - performance ~30-50% lower than CUDA. Apple Metal - available via ports (Brush with WebGPU).
CPU 3DGS rendering is 100-1000x slower than GPU - unusable for real-time, only headless offline rendering.