3D Gaussian Splatting for Real-Time Radiance Field Rendering

Problem

Framing

Radiance fields reached high quality only with slow NeRF optimization and sub-real-time rendering. The paper replaces dense neural fields with optimized 3D Gaussians plus tile-based splatting, matching top view-synthesis quality while rendering at real-time rates. On Mip-NeRF360, it reports 1080p rendering above 30 fps and training in 24–51 minutes.

Currently Used Methods

Direct antecedents

• @mildenhallNeRF2020 — neural volume rendering with continuous radiance fields.
  • Limitation in context: slow rendering and optimization block interactive view synthesis.
• Mip-NeRF 360 — high-quality unbounded-scene NeRF with strong reconstruction fidelity.
  • Limitation in context: up to 48 hours training and sub-real-time rendering.
• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding — fast NeRF training with hash grids.
  • Limitation in context: speed gains come with lower visual quality on hard scenes.
• Plenoxels: Radiance Fields without Neural Networks — explicit sparse voxel radiance fields without MLPs.
  • Limitation in context: structured grids waste resolution and limit quality-speed tradeoffs.
• Point-Based Neural Rendering variants — point primitives with learned appearance and splatting.
  • Limitation in context: geometry control and visibility handling remain too weak for SOTA quality.

Proposed Method

Architecture

The scene is a set of 3D anisotropic Gaussians initialized from SfM points. Each Gaussian carries position, covariance, opacity, and spherical-harmonic color; rendering projects them to screen space and composites them with a differentiable tile rasterizer while adaptive density control splits, clones, and prunes primitives.

Loss / Objective

Training minimizes a photometric reconstruction loss between rendered and ground-truth views:

Sampling Rule / Algorithm

Rendering projects each Gaussian to screen space, evaluates alpha-weighted splats, and composites front-to-back:

Training Procedure

• 30K iterations for reported real-scene training.
• Intermediate quality already strong at 7K iterations.
• Initialize from sparse SfM points.
• Interleave optimization with density control.
• Optimize full anisotropic covariance.
• Use spherical harmonics for view-dependent color.

Evaluation

Datasets

• Mip-NeRF360
• Tanks and Temples
• Deep Blending
• Synthetic Blender scenes

Metrics

• SSIM
• PSNR
• LPIPS
• training time
• rendering FPS
• memory

Headline results

• Mip-NeRF360: SSIM 0.815, PSNR 27.21, LPIPS 0.214.
• Tanks and Temples: best reported quality with real-time rendering in many scenes.
• Deep Blending: competitive or better quality than prior radiance-field baselines.
• Synthetic scenes: 180–300 FPS after training.
• Real scenes: 1080p rendering at 30 fps or more.

Results table

Table 2: Aggregate metrics on Mip-NeRF360.

Ablations

• Full anisotropic covariance beats isotropic splats on reconstruction quality.
• Adaptive densification is critical; fixed primitives underfit geometry.
• Quality at 7K iterations is already close to the 30K endpoint.
• Missing viewing angles produce artifacts, especially near scene corners.

Method Strengths and Weaknesses

Strengths

• Reaches NeRF-class quality with explicit, non-MLP scene parameters.
• Delivers real-time 1080p rendering instead of offline novel-view synthesis.
• Trains in under an hour on real scenes, far below Mip-NeRF360.
• Anisotropic Gaussians fit geometry and appearance better than isotropic splats.

Weaknesses

• Artifacts appear in poorly observed regions and missing-angle captures.
• Memory use exceeds NeRF-based methods on large scenes.
• Performance depends on SfM initialization quality.
• Static-scene formulation does not address dynamic content.

Suggestions from the authors

• Reduce memory footprint for large-scene training.
• Improve robustness in weakly observed scene regions.
• Handle challenging capture gaps and corner views.
• Extend the representation beyond static scenes.

Links

Prior Papers

• @mildenhallNeRF2020 — introduces NeRF, the core radiance-field baseline this paper seeks to replace in practice.

Further Papers

No vault papers identified as further work yet.