Enhancing Gaussian Splatting via Julia

The release of GaussianSplatting.jl 2.0 introduces significant upgrades in both functionality and user experience. A prime example of its power is the reconstruction of the Amulet of St. Michael in Kyiv, Ukraine, which utilizes approximately $6 \times 10^6$ Gaussians.

"The 2.0 update focuses on quality-of-life improvements, broader hardware support, and advanced geometric constraints."

---

🚀 Cross-Platform GPU Support

By leveraging KernelAbstractions.jl, the library allows developers to write a single kernel that compiles for various targets. This means the same code works across different hardware ecosystems.

---

🛠️ Architecture: Multithreaded UI

To eliminate the issue of app freezing during intensive tasks (like JIT compilation or data ingestion), the application now operates on a dual-thread system:

• Frontend: Manages the user interface and OpenGL rendering.
• Backend: Handles the "heavy lifting," including training and dataset loading.

This separation ensures the UI remains fluid. Users now see animated spinners and progress bars with helpful tips instead of a locked screen during the first training iteration. Additionally, the UI now features live loss plots and real-time hyperparameter tracking.

---

🎲 MCMC Densification Strategy

While the standard approach uses cloning and splitting, GaussianSplatting.jl now incorporates a Markov Chain Monte Carlo (MCMC) strategy. This method provides tighter control over the total Gaussian count and is less sensitive to the initial state.

The mathematical objective can be viewed as sampling from a distribution $P(\text{geometry} | \text{images})$. You can implement this in your code as follows:

Trainer(
    rasterizer, 
    gaussians, 
    dataset, 
    opt_params; 
    strategy = MCMCStrategy(; kwargs...)
)

---

📐 Depth & Geometry Supervision

To minimize "floaters" and refine the 3D structure, the tool now supports depth priors (e.g., from Depth-Anything 3).

Implementation Details

1. Storage: Place depth maps in dataset-root/depths/FILENAME.png.
2. Activation: Enable via the UI or by setting OptimizationParams(; use_depth_loss=true).
3. Refitting: Because depth scales vary, maps are refitted to the prior point cloud; those that fail are discarded.

Geometry Regularization

To ensure surface smoothness, users can enable use_normal_loss=true. This performs two critical functions:

• Normal Consistency: Aligns rendered depth map normals with per-Gaussian normals.
• Surface Flattening: Constrains Gaussians along their shortest axis to create thinner surfaces.

---

🌌 The Sky Dome

To prevent the background from bleeding into the foreground (disentanglement), a Sky Dome—a frozen shell of $\sim 32\text{K}$ Gaussians—is rendered in a separate pass.

Sphere vs. Hemisphere:

• Full Sphere: May pull ground geometry upward, reducing opacity.
• Hemisphere: Keeps the bottom half black, ensuring the ground remains unaffected.

For even finer detail (like tree leaves), users can provide sky segmentation masks located at dataset-root/sky/FILENAME.png.

---

📷 Camera & Miscellaneous Updates

The Camera Frustum visualization now includes a miniature preview of the actual image captured by the camera.

Other Enhancements

• Automatic Checkpointing: Saves state every $N$ steps.
• Configuration Files: Use a hyperparameters.toml for custom learning rates and loss weights to ensure reproducibility.
• Capture Mode: Camera paths can now be saved and reloaded exactly across different sessions.

Example .toml snippet:

learning_rate = 0.001
use_depth_loss = true
use_normal_loss = true