Artistic Rendering via Gaussian Splatting

August 3, 2026

Last year, I developed a tool for edge-aware pixelation. Instead of applying a rigid, static grid to an image, I created a system that deformed the grid to align with the image's edges, allowing for pixel art that preserved crisp details and structural integrity.

I eventually realized that this same edge-detection logic could be pivoted toward the realm of digital painting.

The Logic of the Brush

When painting, an artist must decide three primary things:

1. Placement: Where does the stroke go?
2. Scale: How large should the mark be?
3. Flow: In which direction should the brush move?

Most of this information is already embedded in the image's edges. Edges define the boundaries of objects—the very contours a painter would trace. Conversely, a lack of edges suggests a flat region where a few wide, sweeping strokes are sufficient.

I wanted to just use a filter $\rightarrow$ My actual goal was to build an interactive application where users could manipulate sliders to watch a painting reform in real-time. This project also served as a "test drive" for Jolt, allowing me to see if it could handle a non-trivial build.

The Big Question: Would the final output actually look like a painting, or just a blurry mess?

---

Defining the "Computational Stroke"

What is a brush stroke, mathematically? In oil or acrylic painting, a stroke is an elongated mark with a concentrated center of color that tapers off toward the edges. Because paint is translucent, these marks overlap, allowing artists to layer broad base colors and then refine them with smaller, more transparent details.

I discovered that a 2D Gaussian splat is a near-perfect mathematical proxy for this. A splat consists of:

• Mean ($\mu$): The center point of the stroke.
• Covariance Matrix ($\Sigma$): Defines the stretch (elongation) and rotation.
• Color & Opacity: The pigment and transparency.

Using the covariance matrix, the major axis represents the direction of the brush drag, while the minor axis represents the brush width.

Rendering Logic

To mimic natural layering, I used standard over-compositing. Each splat occludes the ones beneath it based on its alpha value. While this doesn't capture the physical texture of impasto paint, it closely resembles digital painting software like Krita or GIMP.

Comparison of Approaches

While projects like DrawingWithGaussians and 2d-gaussian-splatting-Art exist, they differ fundamentally from my approach:

Feature	Gradient Descent Approach	Edge-Guided Approach (Mine)
Method	Random seeds $\rightarrow$ Iterative nudging	Direct derivation from image structure
Speed	Slow / Computationally expensive	Fast / Direct
Transparency	Opaque "black box" optimization	Deterministic and predictable
Visual Result	Lossy image reconstruction	Stylized painting effect

---

Technical Hurdles: The Blending Problem

Initially, I followed a reference rasterizer that utilized additive blending. The formula looked like this:

pixel = background + Σ(intensity * color);

In an optimization regime, this works because the solver adjusts colors to account for overlap. However, when seeding thousands of splats directly from pixel data, the overlap became catastrophic. On a $64 \times 64$ image with 1,200 splats, some pixels reached a sum of 22.06, resulting in blown-out white blobs.

The Solution: I switched to the alpha compositing over-operator. This ensures that the summed color never exceeds $1.0$ and cleanly separates the sampled image color from the splat's opacity.

---

How Edges Guide the Paint

To demonstrate the evolution of the tool, I used a single source photo and toggled features on and off.

The Evolution Pipeline

1. The "Sad" Start

My first attempt produced a uniform mosaic. Because every splat had the same size, rotation, and aspect ratio, the result was just a grid of identical blobs.

2. Emulating the Artist

A real painter varies their technique:

• Flat Areas (Sky/Walls): Use broad, large strokes.
• Detailed Areas (Eyes/Fabric): Use small, numerous strokes that follow contours.

To achieve this, I implemented a Structure Tensor to analyze the image gradient. This involves creating a $2 \times 2$ tensor from the gradient outer product and blurring it over a local neighborhood.

3. The Mathematics of Direction

The eigenvectors of this tensor provide three critical pieces of data:

1. Major Eigenvector: Points across the contour (direction of maximum change).
2. Minor Eigenvector: Points along the edge $\rightarrow$ This is the brush stroke direction.
3. Coherence: The ratio of the eigenvalues, which indicates how "edge-like" the area is.

$\text{Coherence} = \frac{\lambda_1 - \lambda_2}{\lambda_1 + \lambda_2}$

![Image: Comparison of uniform blobs vs. edge-aligned strokes]