Explorative Modeling: Optimizing via the Best of $K$ Guesses

By Alexi Gladstone | Created July 29, 2026 Tags: #generative_models #generalization #diffusion #autoregression

Project Links: Website | GitHub

---

🚀 Executive Summary (TL;DR)

We are introducing Explorative Modeling (XM), a novel generative framework that serves as a "third axis" for pretraining. By integrating XM into existing models, we can achieve true end-to-end generation.

The results are clear: increasing exploration leads to monotonic improvements in language, video, and image synthesis. Furthermore, these benefits scale alongside data and model size.

Performance Gains

Key Takeaway: XMs can match the control capabilities of diffusion models while requiring up to $256\times$ less compute during inference.

---

🐕 The Problem: The "Brown Blur" Paradox

Imagine asking an AI to "generate a dog." In reality, there are billions of valid images that qualify as a "dog." If we attempt to train a neural network to predict a dog image via direct regression, we hit a wall.

Because the model sees thousands of different valid dogs, it attempts to find the single mathematical prediction that minimizes the distance to all of them. This result is the average.

The result? A realistic dog $\rightarrow$ A brown blur.

---

🎯 The Dartboard Analogy

To understand why this happens, consider a game of darts. If you have to guess where the next dart will land on a ringed board, and your score is based on how close you are to the actual point:

1. The mathematically optimal guess to minimize total error is the exact center of the board.
2. However, the center is almost never where a dart actually lands.

This phenomenon isn't limited to images or darts; it is a universal failure of direct prediction:

• 2D Point Clouds: The model predicts a single dot in the center of the clusters.
• Natural Language: The model simply outputs the word "the".

---

🛠️ The Current Solution: Factoring Generation

To avoid the "average" problem, every successful modern generative model uses Factoring Generation. This means breaking a complex output into many small steps, where each step has roughly one correct answer.

Common Factoring Methods:

• Autoregressive Models (e.g., LLMs): They predict one token at a time. In the dart game, this would be guessing the X-coordinate first, then the Y-coordinate.
• Diffusion Models: They start with pure Gaussian noise and take hundreds of tiny steps to refine the image.

The logic is simple: $\text{Many Valid Answers} \xrightarrow{\text{Factoring}} \text{One Right Answer per Step}$

---

👿 The "Evil" of Factoring

While factoring works, it introduces two critical flaws:

• Exposure Bias: Models are trained on single steps but run for thousands during inference. Because the model feeds its own imperfect outputs back into itself, errors compound. This is why:
  • Video models "melt" after a few seconds.
  • Long-form text loses coherence.
• Lack of End-to-End Learning: These models do not run at inference exactly how they were trained.

Historical Context: The deep learning revolution (starting with AlexNet) was driven by end-to-end learning—letting the model learn directly from data rather than using hand-designed pipelines. Generative modeling is currently the only major field that hasn't fully embraced this.

# Pseudo-code of the Exposure Bias problem
def generate(model, seed):
    x = seed
    for step in range(1000):
        # Training was on 1 step, but inference is recursive
        x = model.predict_next_step(x) 
        # Error compounds here ^
    return x

By moving toward Explorative Modeling, we can finally bring generative AI into the end-to-end era.