Dust: Pretraining Transformers Without Backpropagation

Authors: Samip Dahal, Bishwas Mandal, Serdar Gülbahar, Akshay Vegesna
Institution: Q Labs Research
Date: October 2026
Contact: s@qlabs.sh | [Code] | [Cite]

TL;DR: The researchers introduce Dust, the first zeroth-order optimization technique that can compete with backpropagation (backprop) during the pretraining of transformer-based language models.

---

📌 Executive Summary

Dust utilizes node perturbation to modify activations independently for every token. This transforms each token into a "virtual population member," allowing a single forward pass to evaluate the entire population simultaneously.

Key Findings:

• Performance: Dust is competitive with backprop and, in certain high-compute scenarios, actually surpasses it.
• Efficiency: It is vastly more efficient than weight-space Evolution Strategies (ES). Specifically, for sequences $\ge 1\text{M}$ tokens, Dust is estimated to be $10^3$ to $10^4$ times more efficient than EGGROLL (a leading ES method).
• Scaling Paradox: While it is commonly believed that zeroth-order methods fail as networks grow, Dust proves the opposite: larger models are more population-efficient. A model with $243\text{M}$ parameters outperformed a model $120\times$ smaller across most population sizes.
• Alignment: Gradient estimates produced by Dust remain well-aligned with those of backprop up to $1\text{B}$ tokens.

---

1. Introduction

For decades, deep learning has been synonymous with backpropagation. Backprop is currently the only credit assignment mechanism capable of training massive transformer models because it provides first-order gradients via differentiability. Consequently, our hardware and architectures have been designed specifically to support this constraint.

However, the authors argue that as global compute capacity expands, we should shift toward more "brute-force" algorithms that rely on search rather than strict analytic requirements like differentiability.

The Philosophy of Compute

This approach is rooted in "The Bitter Lesson" (Sutton, 2019), which posits that general methods leveraging compute eventually outperform methods relying on human-engineered inductive biases.

"General methods that scale with compute eventually win."

Example: AlphaGo Zero initially learned slower than versions bootstrapped with human data, but eventually overtook them through pure self-play and massive computation.

Similarly, while backprop is an efficient inductive bias for low-compute regimes, it may actually restrict the architectural possibilities in high-compute regimes. Furthermore, gradient-based methods often fail to explore the loss landscape optimally, which may be why current models require such astronomical amounts of data to generalize.

Research Objectives

The goal of this paper is to replace backprop with a search-based credit assignment algorithm.

• Eliminate the need for differentiability.
• Reduce reliance on analytic structure.
• Create a scalable zeroth-order optimizer.
• Leverage activation-space search for better generalization.

---

2. The Dust Mechanism

Dust is a zeroth-order optimizer. Instead of calculating a derivative, it:

1. Perturbs activations.
2. Rewards perturbations based on their ability to lower the loss function.
3. Averages these reward-weighted perturbations across a population to estimate the gradient.

Virtual Populations vs. Weight-Space ES

Traditional Evolution Strategies (like EGGROLL) perturb weights. This is computationally expensive because every population member must be materialized and evaluated separately. Dust solves this via the Virtual Population concept.

Comparison Table: Optimization Approaches

Technical Workflow

Dust perturbs activations independently at every token. Since each token acts as a member of the population, one forward pass evaluates all members in parallel.

The Power of Activation Space

The authors suggest that searching in activation space is more promising than weight space. Recent work in mechanistic interpretability suggests that reasoning (whether explicit or implicit) resides within the activations. By searching here, Dust essentially turns training into a search over latent reasoning.

Mathematical Intuition

The gradient estimate $\hat{g}$ can be viewed as a weighted sum of perturbations $\epsilon$:

$\hat{g} \approx \frac{1}{N} \sum_{i=1}^{N} \frac{\mathcal{L}(\theta) - \mathcal{L}(\theta + \epsilon_i)}{\|\epsilon_i\|^2} \epsilon_i$

(Note: This is a simplified representation of the reward-weighting logic used in zeroth-order methods.)

# Conceptual Pseudo-code for Dust Update
def dust_update(model, tokens, learning_rate):
    # 1. Perturb activations for each token (Virtual Population)
    perturbed_activations = apply_node_perturbation(model, tokens)
    
    # 2. Evaluate loss for the population in one forward pass
    losses = model.forward(perturbed_activations)
    
    # 3. Calculate reward-weighted gradient estimate
    grad_est = compute_weighted_average(perturbed_activations, losses)
    
    # 4. Update weights without backprop
    model.weights -= learning_rate * grad_est

---

3. Conclusion and Scaling

The most striking result is the relationship between model size and efficiency. Contrary to the belief that zeroth-order methods cannot scale, Dust finds that:

• Larger models $\rightarrow$ Higher population efficiency.
• The alignment between Dust and backprop remains stable up to $1\text{B}$ tokens.

This suggests that in a compute-rich future, Dust could potentially surpass the limitations of backpropagation entirely.