Technical Report: DiffusionGemma

Executive Summary: DiffusionGemma is an experimental, open-weight language model designed to shatter the speed limitations of traditional text generation. By utilizing discrete diffusion instead of sequential token prediction, it achieves massive throughput gains while maintaining the core intelligence of its base architecture.

---

🚀 The Paradigm Shift: From AR to Diffusion

Traditional Large Language Models (LLMs) rely on Autoregressive (AR) decoding, where tokens are generated one by one. This creates a significant bottleneck. DiffusionGemma replaces this with a parallel refinement process.

Sequential Token-by-Token Decoding $\rightarrow$ Parallel Block Refinement

Instead of predicting the next single token, DiffusionGemma iteratively polishes blocks of $N=256$ tokens simultaneously.

Model Architecture Specifications

The model is built upon the Gemma 4 Mixture-of-Experts (MoE) framework.

---

🛠️ Training Methodology

The team implemented a compute-efficient, two-stage pipeline. Remarkably, this process consumed less than 10% of the original AR model's total training token budget.

Detailed Training Stages:

1. Supervised Fine-Tuning (SFT): The model is taught the art of bidirectional denoising, allowing it to look at a corrupted block of text and recover the original content.
2. RL & Distillation: A combination of Reinforcement Learning and sampler distillation is used to sharpen the output quality and reduce the number of iterations needed for convergence.

---

📊 Performance & Benchmarks

DiffusionGemma establishes a new Pareto frontier for the balance between speed and capability.

Key Metrics:

• Efficiency: $\approx 20$ tokens generated per single forward pass.
• Throughput: $\approx 1,500$ output tokens per second (on a single H100).
• Comparison: This speed significantly outperforms standard AR models, even those utilizing state-of-the-art speculative decoding.

Preserved Capabilities

Despite the shift in decoding logic, the model retains the following features:

• Thinking Mode: Ability to perform internal reasoning.
• Multimodal Inputs: Processing of non-textual data.
• Long Contexts: Handling of extensive input windows.

---

🧬 Hybrid Potential and Conclusion

One of the most intriguing findings is that DiffusionGemma does not lose its original identity. It remains capable of standard AR generation with only a slight dip in performance.

# Conceptual Hybrid Decoding Logic
if task_requires_precision == True:
    mode = "Autoregressive" # High precision, slower
else:
    mode = "Diffusion"       # High speed, parallel

This suggests a future where models can dynamically switch between diffusion and AR decoding depending on the needs of the prompt.

Metadata Reference:

• arXiv ID: 2608.00146
• Submission Date: July 31, 2026
• License: