Speculative Decoding in vLLM on AMD GPUs

---

TL;DR: Speculative decoding enhances vLLM by allowing the system to validate a batch of "drafted" tokens during a single pass of the target model. Our findings indicate that the impact on output-token throughput is highly variable, depending on the specific drafting technique, the length of the proposals, the model architecture, the draft checkpoint, the nature of the workload, and the acceptance rate.

Introduction

While Large Language Models (LLMs) enable a vast array of capabilities, deploying them at scale necessitates rigorous optimization. The industry standard is autoregressive decoding, where the model generates a single token, appends it to the existing sequence, and repeats the process to find the next token.

While this method is reliable, it is inherently sequential. Because tokens must be produced in a strict left-to-right order, the serving loop only advances by one committed token per iteration.

To break this bottleneck, speculative decoding [1] introduces a "draft-and-verify" architecture. In this setup:

1. A lightweight draft component suggests potential future tokens.
2. The target model (the larger, primary model) verifies these suggestions.

If the target model accepts multiple draft tokens, the system can commit several tokens in a single verification cycle without altering the original model's output distribution.

This article examines the implementation of speculative decoding within vLLM. We will:

• Contrast autoregressive decoding with the draft-and-verify approach.
• Analyze five distinct drafting strategies: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark.
• Discuss the differences in how these methods handle information flow and token generation (sequential, parallel, or hybrid).
• Provide guidance on enabling these features on AMD Instinct™ MI300X and MI355X GPUs via the ROCm™ platform.

---

The Autoregressive Decoding Baseline

In a standard autoregressive setup, every single decode step results in exactly one committed token. To generate four tokens, the system must perform four distinct sequential passes:

$\text{Step 1: } \text{context} \rightarrow \text{model} \rightarrow T_1$ $\text{Step 2: } \text{context} + T_1 \rightarrow \text{model} \rightarrow T_2$ $\text{Step 3: } \text{context} + T_1 + T_2 \rightarrow \text{model} \rightarrow T_3$ $\text{Step 4: } \text{context} + T_1 + T_2 + T_3 \rightarrow \text{model} \rightarrow T_4$

While simple, this $\text{1:1}$ ratio between model passes and output tokens creates a latency bottleneck during long generations. The fundamental goal of speculative decoding is to maintain the target model's exact output behavior while reducing the total number of required decoding rounds.

---

Core Logic of Speculative Decoding

Speculative decoding does not replace the primary model; it supplements it. The original model remains the target model, serving as the final authority on correctness, while a faster proposal stage is placed upstream.

The Two-Stage Process

1. The Draft Phase: A small, efficient draft component proposes a sequence of candidate tokens. These are speculative and not yet committed to the final output.
2. The Verification Phase: The target model evaluates the entire candidate sequence in one single pass.

Verification Rules:

• Acceptance: Draft tokens are checked from left to right.
• Rejection: As soon as a draft token is rejected, all subsequent candidates in that proposal are discarded.
• Correction: When a rejection occurs, the target model provides the correct token for that position.

Comparison of Advancement

---

A Practical Example: Accept/Reject

Consider the following scenario:

Current Prompt: "The weather today is"

Draft Proposal: sunny $\rightarrow$ and $\rightarrow$ warm $\rightarrow$ outside

Verification Process:

Final Output for this round: "The weather today is sunny and clear"

In this instance, the system committed three tokens (sunny, and, clear) using only one target model pass. The next round will begin from the sequence ending in clear.

---

Exploring Drafting Methods

While the high-level logic remains consistent, the mechanism used to generate proposals varies. We have tested the following approaches:

• Native MTP (Multi-Token Prediction)
• Gemma 4 MTP
• EAGLE-3
• DFlash
• DSpark

These methods differ primarily in how the draft component accesses information from the target model and whether they generate candidates sequentially, in parallel, or via a hybrid strategy. Although all speculat...