Deploying DeepSeek V4 Flash on a Single AMD MI300X

This repository provides the specialized configuration, patches, and tuning parameters required to deploy deepseek-ai/DeepSeek-V4-Flash-0731 on a single AMD MI300X GPU for production environments.

The setup utilizes a Docker Compose stack, SHA-256 pinned file overlays, reference diffs against the upstream source, and optimized tuning tables. Notably, the checkpoint is executed exactly as shipped, meaning ~~no additional weight quantization~~ and no weight offloading are required.

🚀 Performance Benchmarks

The following results were obtained using the pinned stack (AITER 0.1.19 and vLLM ROCm nightly 0.26.1rc1.dev229+g124154a88.rocm723):

Metric	Result
Single-stream decode (median per-stream, DSpark-7)	$168.6 \text{ tok/s}$
Prefill (Tuned Kernels)	$\approx 7.9\text{--}8.5\text{K tok/s}$
Prefill (Fresh prompts, shipping profile)	$6,988\text{--}7,019 \text{ tok/s}$
8 Concurrent Streams	$542 \text{ tok/s}$ aggregate ( $90.3 \text{ tok/s}$ median)
64-Stream Burst	$830 \text{ tok/s}$ aggregate (No OOM or engine errors)
Validated Context	$256\text{K}$ (Architecture supports up to $1\text{M}$ )
HBM Weight Footprint	$156.67 \text{ GiB}$

🛠️ Technical Rationale: Why the MI300X?

The AMD MI300X is uniquely suited for this 304B-parameter model due to its massive memory specifications:

HBM3 Capacity: $192 \text{ GB}$ (approximately $2.4\times$ the capacity of an H100 SXM5).
Memory Bandwidth: $5.3 \text{ TB/s}$ .

According to Doubleword, the MI300X's list price is roughly half that of competing high-end NVIDIA hardware, making it a highly cost-effective choice for large-scale deployments.

Because the model weights occupy $156.67 \text{ GiB}$ , the MI300X allows for a simplified single-GPU deployment. There is no need for PCIe weight streaming or layer offloading. This leaves ample room for:

A 20 GB GPU KV pool.
A 96 GiB CPU tier for managing evicted prefix-cache entries.

The FP8 Challenge

A critical nuance of the MI300X (CDNA3) is its implementation of the fnuz variant of E4M3 (AMD/Graphcore). In contrast, the MI325X and newer models utilize the OCP-standard FP8.

Warning: If a kernel assumes OCP semantics on an MI300X, the scale domain can be incorrect by a factor of two. Consequently, ensuring FP8 correctness was the primary objective before performance tuning began.

🔍 Enhancements & Prior Art

While previous work by Fergus Finn and Doubleword identified issues like missing AITER fast paths on gfx942, HIP-graph hazards in sparse MLA decode, and MoE routing bugs, the official vLLM recipe ~~only targets NVIDIA and newer AMD GPUs~~ (like the MI325X/MI355X).

This repository fills the gap for single-MI300X production by adding:

Correctness Overlays: Patches for the pinned ROCm nightly that are not yet upstreamed in vLLM.
Serving Configuration: A validated setup featuring:
- Probabilistic DSpark drafting.
- Block rejection and a static $K=7$ .
- A $2,048$ -token scheduler budget and $1,024$ -token long-prefill cap to prevent "cold prompt" stalls.
AITER Tuning: GEMM tuning tables for gfx942 shapes and an OGS geometry override for MXFP4 experts.
Hybrid KV Strategy: A combination of $20 \text{ GB}$ fp8_ds_mla GPU cache and $96 \text{ GiB}$ native CPU offload, including a fix for the load-path fencing issue (documented in issue #47282).

📂 Repository Structure

compose.yaml: The digest-pinned production stack (vLLM ROCm + Caddy).
Caddyfile.example: Template for the HTTPS proxy (requires hostname/email/CIDR).
vllm-entrypoint.sh: Script to clear stale CPU-KV mmaps from /dev/shm.
SHA256SUMS: Integrity pins for all runtime artifacts.
patches/: Production overlays and unified diffs against the base revision.
tuning/: AITER A8W8 blockscale tuning tables for gfx942.

⚙️ Runtime Configuration

The stack utilizes a pinned vLLM ROCm nightly with the following settings:

Model Logic: --trust-remote-code enabled; uses DeepSeek V4 tokenizer and reasoning parsers.
KV Cache: fp8_ds_mla (UE8M0 block-scaled FP8) with $256$ -token blocks.
Backend: VLLM_ROCM_USE_AITER=1 and --moe-backend triton.
- Triton OGS $\rightarrow$ Grouped MXFP4 experts.
- AITER $\rightarrow$ Attention and dense linear layers.
Decoding: DSpark-7 speculative decoding with probabilistic drafting.
Optimization: Full/breakable CUDA graph capture for single-graph launch per token.
Proxy: Caddy serves as an IP-allowlisted HTTPS gateway.

🚀 Deployment Guide

1. Host Requirements

GPU: 1x AMD MI300X (gfx942, 304 CUs, $\approx 192 \text{ GiB}$ HBM).
Driver: Working AMD kernel driver.
Software: Recent Docker Compose.
RAM: $\approx 235 \text{ GiB}$ (for the CPU KV tier).
Disk: $\approx 500 \text{ GB}$ (Model cache is $\approx 156 \text{ GB}$ ).

2. Pulling the Runtime and Model

Run the following commands to fetch the pinned image and model weights:

VLLM_IMAGE='vllm/vllm-openai-rocm@sha256:e68d18b2ba50298661bfc49baf01158fbf036645c2362cccf3e8a7a79fe6c69a'
MODEL='deepseek-ai/DeepSeek-V4-Flash-0731'
REVISION='7872f01b1d1fe23eabc4c98b48bffcef5a386062'

docker pull "$VLLM_IMAGE"
docker run --rm --entrypoint hf \
  -v /root/.cache/huggingface:/root/.cache/huggingface \
  "$VLLM_IMAGE" download "$MODEL" --revision "$REVISION"

3. Final Preparation

# Setup the proxy configuration
cp Caddyfile.example Caddyfile 
# Edit Caddyfile to set your hostname, email, and remote_ip CIDR

# Create necessary directories
mkdir -p aiter-cache crash-dumps ch