DeepInfer: A High-Performance Inference Engine for DeepSeek Models

DeepInfer is the specialized, large-scale inference framework utilized to power the production-grade services of DeepSeek. While the full production stack is proprietary, this open-source release focuses on providing high-efficiency offline inference capabilities.

🏗️ System Architecture Overview

The following diagram illustrates the conceptual flow of the DeepInfer execution pipeline:

---

🚀 Core Capabilities & Technical Details

🛠️ Optimized Kernels

DeepInfer leverages a suite of highly tuned kernels to maximize throughput and minimize latency:

⚙️ Advanced Feature Set

• Parallelism Strategies:
  • Attention Data Parallelism (DP) / Balanced Context Parallelism (CP).
  • Mixture-of-Experts (MoE) Expert Parallelism (EP) combined with $\text{EPLB}$ (Expert Parallel Load Balancing).
• DSpark & Verification: Accelerates the decoding process via draft tokens. Verification is managed through confidence estimates.
  • Config: Set DEEPINFER_ENABLE_MTP_VARLEN=1. Use DEEPINFER_USE_NEXTN_DECODE to define max draft length (Default: $5$).
• SWA Bounded Replay: Minimizes the computational load during decoder prefill by replaying only a specific bounded suffix.
  • Config: Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140.
  • Note: The suffix must encompass the SWA window $\mathcal{W}$ plus the maximum backtracking length $\mathcal{B}$.
• Asynchronous Scheduling: A zero-overhead batch scheduler that overlaps CPU-side management with GPU/NPU execution.
• Consistency & Determinism:
  • Batch-Invariant Inference: Ensures model outputs remain identical regardless of the batch composition.
  • Deterministic Sampling: Guarantees identical outputs for matching inputs and parameters.
  • Config: Set DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0, and provide a fixed seq_seed in SamplingParams (NVIDIA only).
• Structured Outputs: Supports constrained generation using JSON schema, regex, or EBNF grammar.
  • Usage: Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams.

---

💻 Hardware & Requirements

Supported Accelerators

• NVIDIA: SM100 GPUs
• Huawei: Ascend 950 NPUs

Prerequisites Checklist

• OS: Linux
• Compiler: GCC = 13.0
• Runtime: Python 3.12
• Driver (NVIDIA): CUDA 13
• Driver (Ascend): CANN 9.2

[!IMPORTANT] Production Gap: This release is for offline inference. To build a full production service, you would still need to implement:

• A dedicated KV cache system
• VE/Prefill/Decode disaggregation (though most engine logic is present)
• Global load balancing across DP replicas

---

🛠️ Installation & Usage

Building from Source

Ensure you are using a clean Python 3.12 environment before proceeding.

# For NVIDIA hardware
export DEEPINFER_ACCELERATOR=nvidia 
./build.sh

# For Huawei Ascend hardware
# export DEEPINFER_ACCELERATOR=ascend
# ./build.sh

Running Offline Inference

By default, the framework pulls deepseek-ai/DeepSeek-V4.1-Flash-1017 from Hugging Face. To use a local directory, set DEEPINFER_HF_MODEL_DIR=/path/to/model.

# Execute the provided example script
python examples/offline_inference.py

---

📊 Performance & Credits

Benchmarks

Tests were conducted on the following hardware configurations:

• Prefill: 4 $\times$ SM100 GPUs (single scale-up domain).
• Decode: 64 $\times$ SM100 GPUs (single scale-up domain).

Acknowledgments & Citation

DeepInfer was developed using vLLM as a primary foundation, with further architectural inspiration drawn from SGLang and TensorRT-LLM.

BibTeX Citation:

@misc { deepinfer2026 , 
title = { DeepInfer: High-performance, large-scale inference framework for DeepSeek models } , 
author = { DeepSeek-AI } , 
year = { 2026 } , 
publisher = { GitHub } , 
howpublished = { \url{https://github.com/deepseek-ai/DeepInfer} } , 
}