AI-Driven Hardware: The Emergence of openTPU

The boundary between software and hardware is blurring. openTPU represents a milestone where AI is no longer just the workload running on a chip, but the architect designing it. This project is an open-source AI accelerator created by AI agents, encompassing everything from the raw logic to the high-level compiler.

The Core Hypothesis:

1. To what extent can AI agents master the complexities of hardware engineering?
2. Is it possible for an AI to design the very silicon (or FPGA logic) required to execute its own inference?

---

🛠️ Project Scope & Architecture

The project is housed in a streamlined monorepo, allowing developers to trace the entire path from a high-level Python matrix multiplication down to the physical wires.

The openTPU Stack includes:

• RTL: Hardware descriptions written in SystemVerilog.
• ISA: A custom Instruction Set Architecture.
• Simulator: A bit-exact software simulation environment.
• Compiler: A dedicated kernel language and its associated compiler.
• Host Software: Drivers to manage a physical PCIe card.
• Profiler: Tools to analyze performance.

System Data Flow

---

🚀 Performance Benchmarks

The design was validated using an Inspur YPCB-00338 card, featuring a Xilinx Kintex-7 xc7k480t and dual DDR3 channels. The hardware output matches the simulator's results $\text{bit-for-bit}$.

Inference Results Table

---

🔍 Technical Deep Dive

Hardware Implementation

The current production image runs at 133.33 MHz and utilizes a single bitstream for all supported models. Key components include:

• LiteDRAM Controllers: These are calibrated by an internal CPU within the memory core, taking approximately 12 seconds upon startup without requiring host intervention.
• Systolic Matrix Unit: A four-column design optimized for tensor operations.
• Stream Engine: Detailed in docs/stream.md.

Measurement Methodology

Performance was measured using tools/qual/perf.py with the following parameters:

• Prompt: 512 tokens.
• Decode: 64 greedy tokens.
• Logic: The host handles the argmax in the loop.

The distinction between timing metrics is as follows: $\text{Wall Time} = \text{Device Cycles} + \text{Host Overhead}$

Memory Management (Gemma 4)

The handling of embedding tables varies by model size:

1. Gemma 4 E2B: Tables are stored on-card ($\approx 3.5\text{--}3.6\text{ GiB}$). Note: int8 is too large to fit.
2. Gemma 4 E4B: Tables ($\approx 2.95\text{ GB}$) reside on the host; the host copies a single $11\text{ KB}$ row per token. The total image size is $3.96\text{ GiB}$.

---

📈 Evolution & Iteration

The project has moved through several iterations to optimize DRAM utilization and speed.

se-cand3 $\rightarrow$ deploy_champ_e698dcd7 $\rightarrow$ deploy_fused133c_79c5707a (Build B)

Key Improvements in Build B:

• Speed: 8-10% faster decoding for LFM2-2.6B, SmolLM3, Phi-4-mini, and Gemma 4 E2B.
• Efficiency: DRAM utilization increased to $91\text{--}94\%$ (up from $82\text{--}87\%$).

Compared to the older se-cand3 (which used the Xilinx MIG, a two-column matrix unit, and a 120.755 MHz clock), the new image's decode performance is within $2.3\%$ across all configurations, as decode is primarily bound by DRAM bandwidth.

Example Configuration Snippet

# Mock representation of a model config in openTPU
model_config = {
    "name": "LFM2-2.6B",
    "precision": "int4",
    "head_precision": "int8",
    "clock_speed": "133.33MHz",
    "dram_peak_util": 0.94
}

For further details on board specifications, refer to docs/board.md (Section 5).