🐦 Colibrì: Running GLM-5.2 on Low-Spec Hardware

Colibrì is a lightweight engine designed to execute the massive GLM-5.2 (744B Mixture-of-Experts) model on consumer-grade machines with as little as $\approx 25\text{ GB}$ of RAM. It achieves this through a "zero-dependency" pure C implementation that streams model experts directly from the disk.

$ ./coli chat 
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU ✓ pronto in 32s · residente 9.9 GB 
› ciao!

---

🧠 The Core Concept

The fundamental insight is that while GLM-5.2 has $744\text{B}$ parameters, it is a Mixture-of-Experts (MoE) model. This means for any single token, only $\approx 40\text{B}$ parameters are actually activated. Specifically, only about $11\text{ GB}$ of those parameters change between tokens (the routed experts).

The Strategy: Keep the "dense" components (embeddings, attention, and shared experts $\approx 17\text{B}$ params) resident in RAM at int4 precision ($\approx 9.9\text{ GB}$). The $21,504$ routed experts (across $75$ MoE layers $\times 256$ experts, plus the MTP head) are stored on disk ($\approx 370\text{ GB}$) and streamed on-demand.

🛠️ Memory Hierarchy

The engine utilizes a multi-tiered caching system to minimize disk I/O latency:

1. L1: Per-layer LRU cache.
2. L2: OS page cache (automatic).
3. Optional: A pinned "hot-store" for frequently used experts.

---

⚙️ Technical Implementation

The entire engine is remarkably compact, consisting of a single C file (c/glm.c, roughly $2,400$ lines) and a few headers. It avoids BLAS and Python at runtime, and while it doesn't require a GPU, it offers an optional CUDA tier for pinned experts.

✅ Feature Checklist

• Faithful Forward Pass: The glm_moe_dsa implementation is token-exact compared to the transformers oracle.
• MLA Attention: Implements q/kv-LoRA and interleaved partial RoPE.
• Compressed KV-Cache: Reduced from 32,768 to $576$ floats per token (a $57\times$ reduction).
• DeepSeek-V3 Router: Includes noaux_tc and routed_scaling_factor.
• MTP Speculative Decoding: Uses the layer 78 multi-token-prediction head to draft tokens.
• Integer-Dot Kernels: AVX2 maddubs for int8 activations, providing $1.4\text{--}2.5\times$ speedups.
• Async Readahead: Uses WILLNEED to fetch the next expert block while the current one is processing.
• DSA Sparse Attention: Faithful to the reference indexer, utilizing top-2048 causal key selection.
• KV-Cache Persistence: Saves state to .coli_kv ($\approx 182\text{ KB/token}$) for instant restarts.
• BPE Tokenizer: A native C implementation of the GPT-2 style tokenizer with $320\text{k}$ merges.

---

🔍 Deep Dive: Nuances & Optimizations

Speculative Decoding (MTP)

The MTP head must be int8. If run at int4, the acceptance rate drops to $0\text{--}4\%$, rendering speculation useless. At int8, acceptance rises to $39\text{--}59\%$, yielding $2.2\text{--}2.8$ tokens per forward pass.

Caveat: On a "cold" cache, the extra expert loads required for verification can actually slow down generation. The DRAFT=0 flag or the adaptive guard can be used to mitigate this.

MLA Weight Absorption

To optimize decoding, Colibrì uses the "DeepSeek trick" where the query absorbs $kv\_b$, removing the need for per-token $k/v$ reconstruction.

Router-Lookahead (PILOT=1)

Experimental data shows that the next layer's routing is $\approx 71.6\%$ predictable from the current layer's post-attention state. A dedicated I/O thread uses this to prefetch experts.

RAM Safety

To prevent the Linux OOM-killer from terminating the process, the expert cache is dynamically sized at startup based on MemAvailable, accounting for: $\text{Working Set} + \text{KV Cache} + \text{MTP Row} + \text{Reconstruction Buffers}$

---

📦 Conversion Process

The c/tools/convert_fp8_to_int4.py script allows for the creation of the int4 container without needing $756\text{ GB}$ of free space.

The Workflow:

1. Download one shard ($\approx 5\text{ GB}$).
2. Dequantize (using $128\times 128$ block scales).
3. Requantize into the Colibrì format.
4. Delete the shard and repeat.

---

📊 Performance Metrics

Tested on: WSL2, 12 Cores, 25 GB RAM, NVMe via VHDX