WASTE: Weight-Aware Streaming Tensor Engine

WASTE is a specialized, embeddable inference engine written in C that enables the execution of massive models—specifically the 2.78-trillion-parameter Kimi K3—on consumer-grade hardware by streaming weights directly from NVMe storage.

🚀 The Proof of Concept

Running a model of this magnitude on a laptop was previously considered infeasible. WASTE proves otherwise.

Example Execution:

$ waste run ~/models/k3.waste 'What is the capital of Italy?'
waste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB)
The capital of Italy is **Rome**.

[16 tokens, 25.78 s, 0.62 tok/s | experts 9038 hit / 14514 miss = 38%]

---

🛠️ How it Works

WASTE operates on a simple but effective architectural premise: Idle weights do not need to reside in RAM; they only need to be reachable in time.

Since Mixture of Experts (MoE) models only activate roughly 4% of their parameters per token, the vast majority of the model is dormant at any given moment. WASTE manages this by:

1. Keeping the model trunk permanently in memory.
2. Streaming specific experts from the disk as needed.
3. Utilizing all remaining available RAM as a bounded expert cache.

Data Flow Architecture

📊 Performance Benchmarks

---

🔍 Technical Validation & Nuance

Accuracy

The engine is mathematically sound. Every layer has been verified against a PyTorch reference:

• Final Logits: Match to within $3.6 \times 10^{-6}$.
• Vision Tower: Matches its oracle to within $2.3 \times 10^{-6}$.

The Speed Trade-off

It is admittedly slow—roughly half a token per second. However, this is a feature of feasibility, not a failure of design. While 671B-class models usually require servers with a terabyte of DDR5, WASTE brings trillion-scale models to a single consumer machine.

Note: This is not a comprehensive academic survey. The authors found no other published demonstrations of trillion-scale NVMe streaming on consumer hardware, but they welcome counter-examples.

Engineering Optimizations

The team experimented with several "levers" to increase performance:

• Reducing bytes read per token (The router lacks a "tail" to demote).
• Increasing resident RAM (Hardware limits are absolute).

What actually worked:

• Arithmetic Overlap: Overlapping expert reads with computation provided a $\approx 1.6\text{x}$ boost.
• Predictive Reading: Starting the next layer's reads based on the router's guess (one residual early) increased the cache hit rate from $14\%$ to $38\%$.

Detailed accounts of these experiments can be found in docs/EFFICIENCY.md and docs/LEARNED.md.

---

💡 The Philosophy of "WASTE"

The project is named WASTE because of the inefficiency of current cloud AI. Every token generated by a cloud service is paid for twice:

1. Through the user's financial invoice.
2. Through the massive electrical cost of datacenters running models that could actually fit on a desk-bound machine.

By enabling local execution, WASTE removes the need for network dependencies, per-token billing, and the privacy risks of sending data to an API.

---

📋 System Requirements

Hardware & Storage:

• Disk Space:
  • 982 GB for the final converted container.
  • 1.42 TB of temporary staging space for published shards during conversion.
• RAM:
  • Minimum 29.05 GB to open K3 with a 4K context.
  • 64 GB recommended for the speeds listed above.
• Storage Speed: Must be an internal NVMe drive.

Software & Build:

• Compiler: C11 compliant compiler.
• Build Tool: make.
• Runtime Dependencies: None. (No BLAS, no CUDA, no Python).