🌬️ AirLLM: Running 70B+ Models on a Single 4GB GPU

AirLLM is a revolutionary framework designed to drastically lower the memory barrier for Large Language Model (LLM) inference. It enables the execution of massive models—such as those with 70 billion parameters—on a consumer-grade GPU with as little as 4GB of VRAM.

Crucially, this is achieved without without relying on traditional distillation, pruning, or standard quantization.

🚀 Hardware Capabilities & Model Support

AirLLM leverages a unique streaming approach. For sparse Mixture-of-Experts (MoE) models, it streams only the specific experts required for a token rather than loading entire layers, allowing for unprecedented efficiency.

VRAM Requirements Overview

⚠️ Special Requirements for Kimi K3

To run the 2.8T Kimi K3 model, the following environment is mandatory:

• pip install compressed-tensors flash-attn (Flash Attention is required by the model code).
• PyTorch built with CUDA 12 (since CUDA 13 wheels for flash-attn are unavailable).
• transformers version 4.56.x (Remote code fails on 5.x).

---

📅 Project Evolution & Updates

---

🛠️ Quickstart Guide

AirLLM simplifies the loading process via the AutoModel class, which automatically detects the model architecture.

Basic Inference Implementation

from airllm import AutoModel

# Configuration
MAX_LENGTH = 128 

# Initialize model (works with HF repo IDs or local paths)
# Example: Qwen3-32B
model = AutoModel.from_pretrained("Qwen/Qwen3-32B") 

# For massive models, use the same one-liner:
# model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # ~3GB VRAM
# model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # ~12GB VRAM

input_text = ['What is the capital of United States?']
input_tokens = model.tokenizer(
    input_text, 
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH, 
    padding=False
)

# Execute generation
generation_output = model.generate(
    input_tokens, 
    max_new_tokens=20, 
    use_cache=True, 
    return_dict_in_generate=True
)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

Important Note: During the first run, AirLLM decomposes the original model and saves it layer-by-layer. Ensure your HuggingFace cache directory has ample disk space to store these shards.

---

⚡ Model Compression & Speed-up

AirLLM introduces block-wise quantization-based compression, which can increase inference speeds by up to 3x with negligible impact on accuracy.

How to Enable Compression

1. Install Dependencies: pip install -U bitsandbytes
2. Update AirLLM: pip install -U airllm (Ensure version $> 2.0.0$)
3. Initialize with Compression:

   model = AutoModel.from_pretrained(
       "garage-bAInd/Platypus2-70B-instruct", 
       compression='4bit' # Options: '4bit' or '8bit'
   )
   

Compression vs. Standard Quantization

The primary difference lies in the bottleneck being addressed. In standard inference, the bottleneck is often compute/VRAM; in AirLLM, the bottleneck is disk loading speed.

$\text{Standard Quantization} \rightarrow \text{Quantize(Weights + Activations)} \rightarrow \text{Higher Accuracy Risk}$ $\text{AirLLM Compression} \rightarrow \text{Quantize(Weights Only)} \rightarrow \text{Lower Accuracy Risk}$

---

⚙️ Configurations

When initializing AutoModel, you can utilize the following parameters:

• compression: Set to '4bit', '8bit', or None (default).
• profiling_mode: Used for analyzing performance.
• layer_shards_saving_path: Define a custom directory for the decomposed model layers.

---

🖼️ Project Assets