🌬️ AirLLM: Running 70B+ Models on a Single 4GB GPU

AirLLM is a revolutionary framework designed to drastically lower the memory barrier for Large Language Model (LLM) inference. It enables the execution of massive models—such as those with 70 billion parameters—on a consumer-grade GPU with as little as 4GB of VRAM.

Crucially, this is achieved ~~without~~ without relying on traditional distillation, pruning, or standard quantization.

🚀 Hardware Capabilities & Model Support

AirLLM leverages a unique streaming approach. For sparse Mixture-of-Experts (MoE) models, it streams only the specific experts required for a token rather than loading entire layers, allowing for unprecedented efficiency.

VRAM Requirements Overview

Model	Parameter Count	Required VRAM	Note
Kimi K3	2.8 Trillion	$< 4\text{GB}$	Largest open-source model; $\approx 3.72\text{GB}$ on RTX 6000 Ada
Llama 3.1	405 Billion	$\approx 8\text{GB}$	High-capacity general model
DeepSeek-V3	671 Billion	$\approx 12\text{GB}$	Massive MoE architecture
Qwen3	235 Billion	$\approx 3\text{GB}$	Highly efficient streaming
Standard LLMs	70 Billion	$\approx 4\text{GB}$	Base capability of AirLLM

⚠️ Special Requirements for Kimi K3

To run the 2.8T Kimi K3 model, the following environment is mandatory:

pip install compressed-tensors flash-attn (Flash Attention is required by the model code).
PyTorch built with CUDA 12 (since CUDA 13 wheels for flash-attn are unavailable).
transformers version 4.56.x (Remote code fails on 5.x).

📅 Project Evolution & Updates

🛠️ Quickstart Guide

AirLLM simplifies the loading process via the AutoModel class, which automatically detects the model architecture.

Basic Inference Implementation

from airllm import AutoModel

# Configuration
MAX_LENGTH = 128 

# Initialize model (works with HF repo IDs or local paths)
# Example: Qwen3-32B
model = AutoModel.from_pretrained("Qwen/Qwen3-32B") 

# For massive models, use the same one-liner:
# model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # ~3GB VRAM
# model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # ~12GB VRAM

input_text = ['What is the capital of United States?']
input_tokens = model.tokenizer(
    input_text, 
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH, 
    padding=False
)

# Execute generation
generation_output = model.generate(
    input_tokens, 
    max_new_tokens=20, 
    use_cache=True, 
    return_dict_in_generate=True
)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

Important Note: During the first run, AirLLM decomposes the original model and saves it layer-by-layer. Ensure your HuggingFace cache directory has ample disk space to store these shards.

⚡ Model Compression & Speed-up

AirLLM introduces block-wise quantization-based compression, which can increase inference speeds by up to 3x with negligible impact on accuracy.

How to Enable Compression

Install Dependencies: pip install -U bitsandbytes
Update AirLLM: pip install -U airllm (Ensure version $> 2.0.0$ )

Initialize with Compression:

model = AutoModel.from_pretrained(
    "garage-bAInd/Platypus2-70B-instruct", 
    compression='4bit' # Options: '4bit' or '8bit'
)

Compression vs. Standard Quantization

The primary difference lies in the bottleneck being addressed. In standard inference, the bottleneck is often compute/VRAM; in AirLLM, the bottleneck is disk loading speed.

$\text{Standard Quantization} \rightarrow \text{Quantize(Weights + Activations)} \rightarrow \text{Higher Accuracy Risk}$ $\text{AirLLM Compression} \rightarrow \text{Quantize(Weights Only)} \rightarrow \text{Lower Accuracy Risk}$

Feature	Standard Quantization	AirLLM Block-wise Compression
Target	Weights & Activations	Weights Only
Goal	Reduce VRAM/Increase Compute	Reduce Disk I/O Bottleneck
Accuracy	Prone to outlier impact	Highly stable/negligible loss

⚙️ Configurations

When initializing AutoModel, you can utilize the following parameters:

compression: Set to '4bit', '8bit', or None (default).
profiling_mode: Used for analyzing performance.
layer_shards_saving_path: Define a custom directory for the decomposed model layers.

🖼️ Project Assets

airllm_logo

Support me on Patreon