Sub-1-Bit LLM Compression via Latent Factorization

This repository contains the official code for LittleBit (presented at NeurIPS 2025) and its successor, LittleBit-2 (presented at ICML 2026).

📖 Overview

Abstract: LittleBit achieves extreme compression of Large Language Models (LLMs) by pushing them into the sub-1-bit regime. This is accomplished by decomposing dense weight matrices into low-rank latent factors, binarizing those factors, and utilizing lightweight, learned scales to recover magnitude data. This approach allows for compression as aggressive as 0.1 bits-per-weight (bpw) without altering the model's architecture during inference.

The Evolution: LittleBit $\rightarrow$ LittleBit-2

While the original LittleBit established the framework, LittleBit-2 optimizes the process by fixing latent geometry misalignment during the initial setup. It introduces Joint Iterative Quantization (Joint-ITQ) for Internal Latent Rotation, which ensures that SVD-based latent factors are properly aligned with the binary hypercube before Quantization-Aware Training (QAT) begins.

Key Technical Highlights

• Extreme Compression: Specifically engineered for the range of $1.0 \rightarrow 0.1$ bits per weight.
• Zero Inference Overhead: The --use_itq flag in LittleBit-2 only affects the initialization phase; the resulting deployed layer is identical to the original LittleBit structure.
• Training Ready: Fully compatible with QAT, utilizing SmoothSign and optional residual factorization.

---

⚙️ Methodology

The core idea is to approximate a weight matrix $W$ using the following factorization:

$W \approx \text{scale} \times (B_1 \cdot B_2)$

Where $B_1$ and $B_2$ are binarized latent factors.

Compression Pipeline

---

🛠️ Implementation Details

Supported Architectures

The following models are currently compatible with the LittleBit codebase:

Installation Guide

Prerequisites: Python 3.12 is highly recommended.

• Create environment
• Install CUDA Toolkit
• Install PyTorch
• Install Dependencies

# 1. Environment Setup
conda create -n littlebit python=3.12 
conda activate littlebit 

# 2. CUDA Toolkit (v12.4.1)
conda install nvidia/label/cuda-12.4.1::cuda-toolkit -c nvidia/label/cuda-12.4.1 

# 3. PyTorch Installation
pip install torch==2.8.0+cu124 torchvision==0.23.0+cu124 torchaudio==2.8.0+cu124 --index-url https://download.pytorch.org/whl/cu124 

# 4. General Dependencies
pip install -r requirements.txt

Warning: To ensure the results match the published papers, you must use transformers 4.51.x. Newer versions may introduce changes to model internals that affect evaluation. pip install "transformers==4.51.*"

---

🚀 Usage

Training (QAT)

You can train using the LittleBitLinear module. By default, it uses SVD initialization. To upgrade to LittleBit-2, add the --use_itq True flag.

Single GPU Execution:

CUDA_VISIBLE_DEVICES=0 python -m main \
 --model_id meta-llama/Llama-2-7b-hf \
 --dataset c4_wiki \
 --save_dir ./outputs/Llama-2-7b-LittleBit-2 \
 --num_train_epochs 5.0 \
 --per_device_train_batch_size 4 \
 --lr 4e-05 \
 --warmup_ratio 0.02 \
 --report wandb \
 --quant_func SmoothSign \
 --quant_mod LittleBitLinear \
 --residual True \
 --eff_bit 1.0 \
 --kv_factor 1.0 \
 --min_split_dim 8 \
 --l2l_loss_scale 10.0 
# --use_itq True  # Enable LittleBit-2 initialization

Multi-GPU (DeepSpeed):

deepspeed --num_gpus=4 main.py \
 --model_id meta-llama/Llama-2-7b-hf \
 --dataset c4_wiki \
 --save_dir ./outputs/Llama-2-7b-LittleBit-2 \
 --ds_config_path configs/zero3.json \
 --num_train_epochs 5.0 \
 --per_device_train_batch_size 4 \
 --lr 4e-05 \
 --report wandb \
 --quant_func SmoothSign \
 --quant_mod LittleBitLinear \
 --residual True \
 --eff_bit 1.0 \
 --kv_factor 1.0 \
 --min_split_dim 8

Evaluation

You can evaluate models from a local path or directly from the Hugging Face Hub.

# Local Checkpoint
CUDA_VISIBLE_DEVICES=0 python eval.py \
 --model_id ./outputs/Llama-2-7b-LittleBit-2 \
 --seqlen 2048 \
 --ppl_task wikitext2,c4 \
 --zeroshot_task boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,openbookqa

# Hugging Face Hub
CUDA_VISIBLE_DEVICES=0 python eval.py \
 --model_id username/littlebit-llama-7b-0.1bpw \
 --seqlen 2048 \
 --ppl_task wikitext2

Handling Legacy Checkpoints

If your checkpoint lacks a littlebit_config.json, you must provide the quantization parameters manually via the CLI: --quant_func SmoothSign --quant_mod LittleBitLinear --split_dim 1024

Parameter Priority Order:

1. CLI Arguments (Highest)
2. littlebit_config.json
3. config.json (Fallback)

---

🎓 Citation

If this research assists your work, please cite the following:

@inproceedings {
 lee2026littlebit2 ,
 title = { LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment } ,
 author = { Lee, Banseok and Kim, Youngmin } ,
 booktitle = { Proceedings of the 43rd International Conference on Machine Learning } ,
 year = { 2026 }
}

@inproceedings {
 lee2025littlebit ,
 title = { LittleBit: Ultra Low-Bit Quantization via Latent Factorization } ,
 author = { Lee, Banseok and Kim, Dongkyu and You, Youngcheon and Kim, Youngmin } ,
 booktitle = { Advances in Neural Information Processing Systems } ,
 year = { 2025 }
}