GigaToken: Achieving $\approx 1000\times$ Acceleration in LM Tokenization

GigaToken is a high-performance, drop-in replacement for standard language model tokenizers, designed to push tokenization speeds into the GB/s range.

Important Context: It is worth noting that both tiktoken and HuggingFace (HF) tokenizers are already implemented in multithreaded Rust. GigaToken manages to significantly outperform them regardless.

The library is compatible with a vast array of CPU architectures and supports nearly every widely-used tokenizer currently in existence.

🚀 Getting Started

Installation

To integrate GigaToken into your environment, simply run: pip install gigatoken

Setup Checklist

Install Python 3.x
Run pip install gigatoken
Identify your target model (e.g., Llama 3, GPT-2, Qwen)

🛠 Usage Patterns

GigaToken offers two distinct ways to operate: a Compatibility Mode for ease of integration and a Native API for maximum throughput.

1. Compatibility Mode (The "Easy" Path)

This mode allows you to swap your existing tokenizer with GigaToken with minimal code changes.

import gigatoken as gt

# For HuggingFace users
hf_tokenizer = ... 
tokenizer = hf_tokenizer.as_hf() 
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

# For tiktoken users
tiktokenizer = ...
tokenizer = tiktokenizer.as_tiktoken()
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

Nuance Note: While GigaToken strives for exact output parity with HF tokenizers in this mode, achieving this precision comes at a ~~zero~~ non-negligible performance cost. You will still see massive gains, but you won't hit the peak $1000\times$ speedup.

2. GigaToken Native API (The "Fastest" Path)

To unlock the full potential of the Rust backend, use the native API. This allows the system to read data directly from disk, bypassing Python's overhead and maximizing parallelism.

import gigatoken as gt

# Initialize using an HF model identifier
tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")

# Define a file source with a specific separator
file_source = gt.TextFileSource(["owt_train.txt"], separator=b"|endoftext|")

# Execute high-speed encoding
tokens = tokenizer.encode_files(file_source)

Data Flow Comparison

📊 Performance Benchmarks

The following tests were conducted using the owt_train.txt dataset (size: $11.9\text{ GB}$ ).

Hardware: AMD EPYC 9565 (144 Cores)

Tokenizer	GigaToken	HF Tokenizers	tiktoken	vs HF	vs tiktoken
GPT-2	$24.53\text{ GB/s}$	$24.8\text{ MB/s}$	$36.0\text{ MB/s}$	$989\times$	$681\times$
Phi-4	$24.00\text{ GB/s}$	$29.9\text{ MB/s}$	—	$801\times$	—
GPT-OSS	$23.96\text{ GB/s}$	$49.7\text{ MB/s}$	$42.8\text{ MB/s}$	$482\times$	$560\times$
OLMo 2/3	$23.06\text{ GB/s}$	$27.7\text{ MB/s}$	—	$833\times$	—
Nemotron 3	$22.79\text{ GB/s}$	$49.4\text{ MB/s}$	—	$462\times$	—
Qwen 3	$22.16\text{ GB/s}$	$34.2\text{ MB/s}$	—	$648\times$	—
Llama 3/3.1/3.2	$22.15\text{ GB/s}$	$48.5\text{ MB/s}$	—	$457\times$	—
GLM 5	$20.97\text{ GB/s}$	$74.8\text{ MB/s}$	—	$280\times$	—
Llama 3.3	$20.82\text{ GB/s}$	$48.3\text{ MB/s}$	—	$431\times$	—
Llama 4	$20.77\text{ GB/s}$	$72.7\text{ MB/s}$	—	$286\times$	—
GLM 4	$20.61\text{ GB/s}$	$72.3\text{ MB/s}$	—	$285\times$	—
Phi-4-mini	$20.05\text{ GB/s}$	$27.6\text{ MB/s}$	—	$726\times$	—
DeepSeek V3/R1/V4	$19.69\text{ GB/s}$	$26.2\text{ MB/s}$	—	$750\times$	—
Qwen 2/2.5	$19.12\text{ GB/s}$	$27.7\text{ MB/s}$	—	$691\times$	—
Kimi K2	$18.85\text{ GB/s}$	—	—	—	—
Qwen 3.5/3.6	$15.49\text{ GB/s}$	$27.7\text{ MB/s}$	—	$558\times$	—
Gemma 4	$4.82\text{ GB/s}$	$334.1\text{ MB/s}$	—	$14\times$	—
ModernBERT	$4.18\text{ GB/s}$	$26.9\text{ MB/s}$	—	$155\times$	—
Mistral 7B v0.3	$3.57\text{ GB/s}$	$354.7\text{ MB/s}$	—	$10\times$	—
TinyLlama/Phi-3	$3.48\text{ GB/s}$	$323.6\text{ MB/s}$	—	$11\times$	—
CodeLlama	$3.47\text{ GB/s}$	$347.4\text{ MB/s}$	—	$10.0\times$	—
Gemma 3	$3.43\text{ GB/s}$	$357.2\text{ MB/s}$	—	$9.6\times$	—
Gemma 1	$2.51\text{ GB/s}$	$342.2\text{ MB/s}$	—	$7.3\times$	—

Hardware: Apple M4 Max (16 Cores)

Tokenizer	GigaToken	HF Tokenizers	tiktoken	vs HF	vs tiktoken
GPT-2	$8.79\text{ GB/s}$	$6.9\text{ MB/s}$	$62.8\text{ MB/s}$	$1,268\times$	$140\times$
Nemotron 3	$7.82\text{ GB/s}$	$10.9\text{ MB/s}$	—	$715\times$	—
Phi-4	$7.76\text{ GB/s}$	$7.7\text{ MB/s}$	—	$1,012\times$	—
Llama 3/3.1/3.2	$7.60\text{ GB/s}$	$11.2\text{ MB/s}$	—	$676\times$	—
OLMo 2/3	$7.56\text{ GB/s}$	$5.8\text{ MB/s}$	—	$1,299\times$	—
Llama 3.3	$7.50\text{ GB/s}$	$15.7\text{ MB/s}$	—	$479\times$	—
Phi-4-mini	$6.97\text{ GB/s}$	$7.2\text{ MB/s}$	—	$964\times$	—
Kimi K2	$6.88\text{ GB/s}$	—	—	—	—
Llama 4	$6.81\text{ GB/s}$	$11.6\text{ MB/s}$	—	$590\times$	—
Qwen 2/2.5	$6.37\text{ GB/s}$	$5.8\text{ MB/s}$	—	$1,105\times$	—
Qwen 3	$6.36\text{ GB/s}$	$6.9\text{ MB/s}$	—	$918\times$	—
Qwen 3.5/3.6	$6.31\text{ GB/s}$	$6.3\text{ MB/s}$	—	$994\times$	—
GPT-OSS	$6.20\text{ GB/s}$	$20.2\text{ MB/s}$	$87.2\text{ MB/s}$	$306\times$	$71\times$
GLM 4	$6.17\text{ GB/s}$	$15.8\text{ MB/s}$	—	$392\times$	—
DeepSeek V3/R1/V4	$5.68\text{ GB/s}$	$7.2\text{ MB/s}$	—	$788\times$	—
GLM 5	$5.55\text{ GB/s}$	$12.2\text{ MB/s}$	—	$456\times$	—
ModernBERT	$2.64\text{ GB/s}$	$5.8\text{ MB/s}$	—	$452\times$	—
Mistral 7B v0.3	$1.99\text{ GB/s}$	$95.1\text{ MB/s}$	—	$21\times$	—
Gemma 4	$1.82\text{ GB/s}$	$85.2\text{ MB/s}$	—	$21\times$	—
CodeLlama	$1.73\text{ GB/s}$	$80.2\text{ MB/s}$	—	$22\times$	—
TinyLlama/Phi-3	$1.69\text{ GB/s}$	$80.1\text{ MB/s}$	—	$21\times$	—
Gemma 1	$1.42\text{ GB/s}$	$85.7\text{ MB/s}$	—	$17\times$	—
Gemma 3	$1.38\text{ GB/s}$	$82.2\text{ MB/s}$	—	$17\times$	—

Hardware: AMD Ryzen 7 9800X3D (16 Cores)

| Tokenizer | GigaToken |

GigaToken: Achieving ≈1000×\approx 1000\times≈1000× Acceleration in LM Tokenization