Ferrox: Engineering a Rust-Native Alternative to llama.cpp

By Antonello Fratepietro | August 5, 2026

Over the past few days, I have been developing Ferrox, a high-performance inference engine written entirely in Rust. It is designed to run open-source Large Language Models (LLMs)—including both dense architectures and Mixture-of-Experts (MoE)—locally across CPU, CUDA, and Apple Metal backends.

To be clear: this is not a wrapper. There are no bindings to ggml or llama.cpp. Every single component—from the loaders and scheduling logic to the low-level kernels—was authored from the ground up.

The Motivation: Why build this when llama.cpp is already a gold standard? I wanted to move beyond simply "running a binary." I sought a deep, visceral understanding of inference and a project where performance wasn't just a claim, but a metric earned against a rigorous, industry-standard baseline.

---

🛠️ What is Ferrox?

At its core, Ferrox is a runtime that consumes GGUF files (the same quantized format used by llama.cpp) to execute model inference. It provides two primary interfaces:

1. The CLI (ferrox): A command-line tool utilizing flags compatible with llama.cpp.
2. The Server (ferrox-server): A backend that implements the OpenAI chat-completions API.

Because it mimics the OpenAI API, any existing tool—whether it's a custom chat UI, an autonomous agent framework, or a testing suite—can be pointed at localhost and work immediately.

The Technical Secret Sauce

Ferrox employs a memory-mapping strategy. Weights are mapped directly from the disk and are never fully expanded into RAM. Instead, dequantization is performed as a fused operation during the dot product calculation.

Mathematically, the operation looks something like this: $\text{Output} = \sum_{i=1}^{n} (\text{dequantize}(w_i, \text{scale}, \text{offset}) \cdot x_i)$

This approach allows a laptop with limited RAM to run an 8B-parameter model without requiring 30GB of memory.

System Architecture Flow

---

🚀 Getting Started

Ferrox does not come with weights. You must provide your own .gguf files. I recommend using the huggingface_hub Python library: pip install -U huggingface_hub.

Installation Checklist

• Clone the repository: git clone https://github.com/antonellof/ferrox.git
• Navigate to directory: cd ferrox
• Build the project: cargo build --release -p ferrox-cli -p ferrox-server --features metal
• Create model directory: mkdir -p models

Recommended Models for Testing

Pro tip: Use Q4_K_M for general use and Q8_0 for quick verification tests.

Execution Examples

1. Simple One-Shot Completion

./target/release/ferrox -m models/tinyllama-1.1b-chat-v1.0.Q8_0.gguf \
-p "The capital of France is" -n 32 --temp 0 --no-cnv

2. Interactive Chat (Metal Backend)

./target/release/ferrox -m models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
-p "What is 2+2?" -n 64 --temp 0 -dev metal -ngl all

3. OpenAI-Compatible Server

./target/release/ferrox-server \
-m models/tinyllama-1.1b-chat-v1.0.Q8_0.gguf \
--host 127.0.0.1 --port 8383 -dev metal -ngl all

Test the server with curl:

curl -s -X POST http://127.0.0.1:8383/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"m","messages":[{"role":"user","content":"Hi"}],"max_tokens":32,"temperature":0}'

Note on Flags: Most flags (like -m, -p, -n, -t, --temp, -ngl) mirror llama.cpp. See docs/CLI.md for the full list.

Special Feature: You can use --ctk q8_0 (or the environment variable FERROX_CTK=q8_0) to store the KV cache as Q8_0 on Metal. This reduces the memory footprint for long contexts at the cost of slight precision loss.

---

📈 Provable Performance

In a field where "fast" is a subjective claim, I wanted Ferrox's benchmarks to be falsifiable. The testing suite ensures that both engines run on the same hardware, using the same GGUF and backend, back-to-back.

Every result is backed by a JSON "receipt" in the repository. If you regenerate the test, the numbers must match.

Benchmarks: Apple M2 Pro (Host B)

The following are predicted tokens per second (tok/s) from the fair-chat suite. Gap = llama.cpp / Ferrox (Values < 1.0 indicate Ferrox is faster).

The most significant milestone for me was the Llama-3.1-8B result. Achieving $\sim 0.97\times$ (28.27 vs 27.55 pred) meant that Ferrox was no longer just an academic exercise; it was a viable engine.

The MoE Challenge

The biggest leap recently has been with OLMoE on Metal. Initial tests saw Ferrox trailing by $15\times$ ($\sim 10$ tok/s). After optimizing expert placement on the Metal backend, it now hits 88.4 tok/s. While it still trails llama.cpp (156.8 tok/s), the performance is now in a completely different league.

How to Verify

To regenerate these receipts on your own machine:

# Standard suite
python3 benchmarks/run_suite.py --skip-missing --fit-host

# CLI-specific pins
python3 benchmarks/run_suite.py --skip-missing --fit-host --mode cli

This will update the files in benchmarks/receipts/pins/ and refresh RESULTS.md via render_results.py.