Achieving 5 tokens/sec with Gemma 4 26B on a Legacy 13-Year-Old Xeon (GPU-less)

By Ryan Findley | June 8, 2026

I have a server in my basement that, by all modern standards, should be incapable of running a contemporary Large Language Model (LLM). It is a repurposed HP StoreVirtual storage appliance—roughly thirteen years old—powered by dual Ivy Bridge Xeons and devoid of any GPU acceleration.

Yet, as of this week, this machine is successfully running Gemma 4, Google's 26-billion-parameter open-weights Mixture-of-Experts (MoE) model, at a rate of approximately five tokens per second.

🛠️ The Hardware Profile

The machine is a relic of the enterprise world, costing less than $300.

Performance Metrics:

• Decode Speed: $\approx 5.2$ tokens/sec
• Prompt Evaluation: $\approx 16$ tokens/sec

---

The Philosophy of "AI Competence"

Anyone with a credit card can rent an H100 or pay for a monthly subscription. However, I believe there is a different, more fundamental skill set involved in AI.

"Good with AI" has quietly come to mean "pays for a subscription." I think the real skill is different: knowing a model well enough to point it at a problem nobody packaged for you, and telling whether the answer it hands back is actually correct.

Rather than simply claiming this proficiency, I decided to provide a practical demonstration using hardware that had no business cooperating with a 2026-era model.

The Catalyst: A Hacker News Spark

This project was inspired by a recent viral post on Hacker News titled "A 10 year old Xeon is all you need." The author managed to run Gemma 4 on a 2016 Xeon (no GPU) with 128GB of DDR3 using a specific fork called ik_llama.cpp and a precise set of configuration flags.

That setup utilized several high-end inference optimizations:

• Speculative decoding
• CPU-optimized MoE routing
• CPU-ported Flash Attention
• Dynamic weight repacking

The Compatibility Wall

When I attempted the build, it crashed immediately. My CPUs are Ivy Bridge (v2), whereas the optimized kernels in that fork assume Haswell (v3) instructions—specifically AVX2 and FMA3.

The problem was simple: My silicon was older than the instructions the code was written to use.

The Human-AI Collaboration

I had attempted a fix previously that almost worked, but I couldn't quite cross the finish line. I handed the half-finished approach to Claude, which analyzed the logic, validated the direction, and completed the implementation.

It wasn't a simple "fix this" prompt. The process required:

• Analyzing performance-critical C++ code.
• Identifying why specific kernels failed on the Ivy Bridge microarchitecture.
• Routing around those instructions without sacrificing the core optimizations of the fork.

My role was that of the orchestrator: running the experiments and verifying the validity of the output.

The Result

Gemma 4 26B now generates text at a human reading speed on hardware that was retired before the model's architecture was even conceived. For those interested, the patch is available at ikawrakow/ik_llama.cpp#2138 (currently awaiting review; use the branch for now).

---

🔍 Technical Deep Dive: The Bug

For the engineers and the Hacker News crowd, here is the actual technical breakdown. I will be honest: I can read a stack trace, but I didn't hand-write the quantized matmul kernel fallbacks. The Claude instance running on the server itself diagnosed and patched the bug.

The Root Cause

The target engine was ik_llama.cpp. When you disable GGML_USE_IQK_MULMAT during the build, most of the code falls back to standard scalar/SSE math. However, the Gemma 4 MoE feed-forward network (FFN) utilizes two specific operations:

1. MOE_FUSED_UP_GATE (Fused gate+up matmul with SwiGLU for experts)
2. FUSED_UP_GATE (The dense version)

While these were gated by #if macros in the compute dispatcher, the graph builder continued to emit them regardless of the build flag.

The Symptom

Because the dispatcher had no case for these op enums when the flag was off, the destination tensors for every expert FFN were never computed.

With 30 layers and 8 active experts per token, the model was essentially processing $\approx 240$ tensors of random memory junk per forward pass. This resulted in:

• Deterministic output (at temperature 0).
• No NaN values.
• Complete gibberish (mixing Thai, Korean, and English fragments).

The Diagnosis

Claude instrumented the raw logits before sampling. It observed a mean logit of: $\mu \approx +16$ ...when it should have been near $0$. Roughly $80\%$ of the vocabulary had positive logits. This is a classic sign of uninitialized memory containing small positive floats.

By implementing the proper scalar fallbacks for these fused operations, the hidden state is now correctly computed, allowing the model to function perfectly on legacy silicon.