Tokens Too Cheap to Meter

Published: 2026-09-16 • Category: LLMs & Economics

Executive Summary: Is This Actually Happening?

The financial cost of utilizing machine learning intelligence is plummeting by several orders of magnitude every year, with no indication of a plateau. We are transitioning into an era where the "cost per token" becomes negligible.

The Roadmap of Integration

• Short Term (1-2 Years): LLMs will shift from being standalone products to becoming fundamental computing infrastructure integrated into every layer of software.
• Medium Term (3-6 Years): Frontier-level model quality will be available to run locally on standard consumer hardware.
• The New Constraint: Very soon, the limiting factor for AI adoption will be the number of tokens be the quality of the output and the ease of access.

"Extraordinary claims require extraordinary evidence, so I collected a whole bunch of evidence."

---

The Landscape of AI Models

To understand the economics, we must distinguish between the types of models available:

Note: Improvements in one category do not always translate to the others.

---

1. Improvements Affecting All AI

Hardware: The GPU Evolution

GPU efficiency is increasing exponentially. When analyzing the power efficiency of GPUs over time, we see a trend reminiscent of the 1960s Moore's Law.

Using a logarithmic scale, the efficiency growth can be expressed as: $\text{Efficiency} \propto 10^{1.3t}$ This implies that efficiency roughly doubles every two years.

Model Economics & The Pareto Frontier

A token is a sub-word fragment (roughly $1 \text{ word} \approx 1.5 \text{ tokens}$). Providers charge for both input and output tokens. While the cost of the absolute "smartest" models doesn't always drop, the cost to complete a specific task is crashing.

The Pareto Frontier

A Pareto frontier represents the optimal tradeoff between two variables—in this case, Intelligence (Quality) vs. Cost.

• High End: Claude Fable-5.1 (Extreme intelligence, higher cost).
• Mid/Low End: GPT-5.6 Luna (Lower intelligence, very cheap).

The 2025 Shift: Comparing the start of 2025 to the end of the year, the frontier shifted dramatically:

1. Horizontal View: For any fixed level of intelligence, the cost dropped.
2. Vertical View: For any fixed budget, the intelligence increased.
3. Scale: The X-axis (cost) decreased by two orders of magnitude.

Inference Engines

The software that executes the model on the GPU (the "inference engine") is currently in a state of rapid maturation. We are seeing annual efficiency gains of $10\% \text{ to } 50\%$.

Serving vs. Offline:

• Offline: Processing large batches of tokens.
• Serving: Handling unpredictable, real-time user requests.
• Trend: Serving efficiency is improving much faster than offline efficiency.

Case Study: vLLM

The open-source engine vLLM demonstrates this trajectory. Comparing version 0.5.4 (Sept 2024) to 0.11.1 (Dec 2025), there was a 40% increase in energy efficiency (measured in Joules/token).

Other industry benchmarks confirm this:

• NVIDIA: Up to 50% efficiency gains moving from MLPerf 2.0 $\rightarrow$ 2.1.
• Intel: A $2.4\times$ increase in throughput between MLPerf 6.0 and 6.1 using the same hardware.

---

2. Improvements Specific to Hosted AI

The Architecture Shift: Mixture-of-Experts (MoE)

Early LLMs were "dense," meaning every single parameter was activated for every single token processed. This was computationally expensive.

Modern hosted models use Mixture-of-Experts (MoE). Instead of activating the whole brain, the model only activates specialized "expert" pathways relevant to the specific input.

# Conceptual MoE Logic
def moe_layer(input_token):
    expert = router.select_expert(input_token) 
    return expert.process(input_token) 
    # Only one 'expert' runs, not the entire model!

This architectural shift allows models to have massive total parameter counts (increasing intelligence) while keeping the active parameter count low (decreasing cost and latency).

---

(The original text ends abruptly at the explanation of MoE; this rewrite preserves that cutoff point while maintaining the requested structure and detail.)