Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

Authors: Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, and Christopher Ré.

---

📌 Executive Summary

The current landscape of Large Language Model (LLM) processing is heavily skewed toward centralized cloud infrastructure. However, the explosive growth in demand is outpacing the ability of providers to scale their hardware. This paper explores a paradigm shift toward local inference, enabled by two primary catalysts:

1. The emergence of high-performing small LMs (specifically those with ≤20B active parameters).
2. The development of powerful local accelerators (such as the Apple M4 Max) capable of delivering interactive response times.

Core Research Question: Is it viable to redistribute the computational burden from centralized clouds to local devices by leveraging efficient local inference?

To answer this, the researchers introduce a new benchmark metric: Intelligence per Watt (IPW).

---

📐 The Metric: Intelligence per Watt

The authors define IPW as a unified measure of both the capability and energy efficiency of a specific model-accelerator pairing. Mathematically, it is represented as:

$\text{IPW} = \frac{\text{Task Accuracy}}{\text{Unit of Power}}$

Evaluation Workflow

The study followed a rigorous testing pipeline:

---

🧪 Methodology & Scope

The researchers conducted a massive evaluation to ensure statistical significance:

• Models: Over 20 state-of-the-art local LMs.
• Hardware: 8 different accelerators, spanning both local devices and cloud-based systems.
• Dataset: 1 million single-turn queries focusing on reasoning and general chat.

Experimental Checklist

• Compare local LM win rates against frontier models.
• Quantify energy expenditure per query.
• Track latency for interactive viability.
• Perform longitudinal analysis (2023–2025).

---

📊 Key Findings

1. Capability and Coverage

Local LMs are surprisingly capable, successfully handling 88.7% of real-world queries, though performance varies depending on the specific domain.

2. Longitudinal Growth (2023 $\rightarrow$ 2025)

The efficiency of local AI has seen a dramatic upward trajectory:

3. Local vs. Cloud Efficiency

Interestingly, local accelerators demonstrated a significant advantage over cloud accelerators when running the same models.

• Result: Local accelerators achieved at least 1.4x lower IPW (higher efficiency) than their cloud counterparts.
• Implication: There is substantial room for further optimization in local hardware.

---

💻 Technical Implementation Note

While the paper focuses on metrics, the logic for calculating the efficiency can be conceptualized in the following pseudo-code:

def calculate_ipw(accuracy_score, power_consumption_watts):
    """
    Calculates Intelligence per Watt (IPW).
    accuracy_score: Win rate against frontier models (0.0 to 1.0)
    power_consumption_watts: Average power used during inference
    """
    if power_consumption_watts == 0:
        return 0
    return accuracy_score / power_consumption_watts

🏁 Conclusion

The findings suggest that a substantial portion of AI demand can be shifted away from the cloud. By utilizing IPW as the primary metric, developers and hardware engineers can better track the transition toward a more sustainable, distributed AI ecosystem.