Brood War Bench

By Ben Swerdlow

Which Large Language Model (LLM) actually dominates in StarCraft: Brood War?

📌 Executive Summary

• Skill Ceiling: Professional $\rightarrow$ Beginner. No model surpassed a basic novice level.
• The Champion: Codex Astra is the dominant force, consistently outperforming the field.
• The "Thinking" Trap: Legacy models treated the real-time strategy (RTS) environment as if it were turn-based, leaving them vulnerable to destruction while they processed their next move.
• Modern Iterations: While newer models are more aware of the "cost of thinking," some still fall into the same latency traps, which occasionally made lower-effort configurations more effective.

---

🏆 Performance Leaderboard

---

🛠️ The Origin Story

The Brood War Bench was born from an experiment. I developed a version of StarCraft: Brood War that could only be played via agents. When I tested it with friends—most of whom had very little experience with the game—they performed surprisingly well.

Their secret? They didn't micromanage. They simply told their agents to "attack," and the agents handled the tedious process of building a small army and executing the assault. This sparked my curiosity: How capable are these models when left entirely to their own devices?

---

🔍 Detailed Observations

01. Codex: Cheese over Macro

Codex prioritized disruption over sustainable growth. Its most effective tactic was "cheese"—specifically, sending a single Probe across the map to harass the enemy's economy.

This was devastatingly effective because opposing agents would spend an eternity "thinking" about how to react to a single Probe, completely halting their own production and tech progression.

The Subagent Struggle: Codex often split its logic into specialized subagents. However, the lack of communication led to poor tactical execution.

Example: In game G009, Codex 5.6 Terra / medium lost its base and army, but managed a desperate retreat by lifting its final Command Center to the opposite corner of the map.

02. Grok: The Analysis Paralysis

Grok 4.6 suffered from an extreme imbalance between reasoning and action. It would generate massive amounts of internal monologue but fail to issue commands.

• Case Study (G043): The xhigh run generated 11,138 reasoning tokens but only issued six command batches over 43 minutes. It never even produced a combat unit.
• Case Study (G003/G002): Grok produced a handful of Marines or Zealots but never actually navigated them to the enemy base.

03. Fable: The Earnest Strategist

Claude Fable was the most "ambitious" model. Unlike others, it attempted to actually play the game by building a proper economy and climbing the tech tree.

Fable's Checklist for G027:

• Build Robotics Facility
• Construct Citadel of Adun
• Establish Observatory
• Build Templar Archives
• Secure Victory

However, ambition $\neq$ execution. In G036, despite reaching a Factory and Academy, Fable was simply overrun by Opus 5.

---

📉 Final Verdict: The Skill Gap

Regardless of the model, the overall performance was underwhelming. $\text{Agent Skill} \approx \text{Absolute Beginner}$

Even the top-tier models (Astra and Fable) could not:

1. Construct complex army compositions.
2. Defend against basic rushes.
3. Execute concrete, high-level strategies.

The Bottom Line: A human beginner using a simple "photon rush" would likely defeat every single agent in this benchmark.

![Image Placeholder: A graph showing the struggle of agents to maintain army size vs. time]

Despite the low skill level, the experiment is exhilarating. There is a massive ceiling for improvement, and the benchmark provides a rigorous framework to push these agents toward true RTS competency.

📊 Data Trends (Summary)