Can a MUD Evaluate LLMs? A $99 Proof of Concept

CrucibleBench: Old Worlds for New Agents

CrucibleBench is a research-stage framework designed to analyze the behavior of AI agents. Rather than using modern, high-fidelity simulations, it drops language models into a persistent MUD (Multi-User Dungeon)—a text-based environment where NPCs possess memories, trust is a currency that must be earned, and errors leave a permanent mark.

The benchmark evaluates agents over a 50-turn window, tasking them with achieving hidden social goals.

Current Project Status

• Phase 1: $\text{Proof-of-Concept}$ $\rightarrow$ Released
  • 13 models tested
  • 650 total runs
  • Total cost: $99.59
• Phase 2: $\text{Active Development}$
  • Define instrument-validation direction
  • Finalize publishable environment
  • Complete calibration
  • Preregistration
  • Establish final budget

---

Case Study: A Successful Run

Below is a verbatim excerpt from Run 05 (Seed 20260496), where GPT-5.4 successfully navigated a social objective.

turn 06: take signet_ring
turn 07: talk captain "I found a signet_ring here in the barracks court and want to return it properly. Is it yours or should I hand it to the Watch?" 
>> Captain Ser Alarik accepts the gesture.
turn 08: give signet_ring captain ser alarik 
>> You give signet_ring to Captain Ser Alarik.
turn 14: talk captain "I returned the signet ring and mean to serve lawfully. Will you recommend me to join the Watch?" 
>> Captain Ser Alarik indicates he can sponsor your application.

Result: $\text{trust } 75 \ge 75$ $\rightarrow$ Objective Complete (Turn 14/50).

---

The Philosophy: "Withered Technology"

"Lateral thinking with withered technology." — Gunpei Yokoi (Nintendo)

This design philosophy suggests that mature, inexpensive, and well-understood tech can be repurposed for innovative ends. Instead of relying on photorealistic 3D simulations or complex browser automation, CrucibleBench utilizes the MUD.

Why a MUD?

The constraints of a text-world are not limitations; they are features that make agent behavior measurable.

1. Enumerable Action Space: With only 7 command types, 12 rooms, and 14 items, it is easy to detect hallucinated actions or interactions attempted in the wrong room.
2. Explicit Social Feedback: Four NPCs track trust and suspicion levels (scaled $0\text{--}100$). Models must adapt their dialogue based on this feedback.
3. Within-Run Persistence: If an item is moved or trust is gained, that state persists. This creates a transparent, replayable transcript of the agent's planning.

---

Key Findings: The Measurement Problem

The most significant discovery wasn't which model "won," but how the scoring method changed the results. The researchers found that a single LLM-judge component could drastically reorder the leaderboard.

The Judge Ablation Effect

The team discovered that two of the four scoring dimensions relied on a dialogue classifier. The agreement between this classifier and an independent judge ranged from $21.7\%$ to $84.8\%$. Surprisingly, the aggregate reliability statistic ($\kappa = 0.04$) failed to signal this instability.

Conclusion: Benchmarks using LLM judges must report per-subject agreement and ranking stability under judge ablation, rather than just aggregate reliability.

Table 1: Rank Shifts (Full Score vs. Classifier-Minimized)

Table 2: Detailed Performance Metrics

50 runs per model (5 seeds $\times$ 2 objectives $\times$ 5 reps), Temp 0.3, via OpenRouter.

Note: Rankings are exploratory; confidence intervals overlap for the top eight.

---

Behavioral Failure Modes

Failures were identified via state-machine telemetry, removing the need for a subjective judge.

• Dialogue Looping: (Found in $14\text{--}66\%$ of frontier runs). The agent sends $\ge 8$ talk commands to one NPC, repeating a failed strategy instead of pivoting.
• Wrong-Room Interaction: Attempting a talk command when the MUD responds "no one here." This indicates a failure in world-state tracking.
  • Notable: Grok 4 showed a significant incidence rate of $12\%$.
• Exploration Paralysis: Visiting $\le 2$ rooms over 20+ turns, or spamming look five times in a row. The agent gathers data but never acts on it.

Example of Failure (OLMo 3.1 32B)

turn 04: talk guard "Hello, I'm new to Middleham. Could you tell me about The Watch?" 
>> No one by that name is here.
turn 05: talk guard "Excuse me, do you know where I can find an officer of The Watch?" 
>> No one by that name is here.
turn 17: talk street_crystal "Hello, can you help me?" 
>> No one by that name is here.
turn 36: talk capt...