RuneScape Bench: Evaluating Autonomous Agents in MMO Environments

The quest for Artificial General Intelligence (AGI) requires testing agents in environments that mirror the complexity of the real world. While traditional benchmarks focus on narrow tasks, RuneScape Bench introduces a massive, open-world Massively Multiplayer Online (MMO) game as the ultimate proving ground for LLM-based agents.

Why an MMO?

Most current agent benchmarks are too simple limited to static environments or basic web navigation. RuneScape provides a unique set of challenges:

• Long-Horizon Planning: Tasks can take hours or days of simulated time.
• Complex Dependencies: To achieve Goal C, an agent must first master Skill A and gather Resource B.
• Dynamic Economy: The world state changes based on player (and agent) interaction.
• Multimodal Input: Agents must process text, spatial data, and game state.

Comparison of Agent Benchmarks

---

The Technical Architecture

The framework acts as a wrapper around the game, allowing an LLM to perceive the world and execute actions via a structured API.

The Agent-Environment Loop

The agent interacts using inline_code commands such as move_to(location) or interact(object). To manage the vast state space, the system utilizes a memory buffer to track inventory and quest progress.

Implementation Example

Below is a conceptual representation of how an agent might process a task:

def agent_loop(goal):
    while not goal.is_completed():
        observation = game.get_current_state()
        # The LLM decides the next best action based on the goal
        action = llm.predict_action(observation, goal)
        result = game.execute(action)
        update_memory(result)

---

Task Hierarchy and Evaluation

RuneScape Bench doesn't just ask agents to "play the game"; it assigns specific, tiered objectives.

Agent Requirements Checklist

• Ability to parse game logs
• Long-term memory retention
• Strategic resource management
• Perfect execution of high-level combat (Current Limitation)

"The transition from narrow task-solving to open-world agency represents a paradigm shift in how we evaluate the 'reasoning' capabilities of Large Language Models."

Measuring Success

Success is not binary. The researchers use a weighted formula to determine the efficiency of an agent, considering the time taken and the number of errors made.

The success rate $\mathcal{S}$ can be represented as: $\mathcal{S} = \frac{\sum_{i=1}^{n} \text{CompletedTasks}_i}{n \times \text{EfficiencyFactor}}$

---

Key Challenges & Nuances

The environment exposes several critical weaknesses in current LLMs:

1. The "Hallucination" Gap: Agents often attempt to use items they do not possess.
2. Spatial Reasoning: Understanding the map requires more than just text; it requires a conceptual grasp of coordinates.
3. State Drift: Over long sequences, agents may forget the original objective.

Final Thoughts

RuneScape Bench moves the needle from chatbots to actors. By forcing agents to navigate a world with strict rules, persistent consequences, and intricate goals, it provides a rigorous stress test for the next generation of autonomous AI.