FrontierHarness Eval: Analyzing Cost and Performance Across 12 Harness Configurations

The FrontierHarness Eval provides a comprehensive look at how different agent harnesses impact performance when using the exact same underlying model and runtime. The most striking finding? Even with identical tasks and models, the cost per pass can vary by as much as 17x.

🛠 Methodology & Rigor

To ensure a neutral evaluation with no "home-field advantage," the team utilized Runta to execute every trial. The goal was to eliminate any variables other than the harness itself.

The Gold Standard of Testing: Every single one of the 360 trials began with a fresh restore from a "golden checkpoint." This ensured an identical cold start for every run, meaning vCPU, memory, disk size, disk contents, and memory state were perfectly synchronized.

Execution Flow

To prevent warm-cache bias, formal tasks were never executed prematurely.

---

📊 Comparative Performance Data

The evaluation tested 9 primary harnesses across 12 different configurations. All were powered by the Kimi K3 model.

🏆 The Category Leaders

• Quality Leader: Codex (66.7% pass rate $\cdot$ $3.47/task)
• Balanced Pick: Pi (60.0% pass rate $\cdot$ $2.43/task)
• Cost Leader: Exo Harness ($1.05/task$\cdot$ 53.3% pass rate)
• Speed Leader: DSH Minimal (5m 41s median runtime $\cdot$ 56.7% pass rate)

---

📈 Detailed Rankings

1. Pass Rate (Success Probability)

• 01 Codex: 66.7%
• 02 Claude Code: 63.3%
• 03 DSH Creator: 63.3%
• 04 DSH PTC: 60.0%
• 05 DSH Standard: 60.0%
• 06 Pi: 60.0%
• 07 DSH Minimal: 56.7%
• 08 Kimi Code: 56.7%
• 09 Oh My Pi: 56.7%
• 10 Exo Harness: 53.3%
• 11 Hermes: 50.0%
• 12 OpenCode: 50.0%

2. Efficiency Metrics

The following lists rank the harnesses by median values:

Median Cost per Successful Task

1. OpenCode: $0.0615
2. Pi: $0.0709
3. Exo Harness: $0.0748
4. DSH Creator: $0.1194
5. DSH Standard: $0.1201
6. DSH Minimal: $0.1214
7. Codex: $0.1243
8. Oh My Pi: $0.1354
9. DSH PTC: $0.1370
10. Hermes: $0.1746
11. Kimi Code: $0.1818
12. Claude Code: $0.2880

Median Cost per Task (Overall)

1. Exo Harness: $1.0452
2. Pi: $2.4330
3. Hermes: $2.9043
4. OpenCode: $3.2443
5. DSH Creator: $3.2849
6. DSH Standard: $3.4589
7. Codex: $3.4683
8. Kimi Code: $3.6471
9. DSH PTC: $4.5774
10. DSH Minimal: $4.7180
11. Oh My Pi: $4.7455
12. Claude Code: $18.3368

Median Cache Hit Rate (Successful Tasks)

1. Codex/Kimi Code: 88.0%
2. DSH PTC: 87.2%
3. DSH Standard: 86.5%
4. Hermes: 85.9%
5. DSH Minimal: 84.6%
6. DSH Creator: 84.3%
7. Oh My Pi: 82.2%
8. Pi: 79.4%
9. OpenCode: 78.4%
10. Exo Harness: 70.3%
11. Claude Code: 67.8%

Median Time per Successful Task

1. DSH Minimal: 5m 41s
2. DSH Standard: 6m 17s
3. Exo Harness: 6m 17s
4. OpenCode: 6m 27s
5. Codex: 6m 43s
6. DSH Creator: 6m 44s
7. Oh My Pi: 6m 46s
8. Hermes: 6m 58s
9. Pi: 7m 33s
10. DSH PTC: 7m 44s
11. Kimi Code: 7m 56s
12. Claude Code: 9m 38s

---

🔍 Beyond the Numbers

There are several nuances that the raw tables don't fully capture:

• The OpenCode Paradox: Failures were initially excluded. When you account for failed attempts, the cost for OpenCode rises to $3.24 per task. This highlights a critical point: a cached failure spanning 300 turns can be more expensive than a short cache miss.
• The Claude Code Premium: While Claude Code is highly effective (passing 19 tasks), it is an outlier in spending, reaching a staggering $18.34 per task.

🛠 Versioning Reference

Codex: v0.148.0
DeepSeek Harness: v0.1.0-rc.8
Claude Code: v2.1.237
Pi: v0.84.2
Oh My Pi: v17.4.0
Kimi Code: v0.37.2
Exo Harness: v0.1.0
OpenCode: v1.18.19
Hermes: v0.20.4

Final Note: FrontierHarness v1.0 is specifically designed for terminal-based tasks and software engineering contexts.

Want to test your own harness? If you use Runta, they are offering $100 in credits to get your evaluations started.