Analysis of Coding Agent Harness Architecture: An Empirical Investigation

arXiv Identifier: 2609.20804 | Submitted: September 17, 2026 Authors: Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani, Xiaoyang Wang

📌 Executive Summary

While autonomous coding agents rely on "harnesses" to convert the raw capabilities of a Large Language Model (LLM) into actual software engineering success, most research treats these harnesses as ~~black boxes~~ monolithic systems. This makes it nearly impossible to determine which specific part of the harness is actually driving performance.

The Core Research Question: How do individual harness components—specifically planning, action space, and context management—independently influence the performance of coding agents?

To answer this, the researchers utilized a modular, lightweight harness with a constant execution loop, varying only the three components mentioned above.

🛠️ Experimental Methodology

The study employed a rigorous grid-search approach to isolate variables. They tested four distinct models across two primary benchmarks: SWE-Bench Verified and Terminal-Bench 2.1.

The Harness Architecture

The agent's operation can be visualized as a continuous loop where the model interacts with the environment:

Test Configuration Matrix

The team evaluated 176 unique configurations based on the following variables:

Component	Variable	Options/Details
Context Management	Strategy	5 different strategies
Context Window	Budget	4 different size limits
Planning	Ablation	Enabled vs. Disabled
Action Space	Interface	Predefined Tools vs. Bash-only

📈 Key Findings

The empirical data revealed four primary insights regarding the synergy between the model and its harness:

1. Context Management & Budgeting

The utility of context management is inversely proportional to the available context window.

Observation: As the budget $\text{Budget}_{context} \downarrow$ , the value of management $\uparrow$ .
Primary Driver: Most gains come from simply preventing $\text{context-overflow}$ errors rather than improving the "quality" of the memory.

2. Efficiency of Elision

The researchers compared different ways of handling long histories:

The Winning Combo: $\text{Rule-based Elision} \rightarrow \text{LLM-based Summarization}$ .
The "Useless" Feature: Implementing "recoverable" elided content (allowing the model to ask for deleted text back) added unnecessary complexity without increasing accuracy.

3. The Role of Planning

Planning acts as a variable scaffold depending on the model's inherent strength:

For Weaker Models: Planning is an accuracy scaffold (it helps them actually solve the problem).
For Stronger Models: Planning is a cost saver (it reduces unnecessary steps) but does not significantly boost the final success rate.

4. Action Space: Tools vs. Bash

The ideal interface depends on the model's proficiency with the command line:

Bash-Weak Models: Benefit significantly from $\text{Predefined Tools}$ .
Bash-Capable Models: Perform optimally with a $\text{Bash-only}$ $Bash-only$ interface.
- Result: Lower operational costs.
- Context: Especially effective for tasks centered on the command line.

🔍 Trajectory-Level Analysis

To understand why these results occurred, the authors analyzed the agent's execution paths (trajectories).

Context Management $\rightarrow$ $\text{Extends Trajectory Length}$ : It keeps the agent "alive" longer by preventing crashes, without fundamentally changing how the agent thinks.
Planning $\rightarrow$ $\text{Alters Termination Points}$ : It changes where and when a trajectory ends (either by succeeding faster or giving up sooner).
Action Space $\rightarrow$ $\text{Adjusts Granularity}$ : It changes the "size" of the steps the agent takes when writing or modifying code.

✅ Conclusion & Design Recommendations

The study provides a roadmap for building "Model-Aware" and "Budget-Aware" harnesses.

For high-budget, strong models: Use Bash-only interfaces and minimal planning to save costs.
For low-budget, weaker models: Prioritize rule-based elision and robust planning scaffolds.
General Rule: Avoid complex "recoverable" memory systems; they are rarely utilized by the LLM.

Final Takeaway: By decoupling the harness into modular components, developers can optimize for the specific strengths of their chosen LLM and the constraints of their hardware budget.