acceptodds
Under review as a conference paper at ICLR 2027

An Empirical Study of Harness Design for Coding Agents

Abstract

Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Agent, Coding Agent, Coding Harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a2bXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work creates a react-based coding harness and studies the performance of some Nemotron and Mistral models on SWEBench and TerminalBench, varying three components: context management (T0 none/hard-fail, T1 elision, T2 elision + recall, T3 summarization, T4 staged elision then summarization), planning, and action space (predefined tools vs. bash-only). Context tiers are swept over 32k to 128k windows, and planning and tools are ablated at T4/128k. LLM-judge trajectory labels are validated against human annotators. The findings are:

(1) Context management matters mainly by preventing overflow;
(2) T4 is cheapest at similar accuracy, and recall goes unused;
(3) Planning helps weak models’ accuracy and stronger models’ cost;
(4) Predefined tools help weak models, while bash-only is cheaper for capable ones.

Critical strengths

(1) The controlled design using one harness is a reasonable way to separate the contribution of different mechanisms as opposed to comparing harnesses as a whole.
(2) The work is easily reproducible from prompts and tool schemas provided.
(3) The context-window sweep is a useful axis most context-management work lacks.
(4) The trajectory analysis (phase survival curves, failure-stage attribution, edit granularity) goes well beyond success and cost, with solid judge validation (κ ≈ 0.93).
(5) The limitations section is reasonable and the negative result on recall is included.

Critical weaknesses

Novelty (absence of).
The work generally is an integration of multiple previously known ideas or findings against one setup, which does not clear my novelty or interestingness bar. Several of the related works are cited, others (https://arxiv.org/abs/2608.11386, https://arxiv.org/abs/2508.21433) are not. Either way, each axis presented has already been individually studied, and I do not think the work finds enough nuance in the intersection of the different axes to justify a higher score. I think this is a more empirics focused paper that maybe could be better suited for EMNLP or an ARR Findings.

Realism (absence of).
(a) There isn’t any in-depth analysis of features to popular harnesses like claude-code, open-code, pi, and others.
(b) The models studied are not frontier (nor close) and results in models being used with a harness they aren’t trained for (resulting in things like the acknowledged tool-vocabulary mismatch).
(c) Context size in real harnesses currently is 200k-1m tokens, not 32-128k.
(d) Real harnesses are designed and measure cost around maximizing cache hits, but that is not considered in this analysis.
(e) Subagents, human interaction, search, reasoning effort are all important aspects of harnesses and not considered here.
(f) Recall tool is not realistic, popular harnesses do not have this.

General soundness complaints.
(a) T0 crashes when there is an overflow, so the measured gap vs. T0 measures anything vs. crashing, weak baseline here. truncation could be better.
(b) T4 compacts at a different B1 than T1 to T3.
(c) Weak vs. Strong models is just 30B vs. the rest.
(d) Statistics: single greedy runs, no confidence intervals, only 89 tasks from Terminal bench. This limits my confidence in the results, which is really important for an empirical study of this nature.
(e) The recall_event description tells the model it can usually just re-read or re-run instead, which might be the cause of the negative result.

Presentation:
(a) Prose is mostly fine to read, but it is basically impossible for me to keep track of what all these variables mean and found myself frequently needing to look back to remember. A table to track or some more intuitive naming would be nice than letter-number setup.
(b) If you’re going to make claims about accuracy-cost tradeoffs, you should have a pareto curve which shows this.
(c) Figures are extremely hard to understand. Odd, inconsistent coloring, formatting, and fonts. The fonts in figures are malignantly small. For line plots, there are too many lines in one line plot that make it difficult to understand what you are trying to show. Captions are unclear. Figures 6 and 7 are particularly hard to look at.

Some Quick Nits.
(a) Turn counts cited in the text (101→77, 96→70, 71→15) appear in no table.
(b) “%” is used where percentage points are meant.
(c) Some Table 13 Terminal-Bench rows appear to use 84 rather than 89 as the denominator.
(d) Figure 7 says “24%” where the table gives 23.2 pp.
(e) Model names are inconsistent.
(f) The judge prompt lists “C” twice.
(g) web_fetch and curl could still leak GitHub PRs, and this is not audited.

Questions for authors

Please answer my weaknesses if you have anything to rebut or if my understanding of the work is incorrect.

I have two clarifying questions:
(1) What ends a run when the model (a) emits a turn with no tool call, or (b) calls a tool not in the registry?
(2) Is the cost of summarization included in the measured cost?

Reviewer_wQo8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how different parts of an agent harness affect coding performance. Instead of testing existing harnesses as whole systems, the authors build a modular test harness. They keep the core execution loop the same while testing three parts: context management (5 strategies across 4 context window sizes), planning (on vs. off), and action space (predefined tools vs. bash-only). Experiments are run on SWE-Bench Verified and Terminal-Bench 2.1 across four models (three sizes of Nemotron-3 and Mistral-Medium-3.5). The paper also uses an LLM judge to classify turns in the trajectories into stages like localize, edit, and verify.

Critical strengths
  1. The paper is well organized.

  2. The conducted experiments are relatively comprehensive.

Critical weaknesses
  1. When bash-only shows an inferior performance on SWE-Bench, it is not clear whether the model is weak in bash scripting or the model misses the automated syntax error feedback.

  2. It is clear what the performance would be under tight planning and bash-only budgets.

  3. My major concern with soundness is that the comparisons between tools are bundled together. When comparing the full tool set to bash-only, the setting does not just change the commands. It also changes the system prompts, safety checks, and automatic linting tools. This makes it hard to distinguish which specific factor actually can cause performance change.

Questions for authors
  1. In the bash-only runs, did you log how many tasks failed simply because of basic Python syntax errors that ruff or pyflakes would have caught in the full tool setting?

  2. What happens if you turn planning on or off under a tight 32k window? Does the extra token overhead from updating the plan hurt small models when the window is already small?

  3. In Table 8, models barely ever called recall_event. Did the models actually understand that old information was saved, or was the prompt stub too easy for the model to ignore?

  4. Every setting was run only once per task. Given that LLM agent trajectories can vary between runs, how much run-to-run noise do you expect on the smaller 89-task Terminal-Bench?

Reviewer_R6EEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the effects of 3 coding-harness components (planning, the action space, and context management) on agent accuracy and cost. The authors build a harness with a fixed ReAct loop, vary the three components, and evaluate four models (three Nemotron-3 sizes plus Mistral-Medium-3.5-128B) on SWE-Bench Verified and Terminal-Bench 2.1 across five context-management tiers and four window budgets, for 176 settings. They report four findings: context management pays mainly by preventing window overflow and its value shrinks as the context length grows; a tier that elides before summarizing is the cheapest; planning raises success for the weakest model and lowers cost for the stronger ones; and predefined tools help bash-weak models while bash-only is cheaper for bash-capable ones. An LLM-judge trajectory analysis attributes each effect to a behavioral change.

Critical strengths
  1. Appendix I.2 checks the LLM judge against three human annotators over 200 trajectories and 15,610 labelled units. This human validation buttresses §4's analysis/claims, and is atypical in the literature (most papers report only the judge's own self-consistency).
Critical weaknesses
  1. The paper does not report standard errors for any of its means, and most of its conclusions are based on mean comparisons. I will raise my score if the authors address this weakness.
Questions for authors
  1. [clarification] What happens to the 66% of Nemotron-3 30B bash-only Terminal-Bench runs that stop after out-of-interface emissions (p. 6)? Are they killed after a certain number of failed tool calls, or do they continue running until manually terminated?
  2. Among the runs that planning shortened, is success lower than among the runs it left alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.