- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work creates a react-based coding harness and studies the performance of some Nemotron and Mistral models on SWEBench and TerminalBench, varying three components: context management (T0 none/hard-fail, T1 elision, T2 elision + recall, T3 summarization, T4 staged elision then summarization), planning, and action space (predefined tools vs. bash-only). Context tiers are swept over 32k to 128k windows, and planning and tools are ablated at T4/128k. LLM-judge trajectory labels are validated against human annotators. The findings are:
(1) Context management matters mainly by preventing overflow;
(2) T4 is cheapest at similar accuracy, and recall goes unused;
(3) Planning helps weak models’ accuracy and stronger models’ cost;
(4) Predefined tools help weak models, while bash-only is cheaper for capable ones.
Critical strengths
(1) The controlled design using one harness is a reasonable way to separate the contribution of different mechanisms as opposed to comparing harnesses as a whole.
(2) The work is easily reproducible from prompts and tool schemas provided.
(3) The context-window sweep is a useful axis most context-management work lacks.
(4) The trajectory analysis (phase survival curves, failure-stage attribution, edit granularity) goes well beyond success and cost, with solid judge validation (κ ≈ 0.93).
(5) The limitations section is reasonable and the negative result on recall is included.
Critical weaknesses
Novelty (absence of).
The work generally is an integration of multiple previously known ideas or findings against one setup, which does not clear my novelty or interestingness bar. Several of the related works are cited, others (https://arxiv.org/abs/2608.11386, https://arxiv.org/abs/2508.21433) are not. Either way, each axis presented has already been individually studied, and I do not think the work finds enough nuance in the intersection of the different axes to justify a higher score. I think this is a more empirics focused paper that maybe could be better suited for EMNLP or an ARR Findings.
Realism (absence of).
(a) There isn’t any in-depth analysis of features to popular harnesses like claude-code, open-code, pi, and others.
(b) The models studied are not frontier (nor close) and results in models being used with a harness they aren’t trained for (resulting in things like the acknowledged tool-vocabulary mismatch).
(c) Context size in real harnesses currently is 200k-1m tokens, not 32-128k.
(d) Real harnesses are designed and measure cost around maximizing cache hits, but that is not considered in this analysis.
(e) Subagents, human interaction, search, reasoning effort are all important aspects of harnesses and not considered here.
(f) Recall tool is not realistic, popular harnesses do not have this.
General soundness complaints.
(a) T0 crashes when there is an overflow, so the measured gap vs. T0 measures anything vs. crashing, weak baseline here. truncation could be better.
(b) T4 compacts at a different B1 than T1 to T3.
(c) Weak vs. Strong models is just 30B vs. the rest.
(d) Statistics: single greedy runs, no confidence intervals, only 89 tasks from Terminal bench. This limits my confidence in the results, which is really important for an empirical study of this nature.
(e) The recall_event description tells the model it can usually just re-read or re-run instead, which might be the cause of the negative result.
Presentation:
(a) Prose is mostly fine to read, but it is basically impossible for me to keep track of what all these variables mean and found myself frequently needing to look back to remember. A table to track or some more intuitive naming would be nice than letter-number setup.
(b) If you’re going to make claims about accuracy-cost tradeoffs, you should have a pareto curve which shows this.
(c) Figures are extremely hard to understand. Odd, inconsistent coloring, formatting, and fonts. The fonts in figures are malignantly small. For line plots, there are too many lines in one line plot that make it difficult to understand what you are trying to show. Captions are unclear. Figures 6 and 7 are particularly hard to look at.
Some Quick Nits.
(a) Turn counts cited in the text (101→77, 96→70, 71→15) appear in no table.
(b) “%” is used where percentage points are meant.
(c) Some Table 13 Terminal-Bench rows appear to use 84 rather than 89 as the denominator.
(d) Figure 7 says “24%” where the table gives 23.2 pp.
(e) Model names are inconsistent.
(f) The judge prompt lists “C” twice.
(g) web_fetch and curl could still leak GitHub PRs, and this is not audited.
Questions for authors
Please answer my weaknesses if you have anything to rebut or if my understanding of the work is incorrect.
I have two clarifying questions:
(1) What ends a run when the model (a) emits a turn with no tool call, or (b) calls a tool not in the registry?
(2) Is the cost of summarization included in the measured cost?