- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
StateHarness introduces a training-free runtime framework for maintaining evolving project state in long-horizon coding tasks. It combines persistent project records, a compact working brief, change-impact retrieval, and occasional diagnostic sessions, and evaluates the framework across three repository-level benchmarks. Across multiple long-horizon coding benchmarks, StateHarness improves the task metrics over Claude Code Sonnet 5.
Critical strengths
S1. The work is well motivated. Long coding projects require agents to track changing requirements, prior findings, dependencies, and unfinished work. Maintaining an explicit project state addresses a practical challenge in these tasks.
S2. The method is simple and easy to follow. The paper clearly describes how project records, brief updates, change-impact retrieval, and diagnostic feedback support the working agent during execution.
Critical weaknesses
W1. The evaluation uses only Claude Sonnet 5. Testing additional model families and capability levels (such as GPT-6-astra, claude Opus 5 or 5.5, claude Fable 5 or 5.1) would help distinguish which behaviors stronger agents already exhibit and which limitations still require explicit runtime support.
W2. The gains are not sufficiently attributed to individual components. The evaluation does not isolate the effects of structured project state, reminders and testing guidance, change-impact retrieval, and additional diagnostic sessions. Incomplete accounting of diagnostic tokens further limits assessment of the contribution and cost of extra inference.
W3. It is unclear why this mechanism (to be specific, the four forms of records and runtime support) is necessary, compared to other baselines such as a well-designed memory system. This diminishes the contribution of this work, limiting our understanding of what is actually useful in long-horizon agents.
Questions for authors
Q1. Why are diagnostic-session tokens not fully included in the reported costs? Please report total token usage including these sessions, together with the diagnostic model and the frequency of diagnostic calls. (W2)
Q2. How much of the improvement comes from structured project state, reminders and additional testing, change-impact retrieval, or separate diagnostic sessions? I suggest component ablations and controls with comparable inference and testing budgets. (W2)
Q3. Do the gains persist across model families and capability levels? Which failure modes are already handled by stronger baseline agents, and which still benefit from StateHarness? (W1)
Q4. Could you provide more justification for the mechanism? You could justify the design with a case study or compare this method with more baselines, such as a well-designed memory system for long-horizon coding agents. (W3)
We thank the reviewer for the constructive feedback. We have added component ablations and repeated runs, and provide additional cost accounting and clarification of the proposed mechanism below.
W2 + Q2 — Component contributions and ablations
We conducted a component ablation on all 37 ChainSWE chains, progressively adding StateHarness's runtime mechanisms. V1 is a prompt-only baseline that provides the same record-keeping instructions and templates without runtime support.
Prompted records improve the bug-fix rate from 35.1% to 38.6%, while full StateHarness further improves it to 39.5%. More importantly, runtime support reduces residual debt from 57 to 43 and P2P regressions from 31 to 23 relative to prompted records alone.
Adding diagnosis (V3 → V4) improves bug fixing (38.6% → 39.5%) and reduces residual debt (45 → 43), while P2P regressions increase from 20 to 23. These ablations show the incremental contribution of each component, although inference and testing budgets are not strictly matched.
W2 + Q1 — Diagnostic-session cost accounting
The original submission did not separately report diagnostic-session token usage. We now provide the complete ChainSWE token breakdown, including auxiliary sessions.
StateHarness's diagnostic sub-agent (
drift-analyst) uses Claude Code'ssonnetmodel alias, without explicitly pinning a specific model version. It was invoked 4 times across 37 chains (0.11 calls per chain).Including diagnostic sessions, StateHarness uses 37.5K tokens per chain, compared with 24.9K for Claude Code and 208.0K for LongHorizon-Harness. This shows that StateHarness introduces additional inference cost over the base agent while remaining substantially more token-efficient than LongHorizon-Harness.
We will include this complete breakdown in Appendix B.5.
W1 + Q3 — Generalization across working models
Our benchmark experiments use Claude Sonnet 5 as the working model to isolate the effects of runtime organization.
We added two independent runs on all 37 ChainSWE chains. Across the three runs, StateHarness achieves a mean bug-fix rate of 39.8%, compared with 34.8% for Claude Code, while mean residual debt decreases from 61.3 to 43.7.
These results demonstrate consistent improvements across repeated runs with Claude Sonnet 5. Generalization to stronger models and other model families remains to be evaluated.
W3 + Q4 — Why runtime state maintenance beyond memory?
We agree that structured memory and progress tracking are useful and established practices. Our prompt-only baseline confirms their value, improving bug fixing from 35.1% to 38.6%.
StateHarness addresses a more specific problem: an agent's own code changes can make its previously recorded task understanding outdated. The framework therefore goes beyond preserving information. It maintains a current working brief, retrieves potentially affected files after changes, and uses runtime feedback to revisit overlooked evidence.
Our sqlglot case study (§4.6, Figure 4) illustrates this mechanism. Both agents initially make broad parser changes, but StateHarness performs additional regression checking, detects a previously passing Databricks test that now fails, and narrows its patch. The final patch passes the target test without introducing regressions.
Together, the ablation results and sqlglot case study demonstrate the contribution of runtime state maintenance beyond prompted record-keeping. We will make this methodological distinction clearer in §2.