- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HARNESS², which improves a frozen LLM agent’s harness at test time. For each task, it uses a Propose–Probe–Compose procedure to generate invariant and domain-specific harness edits, execute them, and combine changes based on observed behavioral differences, without access to ground-truth rewards or benchmark scores. It further stores adopted harnesses and distilled improvement experience in a persistent library to guide harness refinement on future tasks. It is evaluated on WorkBuddy-Bench, JobBench, and LAB, showing consistent gains over several baselines.
Critical strengths
- The paper is well written and accompanied by nicely made figures. It was easy to follow the paper.
- The method shows consistent improvements across multiple benchmarks, with extensive ablation studies, which provides solid evidence for its effectiveness.
- I appreciate that the authors compare against Best-of-k style test-time scaling, which helps isolate the effect of additional compute from the benefit of recursive harness refinement itself.
Critical weaknesses
- My main concern is the stability of the system over time. I understand why the authors avoid ground-truth feedback, since it is often unavailable in real-world deployment. However, behavioral analysis alone does not necessarily reflect task correctness. This is especially concerning for cross-task lessons, where an incorrect lesson may be reused and cause errors to accumulate. The paper shows some failures caused by shared incorrect assumptions, but does not directly study this long-term accumulation. While the reflector can reject unsupported lessons, I did not find a mechanism for later removing or revising outdated lessons. It would be useful to evaluate this issue or introduce a filtering mechanism for the library.
- Besides, the evaluated tasks are mostly specialized tool-using agent tasks. I would like to see whether the method also works on qualitatively different tasks, such as mathematical reasoning or more ambiguous conversational tasks.
Questions for authors
Could the authors analyze
- the cross-task stability of the library, including whether filtering or removing outdated lessons improves robustness, and
- demonstrate the method on qualitatively different tasks such as mathematical reasoning?
3 comments
Thank for the additional evidence. My concerns are addressed.
Dear Reviewer j5QF,
Thank you for your thoughtful feedback and for confirming that your concerns have been addressed. We would be grateful if you would consider updating your rating to reflect the additional evidence and our discussion.
Best regards, Authors
Response to Reviewer j5QF
We thank the reviewer for the positive assessment of the presentation, the breadth of the ablations, and the best-of-k comparison. We address the two concerns below. The additional OfficeQA and IMO-AnswerBench experiments reported below use GLM 5.3 Flash as both executor and editor, with no ground-truth answers or evaluation rewards used during improvement. —
We agree that behavioral contrasts do not establish correctness, and that an incorrect lesson could be reused and compound across tasks. The current design limits this risk in three ways, and we add a new experiment evaluating periodic library maintenance.
confounded, and when evidence is insufficient the Reflector writesNO_ENTRYinstead of a guess.To further test the effectiveness of whether a filter mechanism can help the long run, We run Harness² on the 172 tasks from OfficeQA [1] stream used in Q2 and compare two library variants:
The original Harness² improves over the base by 8.6 points in aggregate over this 172-task stream, and periodic meta-reflection adds a further 1.2 points. These results support the usefulness of library maintenance in this setting, but aggregate scores alone do not establish whether errors accumulate over time or whether the improvement persists over longer streams.
We evaluate Harness² on two benchmarks that differ from our main tool-using, artifact-producing tasks:
These results show that Harness² is effective beyond tool-use tasks, with gains of 15.0 percentage points on IMO-AnswerBench and 8.6 percentage points on OfficeQA. In particular, the substantial improvement on IMO-AnswerBench demonstrates that this framework can benefit reasoning-intensive tasks even when tool use is limited. Together, these findings support its broader applicability across tasks with different reasoning demands and output formats.
[1] Databricks. OfficeQA: A grounded reasoning benchmark over real-world documents. 2025.
[2] T. Luong et al. Towards robust mathematical reasoning. Google DeepMind, 2025.