- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies overfitting in recursive self-improvement of LLM agent harnesses. The main observation is that repeatedly proposing and selecting harness modifications on a finite evolution set can improve the score on that set without producing improvements that transfer to unseen tasks. The paper introduces RRSI, which regularizes both the proposal and selection stages of harness evolution. The proposal-side mechanisms include an annealed edit budget, history-aware credit assignment, and structured exploration of underused components. On the selection side, the method uses leakage screening, noise-adjusted acceptance, cost-aware selection, and structural pruning.
The backbone policy remains frozen, while the harness can be modified through prompts, control flow, tools, memory, context management, and related components. The method is evaluated on coding, agentic workspace, and engineering design benchmarks. The results suggest that RRSI may obtain smaller gains on the evolution split than some baselines, while achieving stronger held-out and out-of-distribution performance with lower final policy-token cost.
Critical strengths
The paper focuses on an important issue in harness evolution: a higher score on the evolution set does not necessarily mean that the agent has acquired a reusable mechanism, especially when the same finite set of tasks is repeatedly used to propose and select harness changes.
The main methodological contribution is to introduce regularization into harness self-improvement while constraining both the proposal and selection stages. The proposal-side design is particularly sensible: the temporally annealed edit budget allows broader exploration in earlier rounds and more attributable updates later, while history-based structured exploration helps redirect the search toward previously unexplored components when progress stalls. On the selection side, the noise-aware and cost-aware rules discourage retaining changes that are only marginally useful or unnecessarily expensive. The , and terminology is also appropriately presented as a functional analogy rather than as literal norm-based optimization.
The evaluation is fairly broad for this line of work, covering multiple domains, held-out tasks, OOD benchmarks, different scoring mechanisms, policy robustness, ablations, and repeated runs. Reporting inference-token cost is also useful. The fact that RRSI obtains a smaller evolution-set gain but better transfer performance is consistent with the paper’s central motivation.
Critical weaknesses
One concern is that the OOD gains may reflect a more conservative search rather than the particular mechanisms introduced by RRSI. RRSI obtains a lower evolution-set score but a higher OOD score than unregularized evolution, which is consistent with the paper’s argument. However, the current ablations mainly remove the proposal-side and selection-side regularizers as groups. They do not compare against simpler alternatives such as a fixed one-edit budget, early stopping, or a basic cost-penalized selection rule. Such comparisons would help determine whether the gains come from the proposed history-based exploration and selection design, rather than simply from reducing search capacity and harness complexity.
The current experiments also do not fully separate these possibilities. The regularization choices and their hyperparameters are calibrated through the evolution process itself, with different settings across domains, while all candidate selection is still driven by the evolution split. More fine-grained ablations or comparisons with simple conservative baselines would make it clearer which parts of RRSI are responsible for the observed transfer gains.
Questions for authors
The efficiency comparison would benefit from a clearer accounting of the cost of evolution itself. Are the reported token numbers limited to running the final harness on the benchmark, or do they include proposal generation, critic and analyst calls, candidate evaluation, and the full evolution procedure? Separating search cost from final execution cost would make the comparison easier to interpret.
Several hyperparameters are specified separately for coding, agentic workspace, and engineering design, and the algorithm also includes domain-specific guards. Could the authors provide more details on how these choices were made and report whether the main conclusions are sensitive to reasonable changes in the noise tolerance, cost coefficients, and pruning or stall windows?
One point that seems important for interpreting the comparison is the precise protocol used for the test-time adaptation baseline. RRSI regularizes the harness-evolution process before transfer, whereas TTHE is designed to continue adapting the harness from unlabeled execution traces during test time. These are two different ways of addressing distribution shift. If execution traces from the new task distribution are fed back into the harness optimization process, one might reasonably expect this adaptation to mitigate, at least to some extent, the performance gap between the ID held-out and OOD settings. The relatively large OOD degradation reported for TTHE therefore warrants clarification. Could the authors specify whether execution traces collected from the OOD tasks were actually used to perform further self-supervised harness optimization for TTHE? If they were, the reported OOD scores reflect both transfer and continued adaptation to the OOD distribution, and it would be useful to understand why a substantial gap remains. If they were not, and the harness was evolved only on the designated evolution split and then frozen, then the experiment evaluates transfer of a fixed evolved harness rather than TTHE’s native test-time adaptation setting. This distinction is important for interpreting the comparison.
4 comments
Response to Reviewer bEhi (2/N)
Q3: The efficiency comparison would benefit from a clearer accounting of the cost of evolution itself. Are the reported token numbers limited to running the final harness on the benchmark, or do they include proposal generation, critic and analyst calls, candidate evaluation, and the full evolution procedure? Separating search cost from final execution cost would make the comparison easier to interpret.
A3: Thank you for pointing out this distinction. The token counts in the original Figure 4 measure the final harness's inference cost per trial on the evolve split, not the cost of the evolution procedure. We separately account for all LLM tokens consumed during search, including candidate evaluations and proposer, analyst, critic, and digester calls. As shown in Table R3, RRSI's search cost is comparable to Meta-Harness, AHE, and HarnessX, and substantially below TTHE. For deployment, as shown in Table R4, we also measure the final harness's inference cost separately on each agentic workspace OOD benchmark. RRSI uses fewer inference tokens than unregularized evolution on all three OOD benchmarks, although it remains more expensive than . We have revised the paper and put these details in Section 4.3 and Appendix E.4.
Table R3: Complete search tokens on coding.
| Method | Search tokens (M) |
|---|---|
| RRSI | 101 |
| Meta-Harness | 107 |
| AHE | 103 |
| TTHE | 268 |
| HarnessX | 99 |
| Table R4: OOD inference cost (M tokens/trial). |
| Harness | JobBench | GDPval | APEX-Agents |
|---|---|---|---|
| 1.98 | 2.52 | 2.38 | |
| Unregularized evolution | 3.76 | 4.25 | 4.31 |
| RRSI | 2.69 | 3.13 | 2.86 |
Q4: Several hyperparameters are specified separately for coding, agentic workspace, and engineering design, and the algorithm also includes domain-specific guards. Could the authors provide more details on how these choices were made and report whether the main conclusions are sensitive to reasonable changes in the noise tolerance, cost coefficients, and pruning or stall windows?
A4: We calibrate the noise tolerance from the evaluation variability of the unchanged initial harness on each domain’s evolve set. The resulting tolerances are 0.017 for coding, 0.004 for agentic workspace, and 0.020 for engineering design. The cost coefficients reflect the desired score–token trade-off in each domain’s scoring units. We use structural defaults for the edit budget and the stall and pruning windows.
- Sensitivity. We vary , the gain-dependent cost allowance , and the stall window one at a time on agentic workspace. We also test , which removes within-band score credit, on workspace and engineering design. As shown in Table R4, all tested workspace settings exceed unregularized evolution (40.3 OOD). The default gives the highest OOD score among the tested tolerances. Varying changes OOD by at most 0.4 points, while a larger improves OOD at higher inference cost. Setting lowers OOD by 0.8 points on workspace and 0.3 on engineering design.
- Domain-specific guards. Engineering design rejects a candidate if its valid-output rate falls by more than 3 percentage points or its no-submission rate rises by more than 2 percentage points relative to the incumbent, regardless of the candidate’s score. Coding and agentic workspace use no additional guard.
We report the calibration and the , , and results in Appendix E.1, the results in Appendix E.3, and the guard definition in Appendix D.3.
Table R4. Hyperparameter sensitivity on agentic workspace and engineering design. Tokens per trial measure final-harness inference on the evolve split.
| Domain | Setting | Evolve | ID held-out | OOD Avg. | Tokens/trial (M) |
|---|---|---|---|---|---|
| Workspace | RRSI (default) | 90.5 | 89.2 | 43.6 | 2.42 |
| Workspace | 90.6 | 88.8 | 42.4 | — | |
| Workspace | 90.7 | 88.6 | 42.2 | — | |
| Workspace | 90.1 | 88.2 | 41.9 | — | |
| Workspace | default | 90.3 | 89.1 | 42.9 | 2.12 |
| Workspace | default | 90.8 | 89.4 | 44.2 | 2.70 |
| Workspace | 91.0 | 89.4 | 43.2 | 2.55 | |
| Workspace | 90.3 | 89.2 | 43.4 | 2.27 | |
| Workspace | 90.3 | 88.7 | 42.8 | — | |
| Engineering design | RRSI (default) | 54.9 | — | 22.0 | — |
| Engineering design | 55.3 | — | 21.7 | — |
Response to Reviewer bEhi (3/3)
Q5: One point that seems important for interpreting the comparison is the precise protocol used for the test-time adaptation baseline. RRSI regularizes the harness-evolution process before transfer, whereas TTHE is designed to continue adapting the harness from unlabeled execution traces during test time. These are two different ways of addressing distribution shift. If execution traces from the new task distribution are fed back into the harness optimization process, one might reasonably expect this adaptation to mitigate, at least to some extent, the performance gap between the ID held-out and OOD settings. The relatively large OOD degradation reported for TTHE therefore warrants clarification. Could the authors specify whether execution traces collected from the OOD tasks were actually used to perform further self-supervised harness optimization for TTHE? If they were, the reported OOD scores reflect both transfer and continued adaptation to the OOD distribution, and it would be useful to understand why a substantial gap remains. If they were not, and the harness was evolved only on the designated evolution split and then frozen, then the experiment evaluates transfer of a fixed evolved harness rather than TTHE's native test-time adaptation setting. This distinction is important for interpreting the comparison.
A5: Thank you for highlighting the distinction between frozen-harness transfer and continued test-time adaptation.
- Original comparison protocol. In Table 1, TTHE performs label-free evolution only on the designated Harvey LAB evolve set (20 batches of 6 tasks, , ). Its final harness is then frozen and evaluated on JobBench, GDPval, and APEX-Agents. Thus, the original comparison measures frozen-harness transfer, not TTHE's native continued test-time adaptation.
- Additional target-distribution adaptation experiment. To examine this distinction empirically, we additionally allow TTHE to adapt using execution traces from JobBench. Its JobBench performance improves from approximately 36 to 40.2. We then evaluate the adapted harness on GDPval and APEX-Agents without further adaptation on those benchmarks. As shown in Table R5, allowing TTHE to use target-distribution execution traces substantially improves its performance, confirming the reviewer's observation that continued adaptation can mitigate distribution shift. Nevertheless, the JobBench-adapted TTHE harness remains below frozen RRSI by 0.5 points on JobBench, 2.2 on GDPval, and 1.8 on APEX-Agents, or 1.5 points on the OOD average. Importantly, these are different information-access protocols: the adapted TTHE receives additional JobBench execution traces and adaptation compute, whereas RRSI does not access OOD tasks during evolution. Furthermore, our supplementary experiment performs additional adaptation on JobBench only, not independently on all three OOD benchmarks.
We have revised the paper and put these details in Appendix C.
Table R5: Additional JobBench adaptation versus frozen transfer.
| Method | JobBench | GDPval | APEX-Agents | OOD Avg. |
|---|---|---|---|---|
| TTHE (original frozen transfer) | 35.2 | 47.0 | 31.7 | 38.0 |
| TTHE (+ JobBench adaptation) | 40.2 | 50.1 | 36.1 | 42.1 |
| RRSI (frozen transfer) | 40.7 | 52.3 | 37.9 | 43.6 |
Thanks for the authors' detailed reply; all my questions have been resolved. I will maintain my positive review.
Response to Reviewer bEhi (1/N)
Thank you for your constructive comments and suggestions. We have revised our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.
A1: Thank you for the suggestion. We compare RRSI with three simpler conservative alternatives: a fixed single-edit budget, cost-penalized greedy selection, and validation-based early stopping. Each uses a single constraint rather than the full RRSI framework, and all are repeated with independent proposer seeds. As shown in Table R1, all three controls reduce evolve-set optimization relative to unregularized evolution, but none approaches RRSI's OOD performance. Moreover, removing only the leakage critic lowers RRSI's OOD average to 41.6, and removing the annealed budget lowers it to 42.2. Thus, the results support a contribution from the particular proposal and selection mechanisms, rather than reduced search capacity alone. We add the individual ablations and simple controls to Section 4.3, with repeated-run details in Appendix E.3.
Table R1: Comparison with conservative baselines.
A2: We further compare RRSI with variants having nearly identical evolve-set scores in Table R2. Despite similar evolve scores (90.4–90.7), OOD performance differs by up to 1.7 points. This suggests that the degree of evolve-set conservativeness alone does not determine transfer. We also test validation-based early stopping using 90 evolution tasks and 30 separate validation tasks from Harvey LAB. It achieves only 40.2/40.5 OOD, compared with RRSI's 43.6/43.7. The selection parameters are fixed without consulting the OOD benchmarks; the cost allowances are chosen from the evolve setting and operational cost considerations. We have revised the paper and put these details in Appendix E.1 and Section 4.3.
Table R2: Variants with similar evolve scores.