acceptodds
Under review as a conference paper at ICLR 2027

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

28 views

Abstract

An LLM agent's capability is largely magnified by its harness, namely the prompts, control flow, tooling, memory, and context management surrounding the frozen backbone model. Recent methods increasingly automate this process by iteratively proposing and selecting component-wise edits of an agent harness, practically establishing a form of recursive self-improvement (RSI) at the agent-system level. However, such recursive evolution may overfit by memorizing the training tasks, showing large in-distribution gains that shrink or even vanish on out-of-distribution benchmarks. We introduce Regularized Recursive Self-Improvement of Agent Harnesses (RRSI), which incorporates the principles of regularizations into harness self-improvement by constraining the evolution candidate proposal and selection. The proposer operates with a temporally annealed budget, limiting how many edits a candidate can bundle, and it encourages unexplored trajectories based on evolution history. The selector is equipped with a critic and a pruner: the critic screens benchmark-specific proposals, while the pruner, removes changes that are too small, too expensive, or no longer useful. Together these constraints favor reusable agent mechanisms over benchmark-specific ones or even noises. Across eight benchmarks spanning coding, agentic workspace and engineering design tasks, RRSI gains up to 6.0 points on the split it evolves against and up to 4.3 points on the five out-of-distribution benchmarks, while producing a harness that runs on 30% fewer policy tokens than the unregularized evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self-Improvement, Harness Design

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bEhiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies overfitting in recursive self-improvement of LLM agent harnesses. The main observation is that repeatedly proposing and selecting harness modifications on a finite evolution set can improve the score on that set without producing improvements that transfer to unseen tasks. The paper introduces RRSI, which regularizes both the proposal and selection stages of harness evolution. The proposal-side mechanisms include an annealed edit budget, history-aware credit assignment, and structured exploration of underused components. On the selection side, the method uses leakage screening, noise-adjusted acceptance, cost-aware selection, and structural pruning.

The backbone policy remains frozen, while the harness can be modified through prompts, control flow, tools, memory, context management, and related components. The method is evaluated on coding, agentic workspace, and engineering design benchmarks. The results suggest that RRSI may obtain smaller gains on the evolution split than some baselines, while achieving stronger held-out and out-of-distribution performance with lower final policy-token cost.

Critical strengths

The paper focuses on an important issue in harness evolution: a higher score on the evolution set does not necessarily mean that the agent has acquired a reusable mechanism, especially when the same finite set of tasks is repeatedly used to propose and select harness changes.

The main methodological contribution is to introduce regularization into harness self-improvement while constraining both the proposal and selection stages. The proposal-side design is particularly sensible: the temporally annealed edit budget allows broader exploration in earlier rounds and more attributable updates later, while history-based structured exploration helps redirect the search toward previously unexplored components when progress stalls. On the selection side, the noise-aware and cost-aware rules discourage retaining changes that are only marginally useful or unnecessarily expensive. The L0,L1L_0, L_1, and L2L_2 terminology is also appropriately presented as a functional analogy rather than as literal norm-based optimization.

The evaluation is fairly broad for this line of work, covering multiple domains, held-out tasks, OOD benchmarks, different scoring mechanisms, policy robustness, ablations, and repeated runs. Reporting inference-token cost is also useful. The fact that RRSI obtains a smaller evolution-set gain but better transfer performance is consistent with the paper’s central motivation.

Critical weaknesses

One concern is that the OOD gains may reflect a more conservative search rather than the particular mechanisms introduced by RRSI. RRSI obtains a lower evolution-set score but a higher OOD score than unregularized evolution, which is consistent with the paper’s argument. However, the current ablations mainly remove the proposal-side and selection-side regularizers as groups. They do not compare against simpler alternatives such as a fixed one-edit budget, early stopping, or a basic cost-penalized selection rule. Such comparisons would help determine whether the gains come from the proposed history-based exploration and selection design, rather than simply from reducing search capacity and harness complexity.

The current experiments also do not fully separate these possibilities. The regularization choices and their hyperparameters are calibrated through the evolution process itself, with different settings across domains, while all candidate selection is still driven by the evolution split. More fine-grained ablations or comparisons with simple conservative baselines would make it clearer which parts of RRSI are responsible for the observed transfer gains.

Questions for authors

The efficiency comparison would benefit from a clearer accounting of the cost of evolution itself. Are the reported token numbers limited to running the final harness on the benchmark, or do they include proposal generation, critic and analyst calls, candidate evaluation, and the full evolution procedure? Separating search cost from final execution cost would make the comparison easier to interpret.

Several hyperparameters are specified separately for coding, agentic workspace, and engineering design, and the algorithm also includes domain-specific guards. Could the authors provide more details on how these choices were made and report whether the main conclusions are sensitive to reasonable changes in the noise tolerance, cost coefficients, and pruning or stall windows?

One point that seems important for interpreting the comparison is the precise protocol used for the test-time adaptation baseline. RRSI regularizes the harness-evolution process before transfer, whereas TTHE is designed to continue adapting the harness from unlabeled execution traces during test time. These are two different ways of addressing distribution shift. If execution traces from the new task distribution are fed back into the harness optimization process, one might reasonably expect this adaptation to mitigate, at least to some extent, the performance gap between the ID held-out and OOD settings. The relatively large OOD degradation reported for TTHE therefore warrants clarification. Could the authors specify whether execution traces collected from the OOD tasks were actually used to perform further self-supervised harness optimization for TTHE? If they were, the reported OOD scores reflect both transfer and continued adaptation to the OOD distribution, and it would be useful to understand why a substantial gap remains. If they were not, and the harness was evolved only on the designated evolution split and then frozen, then the experiment evaluates transfer of a fixed evolved harness rather than TTHE’s native test-time adaptation setting. This distinction is important for interpreting the comparison.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bEhi (1/N)

Thank you for your constructive comments and suggestions. We have revised our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.


Q1: One concern is that the OOD gains may reflect a more conservative search rather than the particular mechanisms introduced by RRSI. RRSI obtains a lower evolution-set score but a higher OOD score than unregularized evolution, which is consistent with the paper's argument. However, the current ablations mainly remove the proposal-side and selection-side regularizers as groups. They do not compare against simpler alternatives such as a fixed one-edit budget, early stopping, or a basic cost-penalized selection rule. Such comparisons would help determine whether the gains come from the proposed history-based exploration and selection design, rather than simply from reducing search capacity and harness complexity.

A1: Thank you for the suggestion. We compare RRSI with three simpler conservative alternatives: a fixed single-edit budget, cost-penalized greedy selection, and validation-based early stopping. Each uses a single constraint rather than the full RRSI framework, and all are repeated with independent proposer seeds. As shown in Table R1, all three controls reduce evolve-set optimization relative to unregularized evolution, but none approaches RRSI's OOD performance. Moreover, removing only the leakage critic lowers RRSI's OOD average to 41.6, and removing the annealed budget lowers it to 42.2. Thus, the results support a contribution from the particular proposal and selection mechanisms, rather than reduced search capacity alone. We add the individual ablations and simple controls to Section 4.3, with repeated-run details in Appendix E.3.

Table R1: Comparison with conservative baselines.

MethodEvolveOOD Avg.
Unregularized evolution92.840.3
Fixed single-edit budget91.7 / 92.040.8 / 41.0
Cost-penalized selection91.2 / 91.840.6 / 40.4
Validation early stopping92.5 / 92.040.2 / 40.5
RRSI90.5 / 90.843.6 / 43.7

Q2: The current experiments also do not fully separate these possibilities. The regularization choices and their hyperparameters are calibrated through the evolution process itself, with different settings across domains, while all candidate selection is still driven by the evolution split. More fine-grained ablations or comparisons with simple conservative baselines would make it clearer which parts of RRSI are responsible for the observed transfer gains.

A2: We further compare RRSI with variants having nearly identical evolve-set scores in Table R2. Despite similar evolve scores (90.4–90.7), OOD performance differs by up to 1.7 points. This suggests that the degree of evolve-set conservativeness alone does not determine transfer. We also test validation-based early stopping using 90 evolution tasks and 30 separate validation tasks from Harvey LAB. It achieves only 40.2/40.5 OOD, compared with RRSI's 43.6/43.7. The selection parameters are fixed without consulting the OOD benchmarks; the cost allowances are chosen from the evolve setting and operational cost considerations. We have revised the paper and put these details in Appendix E.1 and Section 4.3.

Table R2: Variants with similar evolve scores.

VariantEvolveOOD Avg.
RRSI w/o pruning90.443.2
RRSI w/o noise floor90.642.4
RRSI w/o cost rule90.743.1
RRSI w/o proposal regularizers90.741.9
RRSI90.543.6
AuthorsOpenReview ↗

Response to Reviewer bEhi (2/N)

Q3: The efficiency comparison would benefit from a clearer accounting of the cost of evolution itself. Are the reported token numbers limited to running the final harness on the benchmark, or do they include proposal generation, critic and analyst calls, candidate evaluation, and the full evolution procedure? Separating search cost from final execution cost would make the comparison easier to interpret.

A3: Thank you for pointing out this distinction. The token counts in the original Figure 4 measure the final harness's inference cost per trial on the evolve split, not the cost of the evolution procedure. We separately account for all LLM tokens consumed during search, including candidate evaluations and proposer, analyst, critic, and digester calls. As shown in Table R3, RRSI's search cost is comparable to Meta-Harness, AHE, and HarnessX, and substantially below TTHE. For deployment, as shown in Table R4, we also measure the final harness's inference cost separately on each agentic workspace OOD benchmark. RRSI uses fewer inference tokens than unregularized evolution on all three OOD benchmarks, although it remains more expensive than H0H_0. We have revised the paper and put these details in Section 4.3 and Appendix E.4.

Table R3: Complete search tokens on coding.

MethodSearch tokens (M)
RRSI101
Meta-Harness107
AHE103
TTHE268
HarnessX99
Table R4: OOD inference cost (M tokens/trial).
HarnessJobBenchGDPvalAPEX-Agents
H0H_01.982.522.38
Unregularized evolution3.764.254.31
RRSI2.693.132.86

Q4: Several hyperparameters are specified separately for coding, agentic workspace, and engineering design, and the algorithm also includes domain-specific guards. Could the authors provide more details on how these choices were made and report whether the main conclusions are sensitive to reasonable changes in the noise tolerance, cost coefficients, and pruning or stall windows?

A4: We calibrate the noise tolerance δ\delta from the evaluation variability of the unchanged initial harness H0H_0 on each domain’s evolve set. The resulting tolerances are 0.017 for coding, 0.004 for agentic workspace, and 0.020 for engineering design. The cost coefficients reflect the desired score–token trade-off in each domain’s scoring units. We use structural defaults for the edit budget and the stall and pruning windows.

  1. Sensitivity. We vary δ\delta, the gain-dependent cost allowance β1\beta_1, and the stall window ww one at a time on agentic workspace. We also test ws=0w_s=0, which removes within-band score credit, on workspace and engineering design. As shown in Table R4, all tested workspace settings exceed unregularized evolution (40.3 OOD). The default δ0\delta_0 gives the highest OOD score among the tested tolerances. Varying ww changes OOD by at most 0.4 points, while a larger β1\beta_1 improves OOD at higher inference cost. Setting ws=0w_s=0 lowers OOD by 0.8 points on workspace and 0.3 on engineering design.
  2. Domain-specific guards. Engineering design rejects a candidate if its valid-output rate falls by more than 3 percentage points or its no-submission rate rises by more than 2 percentage points relative to the incumbent, regardless of the candidate’s score. Coding and agentic workspace use no additional guard.

We report the calibration and the δ\delta, β1\beta_1, and ww results in Appendix E.1, the ws=0w_s=0 results in Appendix E.3, and the guard definition in Appendix D.3.

Table R4. Hyperparameter sensitivity on agentic workspace and engineering design. Tokens per trial measure final-harness inference on the evolve split.

DomainSettingEvolveID held-outOOD Avg.Tokens/trial (M)
WorkspaceRRSI (default)90.589.243.62.42
Workspaceδ=0\delta=090.688.842.4—
Workspaceδ=0.5δ0\delta=0.5\delta_090.788.642.2—
Workspaceδ=2δ0\delta=2\delta_090.188.241.9—
Workspace0.5×0.5\times default β1\beta_190.389.142.92.12
Workspace2×2\times default β1\beta_190.889.444.22.70
Workspacew=2w=291.089.443.22.55
Workspacew=6w=690.389.243.42.27
Workspacews=0w_s=090.388.742.8—
Engineering designRRSI (default)54.9—22.0—
Engineering designws=0w_s=055.3—21.7—
AuthorsOpenReview ↗

Response to Reviewer bEhi (3/3)

Q5: One point that seems important for interpreting the comparison is the precise protocol used for the test-time adaptation baseline. RRSI regularizes the harness-evolution process before transfer, whereas TTHE is designed to continue adapting the harness from unlabeled execution traces during test time. These are two different ways of addressing distribution shift. If execution traces from the new task distribution are fed back into the harness optimization process, one might reasonably expect this adaptation to mitigate, at least to some extent, the performance gap between the ID held-out and OOD settings. The relatively large OOD degradation reported for TTHE therefore warrants clarification. Could the authors specify whether execution traces collected from the OOD tasks were actually used to perform further self-supervised harness optimization for TTHE? If they were, the reported OOD scores reflect both transfer and continued adaptation to the OOD distribution, and it would be useful to understand why a substantial gap remains. If they were not, and the harness was evolved only on the designated evolution split and then frozen, then the experiment evaluates transfer of a fixed evolved harness rather than TTHE's native test-time adaptation setting. This distinction is important for interpreting the comparison.

A5: Thank you for highlighting the distinction between frozen-harness transfer and continued test-time adaptation.

  1. Original comparison protocol. In Table 1, TTHE performs label-free evolution only on the designated Harvey LAB evolve set (20 batches of 6 tasks, G=2G=2, R=3R=3). Its final harness is then frozen and evaluated on JobBench, GDPval, and APEX-Agents. Thus, the original comparison measures frozen-harness transfer, not TTHE's native continued test-time adaptation.
  2. Additional target-distribution adaptation experiment. To examine this distinction empirically, we additionally allow TTHE to adapt using execution traces from JobBench. Its JobBench performance improves from approximately 36 to 40.2. We then evaluate the adapted harness on GDPval and APEX-Agents without further adaptation on those benchmarks. As shown in Table R5, allowing TTHE to use target-distribution execution traces substantially improves its performance, confirming the reviewer's observation that continued adaptation can mitigate distribution shift. Nevertheless, the JobBench-adapted TTHE harness remains below frozen RRSI by 0.5 points on JobBench, 2.2 on GDPval, and 1.8 on APEX-Agents, or 1.5 points on the OOD average. Importantly, these are different information-access protocols: the adapted TTHE receives additional JobBench execution traces and adaptation compute, whereas RRSI does not access OOD tasks during evolution. Furthermore, our supplementary experiment performs additional adaptation on JobBench only, not independently on all three OOD benchmarks.

We have revised the paper and put these details in Appendix C.

Table R5: Additional JobBench adaptation versus frozen transfer.

MethodJobBenchGDPvalAPEX-AgentsOOD Avg.
TTHE (original frozen transfer)35.247.031.738.0
TTHE (+ JobBench adaptation)40.250.136.142.1
RRSI (frozen transfer)40.752.337.943.6

Thanks for the authors' detailed reply; all my questions have been resolved. I will maintain my positive review.

Reviewer_D4kkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposed RRSI, where an LLM repeatedly modifies prompts, tools, and control flow, and selects changes based on performance on a fixed task set. It aims to improve cross-benchmark transfer through constraints on the number of edits per round, historical evidence tracking, structured exploration, leakage screening, noise tolerance, and cost-aware acceptance. Harnesses evolved on one benchmark per domain are then tested unchanged on five other benchmarks. RRSI improves on the starting harness everywhere, and in the agentic domain it transfers better than four recent evolution methods.

Critical strengths
  • The research question is practically important. Transferability matters when repeated adaptive optimization are done on a fixed evaluation set.
  • The gains over the starting harness hold across three domains and two independent runs.
Critical weaknesses
  1. RRSI combines about nine mechanisms, and the paper motivates each one through an analogy to classical regularization. Their necessity is never tested one at a time. The only ablation removes two large groups of rules in a single domain with one run each, and the differences it finds are close to the noise level. Several rules seem to do little in practice, and the leakage critic, for example, rejected one of 38 coding candidates. In the end the agentic harness keeps just four small checks out of 32 candidates. RRSI also spends about 55% more tokens than the starting harness, so a compute-matched baseline is needed as well.

  2. Per-method baselines are reported only in the agentic domain, and the baselines' budgets and adaptations are not described. The method has around ten hand-set hyperparameters with no sensitivity study. One component (DOMAINGUARD) is never defined, and some numbers in the abstract differ from the tables. Together these make the results hard to reproduce and hard to compare with future work.

  3. It is hard to see when a practitioner would choose RRSI. The procedure is heavy. It stacks about nine mechanisms with around ten hyperparameters whose values differ by domain and are partly calibrated by hand. Despite its elaborate optimization procedure, the final agentic harness retains only four relatively simple verification checks. It's unclear whether the complexity of RRSI is necessary to obtain these improvements.

Questions for authors
  1. Could you provide individual ablations of the key components?
  2. How does RRSI compare with standard evolution on a held-out validation set at matched compute?
  3. What does one full search run cost? How should a practitioner set the hyperparameters for a new domain?
6 comments
AuthorsOpenReview ↗

Response to Reviewer D4kk (1/N)

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we’ve revised our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.


Q1: RRSI combines about nine mechanisms, and the paper motivates each one through an analogy to classical regularization. Their necessity is never tested one at a time.

A1: Thank you for the constructive comment. Following your advice, we conduct single-component ablations on agentic workspace, removing each regularizer independently while keeping all other settings unchanged. As shown in Table R1, the leakage critic, annealed budget, noise floor, and structured exploration have the largest observed effects on transfer, with OOD drops of 1.1–2.0 points. The remaining components have smaller transfer effects. In particular, the cost rule and pruning primarily improve efficiency: removing them increases inference tokens by 34% and 10%, respectively. We therefore distinguish transfer-oriented regularizers from efficiency-oriented regularizers, rather than claiming that every component contributes equally to generalization. We have revised the paper and put these details in Section 4.3 and Appendix E.3.

Table R1: Individual ablations on agentic workspace.

VariantOOD Avg.Tokens/trial (M)
RRSI43.62.42
w/o leakage critic41.62.87
w/o annealed budget42.22.79
w/o noise floor42.42.49
w/o structured exploration42.52.64
w/o edit history43.02.54
w/o cost rule43.13.25
w/o pruning43.22.67

Q2: The only ablation removes two large groups of rules in a single domain with one run each, and the differences it finds are close to the noise level.

A2: Grouped ablations with single search runs are insufficient to identify the contribution of individual mechanisms. We provide the single-component results in Table R1 and repeat RRSI and three simpler controls using independent proposer seeds. As shown in Table R2, across these repeated runs, the OOD difference is at most 0.3 points, compared with RRSI's 2.8–3.3 point advantage over the controls. Furthermore, the paired task-level bootstrap comparison against Meta-Harness, the strongest evolution baseline by OOD average, yields a +3.0-point improvement with a 95% CI of [+2.2, +4.0]. We have revised the paper and put these details in Section 4.3 and Appendix E.2-E.3.

Table R2: OOD performance across two independent search runs.

MethodRun 1Run 2
RRSI43.643.7
Greedy, fixed single-edit40.841.0
Greedy, cost-penalized40.640.4
Greedy, validation early stopping40.240.5

Q3: Several rules seem to do little in practice, and the leakage critic, for example, rejected one of 38 coding candidates.

A3: Rejection frequency alone does not measure the importance of a constraint across domains. As shown in Table R1, removing the critic increases the evolve score from 90.5 to 93.0, but decreases the OOD average from 43.6 to 41.6. This is the largest OOD drop among the individual ablations. It suggests that preventing benchmark-specific candidate selection can matter substantially even when the number of explicit rejections is small in another domain. Other rules have different roles. For example, removing pruning changes the OOD average only from 43.6 to 43.2, but increases inference cost from 2.42M to 2.67M tokens/trial. Some components primarily constrain complexity and deployment cost. We have revised the paper and put these details in Section 4.3.


Q4: In the end the agentic harness keeps just four small checks out of 32 candidates.

A4: The small number of retained edits is consistent with the intended goal of RRSI: selecting a compact set of reusable harness improvements rather than accumulating changes that improve only the evolve benchmark. On agentic workspace, RRSI retains four edits from 32 evaluated candidates and achieves 43.6 OOD with 2.42M tokens/trial. Unregularized evolution achieves only 40.3 OOD while consuming 3.80M tokens/trial. Importantly, the simplicity of the final harness does not mean that a simple search procedure necessarily discovers the same improvements. A fixed single-edit baseline reaches 40.8/41.0 OOD, and a cost-penalized baseline reaches 40.6/40.4, both substantially below RRSI. These results support the value of the selection process in producing a compact, transferable harness. They do not establish that each of the four retained edits is individually necessary. We clarify this distinction in Section 4.3 and the qualitative case study in Appendix F.

AuthorsOpenReview ↗

Response to Reviewer D4kk (2/N)

Q5: RRSI also spends about 55% more tokens than the starting harness, so a compute-matched baseline is needed as well.

A5: As shown in Table R3, we increase the execution-turn budget of the unchanged H0H_0, bringing its cost to 2.38M tokens/trial, close to RRSI's 2.42M. At approximately matched inference cost, RRSI improves ID held-out performance by 1.7 points and OOD performance by 3.2 points over the unchanged harness with additional turns. This indicates that simply allocating more execution steps does not reproduce the benefits of harness optimization. We have revised the paper and put these details in Section 4.3 and Appendix E.4.

Table R3: Compute-matched comparison.

HarnessTokens/trial (M)ID held-outOOD Avg.
Original H0H_01.5686.939.7
H0H_0, increased turn budget2.3887.540.4
RRSI2.4289.243.6

Q6: Per-method baselines are reported only in the agentic domain, and the baselines' budgets and adaptations are not described.

A6: We have extended the comparison with all four evolution baselines to coding and engineering design. As shown in Table R4, RRSI achieves the highest OOD score in both additional domains, exceeding the strongest competing baseline by 1.1 points on SWE-bench Verified and 1.6 points on Frontier-Eng. Notably, Meta-Harness achieves a higher coding evolve score (81.2 versus 80.2), while HarnessX achieves a slightly higher engineering evolve score (55.0 versus 54.9). This provides additional evidence across distinct task and evaluation settings that maximizing evolve-set performance does not necessarily yield the best cross-benchmark transfer. We also clarify how each baseline is adapted to harness evolution. All methods start from the same domain-specific initial harness and frozen backbone policy, and receive feedback only from the designated evolution tasks. As shown in Table R5, the search-token totals include both candidate evaluations and auxiliary LLM calls, rather than only final-harness execution. Baselines retain their method-specific search widths, so equal evolution horizons should not be interpreted as identical total search compute. In particular, TTHE is evaluated under a frozen-transfer protocol, without continued adaptation on OOD execution traces. We have revised the paper and put these details in Appendix E.5 and Appendix E.4.

Table R4: Cross-domain comparison with prior harness-evolution methods.

MethodTerminal-Bench 2.1 (Evolve)SWE-bench Verified (OOD)EngDesign (Evolve)Frontier-Eng (OOD)
Meta-Harness81.282.753.820.2
AHE78.681.853.018.3
TTHE79.782.254.519.0
HarnessX79.282.755.020.4
RRSI80.283.854.922.0

Table R5: Baseline configurations and adaptation protocols.

MethodSearch and selection protocolComplete coding search tokens
Meta-HarnessOne candidate per round; 2 evaluation trials per candidate107M
AHEObservability-driven evolution; 2 evaluation trials per candidate; no RRSI-style acceptance gate103M
TTHELabel-free evolution; batches of 6 tasks; G=2, R=3G=2,\ R=3; final harness frozen for OOD evaluation268M
HarnessXModular harness evolution; 1 evaluation trial per candidate; cost-weighted acceptance99M
RRSIRegularized proposal and selection101M
AuthorsOpenReview ↗

Response to Reviewer D4kk (3/N)

Q7: The method has around ten hand-set hyperparameters with no sensitivity study.

A7: We vary four hyperparameters one at a time. We test the noise tolerance $\delta, the gain-dependent cost allowance β1\beta_1, the structured-exploration stall window ww, and the within-band score weight wsw_s. We also test ws=0w_s=0 on engineering design. We calibrate the default δ0=0.004\delta_0=0.004 from evaluation variability of the initial harness H0H_0. As shown in Table R7, all tested workspace settings outperform unregularized evolution (40.3 OOD). The default δ\delta performs best among the tested tolerances, and changing ww shifts OOD scores by at most 0.4 points. Halving β1\beta_1 saves 12.4% in inference tokens at a cost of 0.7 OOD points, while doubling it uses 11.6% more tokens and gains 0.6 points. Setting ws=0w_s=0 lowers OOD scores by 0.8 points on workspace and 0.3 on engineering design. We have revised the paper and put these details in Appendix E.1 and Appendix E.3.

Table R7. Hyperparameter sensitivity. Tokens per trial measure final-harness inference on the evolve split.

DomainSettingEvolveID held-outOOD Avg.Tokens/trial (M)
WorkspaceDefault90.589.243.62.42
Workspaceδ=0\delta=090.688.842.4—
Workspaceδ=0.5δ0\delta=0.5\delta_090.788.642.2—
Workspaceδ=2δ0\delta=2\delta_090.188.241.9—
Workspace0.5×0.5\times default β1\beta_190.389.142.92.12
Workspace2×2\times default β1\beta_190.889.444.22.70
Workspacew=2w=291.089.443.22.55
Workspacew=6w=690.389.243.42.27
Workspacews=0w_s=090.388.742.8—
Engineering designDefault54.9—22.0—
Engineering designws=0w_s=055.3—21.7—

Q8: One component (DOMAINGUARD) is never defined and some numbers in the abstract differ from the tables.

A8: Thank you for identifying these issues.

  1. DomainGuard. Only engineering design uses an additional non-compensatory guard. A candidate is rejected if its valid-submission rate decreases by more than 0.03 or its no-submission rate increases by more than 0.02 relative to the incumbent. Coding and agentic workspace use no additional guard ((g=1)). We add this definition to Appendix D.3.
  2. Numerical corrections. The maximum OOD improvement is 4.7 points on JobBench, rather than 4.3. The reduction in policy tokens relative to unregularized evolution is approximately 36%, rather than 30%, calculated from 2.42M versus 3.80M tokens/trial.

We have revised the paper and put these details in the Abstract and Introduction.


Q9: It is hard to see when a practitioner would choose RRSI. The procedure is heavy. It stacks about nine mechanisms with around ten hyperparameters whose values differ by domain and are partly calibrated by hand. Despite its elaborate optimization procedure, the final agentic harness retains only four relatively simple verification checks. It's unclear whether the complexity of RRSI is necessary to obtain these improvements.

A9: RRSI is designed for practitioners who have a limited task suite for optimization and expect to deploy the resulting harness repeatedly on tasks from different distributions. The offline search screens out edits that exploit quirks of the evolve set and prunes checks that add little value. Retaining only four verification checks is consistent with this goal of keeping the deployed harness compact.

As shown in Table R9, across two runs, RRSI achieves 43.6–43.7 OOD, compared with 40.2–41.0 for the three simpler controls, and uses fewer inference tokens per trial. The leakage-critic ablation illustrates why screening matters: disabling it raises the Evolve score to 93.0 while lowering the OOD average to 41.6. These comparisons support using RRSI when transfer matters and the one-time search cost can be spread over many deployments. We added this discussion to Section 4.3.

Table R9. Conservative search alternatives. Entries show run 1 / run 2. Tokens per trial measure final-harness inference on the evolve split.

MethodEvolveOOD Avg.Tokens/trial (M)
Fixed single-edit budget91.7 / 92.040.8 / 41.03.01 / 3.12
Cost-penalized selection91.2 / 91.840.6 / 40.42.76 / 2.90
Validation early stopping92.5 / 92.040.2 / 40.53.24 / 3.06
RRSI90.5 / 90.843.6 / 43.72.42 / 2.32
Reviewer_j5wLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RRSI, a framework for improving agent harnesses while keeping the backbone model frozen. RRSI combines an annealed edit budget, history-based proposal guidance, structured exploration, leakage screening, noise-aware selection, cost constraints, and structural pruning. The goal is to reduce overfitting to the tasks used during harness evolution. Experiments span eight benchmarks across coding, agentic workspace, and engineering design. The reported results show improved transfer to unseen benchmarks, including an increase in the workspace OOD average from 39.7 to 43.6, alongside lower inference-token consumption than unregularized evolution.

Critical strengths
  1. The paper addresses an important evaluation problem. Improvements on repeatedly reused evolution tasks do not necessarily indicate transferable agent capabilities. Explicitly separating evolution performance from held-out and cross-benchmark performance makes the study practically relevant.

  2. The transfer evaluation is reasonably broad. The experiments cover three domains and include both model-judged tasks and tasks evaluated by executable tests or simulators. The additional policy-family experiments and transfer to a smaller, unseen backbone provide useful evidence beyond a single model–benchmark pairing.

  3. The paper provides meaningful reliability checks. Appendix E.2 reports paired task-level bootstrap intervals and two independent RRSI search runs. Gains over the initial harness are reproduced across the reported splits. The qualitative examples also help explain what the evolved harnesses actually change.

Critical weaknesses
  1. The ablations do not isolate the mechanisms responsible for improved transfer. Table 2 removes proposal-side and acceptance-side regularizers as groups, but each group contains several distinct interventions. Consequently, it remains unclear whether the gains primarily come from leakage screening, conservative acceptance, smaller edits, pruning, or their interaction. Simpler controls, such as a fixed single-edit budget or cost-constrained selection alone, would help establish the value of the complete framework.

  2. The noise-handling rules provide weaker protection than some of the discussion suggests. A tolerance estimated from repeated evaluations of the initial harness does not by itself establish reliability under repeated adaptive selection. Moreover, Equation 17 uses a positive score weight in the workspace and engineering settings. With unchanged cost and no novelty bonus, a positive score fluctuation below the noise threshold can therefore still satisfy the acceptance condition. This needs reconciliation with the stated aim of preventing small fluctuations from being sufficient evidence for acceptance.

  3. The credit-assignment and pruning signals do not directly measure component utility. Appendix D.2 assigns the same candidate-level gain to every bundled edit. In addition, Equations 11 and 14 mark a previously explored component as a pruning target when it has no measurements within the recent window, because the maximum over an empty set is defined as negative infinity. A useful, stable component could therefore be targeted simply because it has not recently been modified. Although subsequent candidate evaluation provides a safeguard, the rationale for this pruning signal needs stronger validation.

  4. The efficiency evidence has a limited scope. RRSI uses 2.42 million policy tokens per trial compared with 3.80 million for unregularized evolution, but approximately 55% more than the initial harness’s 1.56 million. The paper acknowledges this increase. However, Figure 4 measures cost on the evolution split while reporting performance on OOD benchmarks, and the total search cost—including proposal generation, analysis, criticism, and candidate evaluation—is not quantified. These results establish a useful comparison among evolved harnesses, but not yet the full cost of obtaining and deploying the improvement.

  5. Several implementation details require correction or clarification. Equation 4 uses a ceiling operation, while Appendix D.1 specifies rounds t=0,…,T−1t=0,\ldots,T-1. With the reported parameters, the resulting budget never reaches one, contrary to the explanation that the closing rounds contain a single edit. Appendix D.3 also states that the weights ws,wc,wnw_s,w_c,w_n appear in Table 5, but they are absent. The domain-specific guards in Algorithm 2 are insufficiently specified. These details affect the actual search behavior and reproducibility.

Questions for authors
  1. Could you provide individual ablations of the edit schedule, leakage critic, noise floor, and pruning rule, together with a simpler cost-constrained baseline? Are the comparative gains stable across repeated baseline and ablation searches?

  2. How is the noise tolerance calibrated precisely, and how sensitive are the results to it? What happens when ws=0w_s=0 is also used for workspace and engineering tasks?

  3. How does pruning distinguish a component that remains useful from one that has merely not been edited recently? Have you compared the current history-based rule with direct removal evaluations?

  4. Which edit-budget schedule was actually implemented? Please provide the missing selection weights and domain guards, and reconcile the implementation with the stated single-edit closing rounds.

  5. What are the complete evolution costs and the inference costs on each OOD benchmark? Does RRSI retain its advantage when compared with the initial harness under matched inference budgets?

  6. Appendix B.5 specifies 185 GDPval tasks, evaluation in both presentation orders, and 204 comparisons per judge. Could you reconcile these counts and provide the exact evaluated task set? Also, could you report paired confidence intervals against the strongest evolution baseline, in addition to those against the initial harness?

6 comments
AuthorsOpenReview ↗

Response to Reviewer j5wL (1/N)

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.


Q1: The ablations do not isolate the mechanisms responsible for improved transfer. Table 2 removes proposal-side and acceptance-side regularizers as groups, but each group contains several distinct interventions. Consequently, it remains unclear whether the gains primarily come from leakage screening, conservative acceptance, smaller edits, pruning, or their interaction. Simpler controls would help establish the value of the complete framework.

A1. Following your advice, we conduct additional ablations on the agentic workspace instance, where each regularizer is removed from RRSI one at a time. Table R1 includes grouped ablations and three simple controls.

  • (a) Leakage screening. Disabling the leakage critic raises the Evolve score to 93.0 but lowers the OOD average to 41.6, compared with 43.6–43.7 for full RRSI.
  • (b) Conservative acceptance. Removing the noise floor lowers the OOD average to 42.4. Removing the cost rule yields 43.1 and raises token use to 3.25M per trial, compared with 2.32–2.42M for full RRSI.
  • (c) Smaller edits. Keeping the edit limit fixed throughout search yields 42.2 OOD. The greedy single-edit control scores 40.8–41.0 OOD across two runs.
  • (d) Pruning. Disabling pruning yields 43.2 OOD and increases token use to 2.67M per trial.
  • (e) Interaction. Removing all proposal-side or all acceptance-side regularizers yields 41.9 or 41.0 OOD, respectively. Each grouped removal gives a larger drop than any individual removal on the same side. The grouped ablations measure the effect of removing several components together. We have revised the paper and put these details in Section 4.3 and Appendix E.3.

Table R1. Detailed ablations. Entries with two values report run 1 / run 2.

VariantEvolveID held-outOOD avgTokens/trial (M)
H0H_089.486.939.71.56
Unregularized evolution92.888.940.33.80
Simple controls
Greedy, fixed single-edit budget91.7 / 92.088.8 / 88.640.8 / 41.03.01 / 3.12
Greedy, cost-penalized selection91.2 / 91.888.8 / 88.940.6 / 40.42.76 / 2.90
Greedy, early stopping on validation split92.5 / 92.088.6 / 88.340.2 / 40.53.24 / 3.06
Proposal side
RRSI w/o annealed budget91.488.742.22.79
RRSI w/o history91.388.843.02.54
RRSI w/o exploration91.089.042.52.64
RRSI w/o proposal regularizers90.788.841.92.69
Acceptance side
RRSI w/o leakage critic93.088.641.62.87
RRSI w/o noise floor90.688.842.42.49
RRSI w/o cost rule90.789.043.13.25
RRSI w/o pruning90.489.043.22.67
RRSI w/o acceptance regularizers91.588.741.03.59
RRSI90.5 / 90.889.2 / 88.843.6 / 43.72.42 / 2.32

Q2: The noise-handling rules provide weaker protection than some of the discussion suggests. A tolerance estimated from repeated evaluations of the initial harness does not by itself establish reliability under repeated adaptive selection. Moreover, Equation 17 uses a positive score weight in the workspace and engineering settings. With unchanged cost and no novelty bonus, a positive score fluctuation below the noise threshold can therefore still satisfy the acceptance condition. This needs reconciliation with the stated aim of preventing small fluctuations from being sufficient evidence for acceptance.

A2: Thank you for your valuable suggestions. (1) Adaptive selection. The tolerance δ\delta, estimated from repeated evaluations of H0H_0, characterizes empirical evaluation variability but does not provide a statistical guarantee under repeated adaptive selection. We now explicitly describe the noise-adjusted floor as a heuristic stability constraint on the observed evolution trajectory, rather than a multiple-testing correction. Specifically, anchoring the floor to the best observed retained score prevents a sequence of small measured regressions from accumulating beyond δ\delta, but does not guarantee that accepted edits improve true or held-out performance. (2) Within-band acceptance. The original positive within-band score weights can allow small observed score fluctuations to contribute to candidate acceptance. This rule is a heuristic performance–cost–novelty trade-off, not a statistical guarantee of rejecting noise-level improvements. We additionally rerun the workspace and engineering experiments with (w_s=0). The OOD averages remain 42.8 and 21.7, compared with 43.6 and 22.0 under the original settings. We have revised the paper and put these details in Appendix D.3 and E.3.

AuthorsOpenReview ↗

Response to Reviewer j5wL (2/N)

Q3: The credit-assignment and pruning signals do not directly measure component utility. Appendix D.2 assigns the same candidate-level gain to every bundled edit. In addition, Equations 11 and 14 mark a previously explored component as a pruning target when it has no measurements within the recent window, because the maximum over an empty set is defined as negative infinity. A useful, stable component could therefore be targeted simply because it has not recently been modified. Although subsequent candidate evaluation provides a safeguard, the rationale for this pruning signal needs stronger validation.

A3: Thank you for mentioning it. Our history-based signals are heuristic proxies for component utility, rather than direct causal estimates. On credit assignment, a candidate containing multiple edits produces a single performance measurement, which is associated with each constituent edit. This cannot disentangle their individual contributions or interactions. The annealed edit budget reduces the number of bundled changes in later rounds, making the feedback more attributable, but does not eliminate this ambiguity. We have clarified this limitation and now refer to these signals as candidate-level evidence associated with components, rather than direct measurements of component utility. On pruning, the original empty-window convention could incorrectly flag a previously useful component simply because it had not been modified recently. We correct the rule so that a component is considered a pruning target only if it has at least one measurement within the recent window and none of those measurements shows a positive gain. Components without recent measurements are not marked as unproductive. Importantly, pruning targets are only suggestions to the proposer, not automatic deletions. Any proposed removal must undergo candidate evaluation and satisfy the same stability, complexity, and domain-specific acceptance criteria. Thus, the history signal guides which components to reconsider, while the acceptance mechanism determines whether a deletion is retained. We have revised the paper and put these details in Appendix D.2.


Q4: The efficiency evidence has a limited scope. RRSI uses 2.42 million policy tokens per trial compared with 3.80 million for unregularized evolution, but approximately 55% more than the initial harness’s 1.56 million. The paper acknowledges this increase. However, Figure 4 measures cost on the evolution split while reporting performance on OOD benchmarks, and the total search cost—including proposal generation, analysis, criticism, and candidate evaluation—is not quantified. These results establish a useful comparison among evolved harnesses, but not yet the full cost of obtaining and deploying the improvement.

A4: Thank you for pointing this out.

  1. Deployment cost. Table 2 and Figure 4 report the final harness’s inference cost per trial on the agentic workspace evolve split. As shown in Table R2, RRSI uses 2.42M policy tokens per trial, compared with 3.80M for unregularized evolution and 1.56M for the initial harness.
  2. Evolution cost. We count all search-time LLM tokens on the coding instance, including candidate evaluation and calls to the proposer, analyst, critic, and digester where applicable. As shown in Table R3, RRSI uses a similar number of search tokens to Meta-Harness, AHE, and HarnessX, and substantially fewer than TTHE. We have revised the paper and put these details in Appendix E.4.

Table R2. Inference cost of the final harness on the agentic workspace evolve split.

Final harnessPolicy tokens per trial (M)
Initial harness (H0H_0)1.56
Unregularized evolution3.80
RRSI2.42

Table R3. Total search-time LLM token consumption on the coding instance, including candidate evaluation.

MethodTotal search tokens (M)
RRSI101
Meta-Harness107
AHE103
TTHE268
HarnessX99
AuthorsOpenReview ↗

Response to Reviewer j5wL (3/N)

Q5: Several implementation details require correction or clarification. Equation 4 uses a ceiling operation, while Appendix D.1 specifies rounds (t = 0, ..., T-1). With the reported parameters, the resulting budget never reaches one, contrary to the explanation that the closing rounds contain a single edit. Appendix D.3 also states that the weights (w_s, w_c, w_n) appear in Table 5, but they are absent. The domain-specific guards in Algorithm 2 are insufficiently specified. These details affect the actual search behavior and reproducibility.

A5: Thank you for your detailed suggestions.

  1. Edit-budget schedule. The implementation follows the ceiling-based schedule in Equation 4 with rounds indexed by t=0,…,T−1t=0,\ldots,T-1. Specifically, the coding instance uses budgets of 4, 3, and 2 edits across rounds 0–7, 8–12, and 13–19; the agentic workspace instance uses 3 edits in rounds 0–9 and 2 in rounds 10–19; and engineering design uses 4, 3, and 2 edits across rounds 0–15, 16–24, and 25–39. Thus, the actual schedule progressively restricts bundled edits but ends at two edits rather than one.
  2. Selection weights. Equation 17 uses the weights below (Table R4). One extra satisfied criterion adds about 0.1 to the within-band score term in workspace, and one extra passing trial adds 1 in engineering design. As shown in Table R5, setting ws=0w_s=0 removes within-band score credit. OOD performance drops by 0.8 points in workspace and 0.3 in engineering design, with both variants still outperforming their initial harnesses.
  3. Domain-specific guards. Only the engineering-design instance defines an additional guard. A candidate is rejected if its valid-submission rate decreases by more than 0.03 or its no-submission rate increases by more than 0.02 relative to the incumbent, regardless of its primary score. These guards prevent improvements in pass rate from compensating for substantial degradation in execution validity. Coding and agentic workspace use no additional guard, so g=1g=1 for all candidates.

We report the weights in Appendix D.3 and the sensitivity results in Appendix E.3.

Table R4. Within-band weights in Equation 17.

WeightCodingAgentic workspaceEngineering design
Score wsw_s01414244
Cost wcw_c15152
Novelty wnw_n0.50.50.5

Table R5. Sensitivity to ws=0w_s=0. Each cell shows the original setting → ws=0w_s=0.

DomainEvolveID held-outOOD avg.
Agentic workspace90.5 → 90.389.2 → 88.743.6 → 42.8
Engineering design54.9 → 55.3—22.0 → 21.7

Q6: Could you provide individual ablations of the edit schedule, leakage critic, noise floor, and pruning rule, together with a simpler cost-constrained baseline? Are the comparative gains stable across repeated baseline and ablation searches?

A6: We provide individual ablations of the edit schedule, leakage critic, noise floor, and pruning rule, together with a simple cost-penalized greedy baseline. All variants use the same initial harness, policy, evolve split, and search budget. As shown in Table R6, RRSI and the simple baseline were repeated with independent proposer seeds; their OOD scores differ by at most 0.2 points across the two runs. We have revised the paper and put these details in Section 4.3 and Appendix E.3.

Table R6: Key ablations and conservative baseline (agentic workspace).

VariantOOD Avg.
RRSI43.6 / 43.7
w/o annealed budget42.2
w/o leakage critic41.6
w/o noise floor42.4
w/o pruning43.2
Greedy, cost-penalized selection40.6 / 40.4

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. RRSI regularizes harness search on a finite evolve set to select compact, reusable edits. We added ablations, cross-domain baselines, sensitivity analyses, and cost comparisons.

G1. What contributes to transfer?

We ablated seven regularizers individually on agentic workspace and ran RRSI and three greedy controls with two independent proposer seeds each. As shown in Table G1, each method varies by at most 0.3 OOD points across runs. Removing the leakage critic raises the evolve score from 90.5 to 93.0 and lowers OOD from 43.6 to 41.6. Removing the annealed budget, noise floor, or structured exploration lowers OOD by 1.1–1.4 points. Removing the cost rule or pruning increases inference tokens by 34% or 10%, respectively. Each individual ablation was run once. A paired task-level bootstrap against Meta-Harness, the strongest workspace evolution baseline, gives a +3.0-point OOD gain (95% CI [+2.2, +4.0]), with intervals above zero on all three OOD benchmarks. (Section 4.3 and Appendices E.2–E.3.)

Table G1. Workspace OOD performance across two searches.

MethodRun 1Run 2
Greedy, fixed single-edit budget40.841.0
Greedy, cost-penalized selection40.640.4
Greedy, validation-based early stopping40.240.5
RRSI43.643.7

G2. Cross-domain transfer and TTHE adaptation

As shown in Table G2, RRSI leads the strongest competing baseline by 1.1 points on SWE-bench Verified and 1.6 on Frontier-Eng. With additional JobBench traces and compute, TTHE improves from 35.2 to 40.2 on JobBench and from 38.0 to 42.1 on the three-benchmark average. Frozen RRSI averages 43.6. TTHE adapts only on JobBench in this supplementary experiment. (Appendices C.1 and E.5.)

Table G2. Cross-domain comparison under frozen transfer.

MethodTerminal-Bench 2.1 (Evolve)SWE-bench Verified (OOD)EngDesign (Evolve)Frontier-Eng (OOD)
Meta-Harness81.282.753.820.2
AHE78.681.853.018.3
TTHE79.782.254.519.0
HarnessX79.282.755.020.4
RRSI80.283.854.922.0

G3. Calibration and sensitivity

For a new domain, start with the structural defaults, estimate noise from repeated evaluations of the unchanged initial harness on the evolve set, and set the cost allowance from the deployment budget. The noise floor constrains measured scores and provides no statistical guarantee under adaptive selection. As shown in Table G3, all tested workspace settings beat unregularized evolution (40.3 OOD). Varying β₁ changes inference cost from 2.12M to 2.70M tokens per trial. Both zero-weight variants still beat their initial harnesses. (Section 3.3, Appendices D.3 and E.1)

Table G3. One-at-a-time sensitivity.

ParameterTested settingsOOD average
Noise tolerance δ, workspace0 / 0.5δ₀ / δ₀ / 2δ₀, with δ₀ = 0.00442.4 / 42.2 / 43.6 / 41.9
Cost allowance β₁, workspaceHalf / default / double42.9 / 43.6 / 44.2
Stall window w, workspace2 / 3 (default) / 643.2 / 43.6 / 43.4
Within-band score weightDefault → zeroWorkspace 43.6 → 42.8, engineering 22.0 → 21.7

G4. Search and deployment cost

A full coding search uses 101M LLM tokens for RRSI, including candidate evaluations and auxiliary calls, versus 107M for Meta-Harness, 103M for AHE, 268M for TTHE, and 99M for HarnessX. On the workspace evolve split, RRSI uses 2.42M policy tokens per trial, versus 3.80M for unregularized evolution and 1.56M for the initial harness. RRSI also uses fewer tokens than unregularized evolution on each workspace OOD benchmark. Giving the unchanged initial harness more turns raises its cost to 2.38M per trial, close to RRSI’s 2.42M. At these costs, RRSI scores 89.2 vs. 87.5 on ID held-out and 43.6 vs. 40.4 on OOD. (Section 4.3 and Appendix E.4.)

AuthorsOpenReview ↗

Summary of Paper Revision

We sincerely appreciate all reviewers for their insightful and constructive feedback. According to these comments, we have improved the paper (new pdf uploaded) and highlighted the main changes with blue text. Below, we summarize all changes:

  1. Component ablations and simpler controls (Section 4.3 and Appendix E.3). We added seven single-component ablations and three greedy controls covering a fixed single-edit budget, cost-penalized selection, and validation-based early stopping. We report both independent runs for full RRSI and the controls, and specify which ablations were run once.
  2. Baseline coverage and protocols (Appendices C.1 and E.5). We documented the porting and search protocols for Meta-Harness, AHE, TTHE, and HarnessX, and extended the comparison to coding and engineering design. We clarified the frozen-transfer protocol for TTHE and added a separate continued-adaptation experiment using JobBench traces.
  3. Statistical evidence (Appendix E.2). We extended the paired task-level bootstrap to compare RRSI directly with Meta-Harness. The workspace OOD-average improvement is +3.0 points, with a 95% confidence interval of [+2.2, +4.0].
  4. Search and deployment costs (Section 4.3 and Appendix E.4). We clarified the scope of the inference-cost figure, added full coding search-token totals and per-benchmark workspace OOD inference costs, and compared RRSI with the initial harness at approximately matched inference cost.
  5. Calibration and sensitivity (Section 3.3 and Appendices D.3 and E.1). We clarified how the empirical noise tolerance is calibrated, what the noise floor constrains, and how the within-band rule uses measured score changes. We added guidance for configuring a new domain and sensitivity results for the noise tolerance, gain-dependent cost allowance, and stall window.
  6. Edit schedule, pruning, and domain guards (Section 3.2 and Appendices D.1–D.3 and E.1). We corrected the description of the realized edit schedule, which ends at two edits per candidate in the reported settings. We revised the pruning rule to require recent measured evidence before flagging a component and specified the engineering-design guard for valid-output and no-submission rates.
  7. Evaluation and headline figures (Abstract, Introduction, and Appendix B.5). We corrected the maximum OOD gain over the initial harness to 4.7 points and the workspace inference-token reduction versus unregularized evolution to approximately 36%. We also clarified that each GDPval judge evaluates 185 pairs in randomized order and re-evaluates 19 in reverse order, for 204 comparisons per harness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.