acceptodds
Under review as a conference paper at ICLR 2027

RSIBench: Can AI Agents Improve Their Own Harnesses?

Abstract

Recursive self-improvement (RSI) promises a simple yet powerful loop: an AI system improves itself, and the modified version continues the process. As agent harnesses increasingly shape how language models use context, tools, and feedback, evolving those harnesses offers a plausible near-term way to realize this loop. But can today's agents improve their own harnesses? Do these gains generalize to unseen tasks? Which harness changes help? To answer these questions, we introduce RSIBench, a benchmark that tests whether agents can improve their own harnesses. At each iteration, an agent uses task trajectories to revise its open-source Pi harness , while the model weights remain fixed. If a harness revision improves performance on training tasks, it replaces the current harness. The agent then uses the revised harness to solve tasks and propose the next revision. Unlike most prior work, we evaluate the final harness on held-out tasks. We evaluate seven models on 100 tasks across Terminal, Environment Learning, SWE, and Science. We define RSI lift as the final harness's held-out score minus the initial harness's held-out score. Claude Opus 5 achieves an RSI lift of +8.3 percentage points (36.4→44.7) across all held-out tasks. However, for GPT-5.5 the gains are less clear: it improves on Science training tasks but loses 8.3 points on held-out tasks (11.1→2.8), and cross-domain transfer is uneven, underscoring the importance of evaluating RSI on held-out tasks. Overall, RSIBench tests a concrete step toward RSI: whether a model can improve its own harness and, crucially, generalize those gains to unseen tasks.

Primary area:
datasets and benchmarks
Keywords:
recursive self-improvement, agent harnesses, self-evolving agents, benchmark, harness engineering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JJzCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RSIBench, a benchmark for evaluating recursive harness modification using frozen LLMs. Starting from the Pi harness, an agent uses training trajectories to edit its prompts, tools, and execution logic. Accepted edits are inherited for both solving future tasks and generating subsequent edits. The benchmark spans 100 tasks (50/50 train/test split) across four domains, using "RSI lift" (held-out score difference between final and initial harnesses) as the main metric. The authors evaluate seven models, analyze cross-domain transfer, ablate editable modules, and run 30-round extensions, alongside detailed trajectory case studies.

Critical strengths

An important question and concrete setup.

Testing whether an agent can turn interaction experience into lasting system improvements is important for stable post-deployment adaptation. RSIBench provides a practical testbed with a broad editable surface, strict held-out evaluation, and an inheritance loop where updated harnesses drive future edits.

Insightful failure-mode analysis.

Appendix H nicely separates fixing a real bug from fixing the actual bottleneck. In the failure case, the agent patches a genuine Bash-runtime issue, yet that code branch is never even triggered during the diagnosed task's rollouts. Contrasting this with a successful stopping-rule fix clearly illustrates why valid code edits and higher training scores don't automatically translate into generalizable improvements.

Critical weaknesses

Main concern: RSI lift doesn't test the recursive part.

RSI lift compares the held-out scores of the first and last harness. A proposer that never changes could produce the same lift, so the metric can't tell us whether the system got better at improving itself. Edits are inherited, but the paper never checks whether inheriting them matters. This is the central question, since Evo-Bench and SEAGym already cover harness evolution and transfer, and the recursive loop is what RSIBench adds. Two controls would settle it: freeze the proposer at round 0 while keeping the editable surface, feedback and budget unchanged, or start the initial and evolved proposers from the same harness and compare their edits.

I also suspect the greedy gate. It keeps only edits that raise training scores right away, so an edit that only makes later diagnosis or patching better gets thrown out. That may be why 30 rounds stall, and the paper doesn't trace where the stall comes from. Given DGM, HGM and Hyperagents, I'd expect a benchmark in this space to isolate meta-improvement more directly.

Second concern: the numbers are too noisy to rank.

In Table 2, Opus 5 Agent Loop looks like +13.33 against the main baseline, but its own matched initial-to-final score drops from 53.33 to 46.67. Appendix E.2 reports a 15. I appreciate the transparency, but at this level of noise I can't trust even the sign of many module and transfer gains. These claims need multiple runs with error bars.

Questions for authors
  • Did you test freezing the proposer at its initial state while keeping the full editable surface, feedback, and budget? Or did you compare initial vs. evolved proposers starting from the same harness to see if the evolved one generates better edits?

  • Did you observe rejected edits that improved diagnostics or code generation without immediately raising training scores? Could this strict gate explain the 30-round stagnation?

  • Which module and transfer claims still hold after accounting for baseline noise? Can you share matched initial-vs-final scores or error bars for the main claims?

3 comments
AuthorsOpenReview ↗

(1/2) Author Responses

Dear reviewer JJzC,

We sincerely thank you for your detailed and constructive review. We are especially grateful that you recognized the importance of turning interaction experience into long-lasting system improvements, our strict held-out evaluation and the design of inheritance loop where updated harnesses do future edits, and the failure-mode analysis that separates fixing a real bug from fixing the actual bottleneck. We address each concern below.

W1 and Q1. Is the recursive part tested?

Yes. The proposer (the agent writing each revision) in round tt runs on the current harness HtH_t (Section 3.1), so after an accepted revision, an edited harness proposes the next one. We test whether this helps with per-run proposer records (subsection 1) and provides an experiment about frozen-proposer (subsection 2).

Edits accepted in earlier rounds also run inside later proposers: GPT-5.5 Terminal accepted a Bash tool revision during round-3 that appends the duration of long-running commands, and the round-4 and round-5 proposers' own build checks end with notes such as "[Command duration: 14.1s]".

1. Initial proposer versus evolved proposer

In the table below, both H0H_0 and an edited harness proposed accepted revisions. Each row compares the two proposers under the same model, tasks and acceptance rule. The edited harness proposes the revision with larger gains.

RunFirst revision, proposed by H0H_0 (Δ\Delta training score)Later revisions, proposed by the edited harness (Δ\Delta training score)Δ\Delta Held-out score
Kimi K3 Science+5.56+13.89+11.11
GPT-5.6-Sol Terminal+2.22+11.11+2.22
GLM-5.3 Environment Learning+2.35+4.09+7.02
Claude Opus 5 Environment Learning+0.01+3.18+3.32
GPT-5.5 Terminal+2.22+4.44+4.44
GPT-5.5 SWE+4.17+4.17+8.33
Kimi K3 Environment Learning+0.82+3.48+10.17
GPT-5.5 Environment Learning+0.39+0.93-0.40

2. Frozen-proposer experiment

We also ran an experiment comparing recursive self-improvement (where the proposer runs on the current, edited harness) with iterative improvement (where the proposer always runs on the initial harness H0H_0). We evaluate GPT-5.6-Sol on Science tasks, and both runs use the same rounds and editable modules. The recursive run ends at a training score of 16.67% and the iterative run at 8.33%.

SettingProposer runs onAccepted revisionsFinal training score
Recursive self-improvementThe current, edited harnessRound 1, Round 216.67%
Iterative improvementH0H_0 in every roundRound 38.33%

3. What RSIBench adds

The table below adds the recursive self-improvement methods to the comparison of Table 12. RSIBench focuses on recursive self-improvement, where the evolved harness proposes the next revision.

WorkTypeWho proposes revisionsEvaluation
DGM [1]MethodAn agent from an archive edits its own code, guided by a fixed model's diagnosisSWE-bench and Polyglot
HGM [2]MethodThe coding agent; variants chosen by how much their descendants improveCoding
Hyperagents [3]MethodSelf-editing meta agentCoding, review, robotics, math
Self-Harness [4]MethodCurrent harness; training and held-out scores both decide which edits are keptTerminal
RHI [5]MethodFixed LLM optimizer, pairwise feedbackSynthetic ML research tasks
Evo-Bench [6]BenchmarkSeparate, fixed evolverSearch, Office, General
SEAGym [7]BenchmarkMethod-specific update ruleTerminal-Bench and HLE
RSIBenchBenchmarkAccepted harness proposes the next revision7 models, 4 domains; final test never used for selection; frozen harness tested on other domains

Q2. Can a rejected diagnostic edit still help later rounds?

Yes. A diagnostic edit makes failures easier to see without fixing them, so it may not raise the training score, and the acceptance gate then rejects it. In a GPT-5.6-Sol run on Terminal tasks, the gate rejected diagnostic edits in rounds 2 and 5. We ran the proposer three times on each of these two harnesses and three times on the initial harness, and checked whether each proposed patch passes the local build and command-line check. So a diagnostic edit that the gate rejects still helps later proposals build and run.

Proposer runs onProposals that pass the build and command-line check
A harness carrying the rejected diagnostic edit6 of 6
The initial harness1 of 3
AuthorsOpenReview ↗

(2/2) Author Responses

W1 and Q2. Why the 30-round run stops improving

The acceptance gate keeps a candidate revision when, compared with the current harness, it raises the mean training score, improves at least two training tasks and causes no severe drop on a solved task. In the 30-round GPT-5.6-Sol run on Terminal tasks (Table 6), the training score stops rising after round 4 because no later candidate scored higher. Table 6 records each round's scores and decisions: none of the 26 candidates after the round-4 acceptance scored above the current harness (80.00). After round-4, the harness already has good performance and further self-improvement becomes more difficult.

W2 and Q3. Which claims hold after accounting for noise

Two runs of the same harness can differ. RSI lift (final minus initial held-out score) across Table 1, the Opus 4.8 transfer gain on Science and the model dependence of transfer gain all hold after accounting for this noise.

1. RSI lift

If RSI lift were noisy, scores would rise and fall equally often, however large the noise. We analyze the RSI lift below, and the direction of RSI lift holds across models and domains.

7 models × 4 domains = 28 resultsResult
Results that change between initial and final harness20
Rises among these 2016
Falls4
Mean change over the 28 results+3.92 points, 95% bootstrap interval [+2.1, +5.8]
Models with positive mean RSI lift7 of 7
Domains with positive mean RSI lift4 of 4

2. Transfer acoss domains

Figure 4 applies three models' frozen final harnesses to the other three domains. Resampling the held-out tasks (paired bootstrap, 10,000 resamples) are provided below.

Frozen harness and targetΔ\Delta (points)95% interval
Opus 4.8 SWE harness → Science+22.22 (16.67 → 38.89)[+5.56, +41.67]
Opus 4.8 Terminal harness → Science+16.67 (16.67 → 33.33)[+2.78, +33.33]

Both Opus 4.8 harnesses, one from SWE and one from Terminal, raise the Science score, and the low end of each interval stays above zero (+5.56 and +2.78 points). So the Opus 4.8 transfer gain on Science holds under task resampling.

3. Modules

RSIBench checks every module claim in two ways. First, Table 2 compares each final harness with the main-experiment baseline. Second, for a single edit, the stored rollouts, including the messages the harness inserts into them, show whether the edited code actually ran. In this way, we can observe RSI lift when editing specific harness modules.

  • Opus 5 (Appendix H.2): the edit adds a checkpoint message, and this message appears in the following rollout. The edited code actually ran, so the edit ran and the gain belongs to it.
  • GPT-5.5 (Appendix H.3): the edited code runs in 1 of the 45 training rollouts, so its +4.44-point training gain came from rollout noise, not from the edit. Its training score rose from 33/45 to 35/45 passes (+4.44 points) by chance.

So RSIBench credits a RSI gain to a module edit when the edited code ran, and catches gains that come from noise.

We sincerely appreciate your detailed and thoughtful review. Our additional experiments show that a recursive harness proposer achieves twice the training score of a frozen proposer. We also find that rejected diagnostic edits still help subsequent harness proposals build and run successfully. Furthermore, both the RSI lift and the domain transfer gain remain robust after accounting for rollout noise. We hope these additional results address your concerns, and we would appreciate your reconsideration of our paper.

[1] Zhang, Jenny, et al. "Darwin Gödel machine: Open-ended evolution of self-improving agents." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Wang, Wenyi, et al. "Huxley-g" odel machine: Human-level coding agent development by an approximation of the optimal self-improving machine." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Zhang, Jenny, et al. "Hyperagents." arXiv preprint arXiv:2603.19461 (2026).

[4] Zhang, Hangfan, et al. "Self-harness: Harnesses that improve themselves." arXiv preprint arXiv:2606.09498 (2026).

[5] Lee, Hyunin, et al. "Recursive Harness Self-Improvement." arXiv preprint arXiv:2607.15524 (2026).

[6] Huang, Lisheng, et al. "Evo-Bench: Can language models improve agent harness?." arXiv preprint arXiv:2608.09096 (2026).

[7] Zheng, Congjie, et al. "SEAGym: An Evaluation Environment for Self-Evolving LLM Agents." arXiv preprint arXiv:2606.17546 (2026).

Dear Reviewer JJzC,

Thank you again for your detailed and constructive feedback. As the discussion period wraps up, we would appreciate your thoughts on our responses, including the frozen-proposer control experiment, the analysis of rejected diagnostic edits, and the bootstrap confidence intervals confirming that our RSI lift and transfer gains remain robust after accounting for rollout noise. If these results resolve your concerns, we would sincerely appreciate it if you could consider raising your score.

Reviewer_tj8oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RSIBench, a framework that measures how much LLM agents can achieve recursive self-improvement by modifying their own harness without modifying the LLM weights. They evaluate seven LLMs on 100 tasks and ensure the final evaluation is measured on a held out set of tasks so agents don’t overfit on their eval set.

Critical strengths

An important aspect is that the held out set of tasks is completely not part of the optimization process and is only used for evaluation at the end unlike ‘self-harness’.

Critical weaknesses

The paper’s main advantage over ‘self-harness’ [1] (which is evaluating the final solution on a held out set of tasks) seems like a minor change to the evaluation that does not justify an entirely new evaluation. The paper lacks comparison with ‘self-harness’ and a demonstration that the final held out set fixes a problem in results obtained by ‘self-harness’.

Darwin Gödel Machines (DGM) [2] seem to cover the concept of a continuously evolving LLM harness that is then evaluated on held-out tasks in addition to more evaluations.

The evaluation lacks a comparison with an agent autoresearching harness improvements without using the harness improvements it found (i.e. non recursive). If the benefits in this setup are close to or exceed RSIBench results then RSIBench isn’t truly measuring a form of recursive self improvement capabilities.

[1] Zhang, H., Zhang, S., Li, K., Zhang, C., Chen, Y., Zhang, Y., ... & Hu, S. (2026). Self-harness: Harnesses that improve themselves. arXiv preprint arXiv:2606.09498.

[2] Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2026, April). Darwin gödel machine: Open-ended evolution of self-improving agents. In International Conference on Learning Representations (Vol. 2026, pp. 104223-104294).

Questions for authors

Does RSIBench offer benefits that ‘self-harness’ with a final held out task-set does not offer? What about DGM? What fundamental insight does RSIBench establish beyond those previous methods/benchmarks?

Can the authors provide evidence that the model at time t proposing a change from t to t+1 is more powerful than a base model proposing a change from t to t+1. Thus, beyond model t getting higher benchmarks than model 0, does it also consistently make better proposals than previous time steps?

Questions do not affect my recommendation:

Can the authors provide plots where x-axis is time/iteration and y-axis is score.

Can the authors update table 12 to include recursive benchmarks, currently only RSIBench is the recursive benchmark listed in the table while previous recursive benchmarks exist.

3 comments
AuthorsOpenReview ↗

(1/2) Author Responses

Dear reviewer tj8o,

We sincerely thank you for your careful and insightful review. We are especially grateful that you highlighted how RSIBench keeps the held-out tasks completely out of the optimization process and uses them for the final evaluation. We address each concern below.

W1. What the untouched final test adds beyond Self-Harness

Self-Harness also scores edits on held-out tasks, but it uses those scores to decide which edits to keep, and scores used for selection overstate the gain. RSIBench keeps its final test out of selection process: training scores select revisions, and no held-out task runs until the RSI rounds end (Section 3.2). Across the runs of Section 4.1, the final harnesses gain 145.3 training points in total over the initial ones and 87.0 held-out points, which shows that the self-improved harness can also perform well on tasks it has never seen before.

In five runs, accepted revisions raised the training score while the final harness stayed at or below its initial held-out score:

RunTraining change (points)Held-out change (points)
GPT-5.5 Science+5.56-8.33
Claude Opus 4.8 Terminal+13.330.00
GPT-5.6-Sol Environment Learning+5.99-0.74
GPT-5.5 Environment Learning+1.31-0.40
Claude Opus 4.8 Environment Learning+0.67-0.76

Judged by training scores alone, all five self-improved harnesses look like successful improvements but cannot generalize to unseen tasks. RSIBench's final test never participates in deciding which revisions to keep. Self-Harness uses held-out scores to select edits, so such false positive gains stay hidden in its results.

W3 and Q2. Does the edited harness propose better revisions than the initial one?

Yes. In RSIBench, the proposer (the agent that writes the revision) runs on the current harness in each round, so after an accepted revision, the evolved harness proposes the next one. The edited harness proposes the revision with larger gains. Each row compares the two proposers in the same run, using the same model, tasks and acceptance rule:

RunFirst revision, proposed by H0H_0 (Δ\Delta training score)Later revisions, proposed by the edited harness (Δ\Delta training score)Δ\Delta Held-out score
Kimi K3 Science+5.56+13.89+11.11
GPT-5.6-Sol Terminal+2.22+11.11+2.22
GLM-5.3 Environment Learning+2.35+4.09+7.02
Claude Opus 5 Environment Learning+0.01+3.18+3.32
GPT-5.5 Terminal+2.22+4.44+4.44
GPT-5.5 SWE+4.17+4.17+8.33
Kimi K3 Environment Learning+0.82+3.48+10.17
GPT-5.5 Environment Learning+0.39+0.93-0.40
AuthorsOpenReview ↗

(2/2) Author Responses

W2 and Q1. How RSIBench differs from DGM and Self-Harness

Self-Harness and DGM are both improvement methods: they propose one search procedure and show that it improves an agent. DGM evolves a coding agent's code and tools with an archive of agent variants on SWE-bench and Polyglot. RSIBench is a benchmark: every model starts from the same initial harness (the Pi coding agent), can edit the same 18 modules, and follows the same acceptance rule, budgets, tasks, splits and verifiers (Section 3), across four domains. Among benchmarks, Evo-Bench also holds out tasks but proposes edits with a separate, fixed harness; in RSIBench the inherited harness proposes each next revision. RSIBench focuses on recursive self-improvement. Our findings are summarized below.

FindingEvidence
Agents improve their own harnesses on unseen tasksAll seven models have positive mean RSI lift (final minus initial held-out score)
RSI lift ranks models differently from their initial scoreSpearman ρ = -0.018 between initial score and RSI lift (Figure 5)
The same module edit helps one model and hurts anotherPrompt Loader edits on Science: Opus 4.8 +11.11, GPT-5.5 -8.33 points

Q3. Does the paper show scores by round?

Yes. Figure 1 (middle) demonstrates the training score over five rounds for Terminal runs, and Table 6 lists all rounds of a longer, 30-round GPT-5.6-Sol run. The training score of the current harness after each round is summarized below. The mean training score rises in every round.

Round012345
Mean training score (%)38.6540.4941.7543.0543.9744.52
Revisions accepted in the round105665

Q4. Recursive self-improvement methods in Table 12

The table below adds the recursive self-improvement methods to the comparison of Table 12.

WorkTypeWho proposes revisionsEvaluation
DGM [1]MethodAn agent from an archive edits its own code, guided by a fixed model's diagnosisSWE-bench and Polyglot
HGM [2]MethodThe coding agent; variants chosen by how much their descendants improveCoding
Hyperagents [3]MethodSelf-editing meta agentCoding, review, robotics, math
Self-Harness [4]MethodCurrent harness; training and held-out scores both decide which edits are keptTerminal
RHI [5]MethodFixed LLM optimizer, pairwise feedbackSynthetic ML research tasks
Evo-Bench [6]BenchmarkSeparate, fixed evolverSearch, Office, General
SEAGym [7]BenchmarkMethod-specific update ruleTerminal-Bench and HLE
RSIBenchBenchmarkAccepted harness proposes the next revision7 models, 4 domains; final test never used for selection; frozen harness tested on other domains

RSIBench focuses on recursive self-improvement, where the evolved harness proposes the next revision.

We sincerely appreciate your careful review and insightful feedback. Our analyses show that keeping final test hidden during the training process exposes training gains that disappear on unseen tasks. We also demonstrate that revisions proposed by an edited harness gain more than those proposed by the initial harness. RSIBench measures whether a model's own edited harness keeps improving itself on tasks it never saw. We hope these results address your concerns, and we would be grateful for your reconsideration of our work.

[1] Zhang, Jenny, et al. "Darwin Gödel machine: Open-ended evolution of self-improving agents." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Wang, Wenyi, et al. "Huxley-g" odel machine: Human-level coding agent development by an approximation of the optimal self-improving machine." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Zhang, Jenny, et al. "Hyperagents." arXiv preprint arXiv:2603.19461 (2026).

[4] Zhang, Hangfan, et al. "Self-harness: Harnesses that improve themselves." arXiv preprint arXiv:2606.09498 (2026).

[5] Lee, Hyunin, et al. "Recursive Harness Self-Improvement." arXiv preprint arXiv:2607.15524 (2026).

[6] Huang, Lisheng, et al. "Evo-Bench: Can language models improve agent harness?." arXiv preprint arXiv:2608.09096 (2026).

[7] Zheng, Congjie, et al. "SEAGym: An Evaluation Environment for Self-Evolving LLM Agents." arXiv preprint arXiv:2606.17546 (2026).

Dear Reviewer tj8o,

Thank you again for your review. With the discussion period nearing its end, we would appreciate it if you could take a look at our responses, especially our analysis of why an untouched final test set matters beyond Self-Harness, the evidence that evolved harnesses propose stronger revisions, and our updated Table 12 comparison with DGM and related work. If these address your main concerns, we would be grateful if you could consider raising your rating. We are happy to answer any remaining questions.

Reviewer_xCATOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Summary: RSIBench introduces a benchmark designed to evaluate whether AI agents can recursively improve their own harnesses, which comprise the code surrounding the model that manages context, tools, verification, and execution, while model weights remain fixed. The principal contribution lies in testing not merely whether agents can propose improvements but whether those improvements persist and generalize to unseen tasks. The benchmark employs 100 tasks distributed across four domains including Terminal, Environment Learning, SWE, and Science, with 50 training tasks designated for proposing and selecting revisions and 50 held-out tasks reserved for final evaluation. Each model initializes from the same open-source Pi harness and may revise 18 designated modules across five rounds. Results demonstrate positive average RSI lift, defined as held-out score improvement, for all seven models tested, with Claude Opus 5 achieving the highest improvement at +8.33 percentage points. However, the results exhibit considerable variability. GPT-5.5 improves on Science training tasks but loses 8.33 points on held-out tasks, and cross-domain transfer varies substantially depending on model and source domain.

Contributions:

  1. RSIBench benchmark: A concrete test for recursive harness self-improvement in which revised harnesses participate in producing their subsequent revision, with held-out evaluation to measure generalization capability.
  2. Empirical findings on seven frontier models: The paper documents that RSI lift ranges from +0.55 to +8.33 points across models, with Claude Opus 5 improving in all four domains. The analysis reveals that training-task gains do not consistently transfer to held-out tasks, as 5 of 19 runs with training gains exhibit no held-out improvement.
  3. Cross-domain transfer analysis: The paper demonstrates that SWE-evolved harnesses can improve performance in other domains. For instance, Claude Opus 4.8's SWE harness gains +15.56 points on Terminal and +22.22 points on Science. However, transfer effectiveness is model-dependent.
  4. Model-specific harness preferences: The analysis shows that evolution of the same harness module benefits certain models while harming others. Hooks provides +11.11 points for GPT-5.6-Sol whereas Skills provides +31.11 points for Opus 5, thereby informing future harness research directions.

Validation: The paper validates its claims through multiple approaches. The evaluation encompasses 100 tasks across four heterogeneous domains with fixed train/test splits. Seven frontier models are tested under consistent protocols. Detailed trajectory analysis examines both success and failure cases, including Opus 5's completion check and GPT-5.5's misattributed edit. Cross-domain transfer experiments are conducted for three models. Single-module ablations are performed on Terminal tasks. Extension experiments spanning 30 rounds test longer evolutionary horizons. Cost comparisons with estimated post-training expenditures are provided.

Critical strengths
  1. The held-out evaluation design is genuinely helpful : Unlike prior harness-evolution work that evaluates performance on training tasks or employs fixed meta-agents, RSIBench mandates that revised harnesses be evaluated on unseen tasks that never influence proposal or selection decisions. This methodological choice matters because it exposes a critical failure mode. The paper documents multiple instances in which training-task improvements lead to held-out regressions. GPT-5.5 Science exhibits +5.56 training improvement followed by -8.33 held-out regression. Without held-out evaluation, these cases would incorrectly appear as successful instances of RSI. The detailed failure case presented in Appendix H.3 demonstrates how GPT-5.5's ostensibly correct bug fix passed training selection yet never activated in the passing rollout and subsequently regressed on held-out tasks. This example is particularly valuable because it reveals that the selection mechanism itself can be deceived by rollout variance. This design choice directly determines the paper's primary conclusion, which is that measuring RSI requires testing generalization rather than merely assessing training performance.

  2. The recursive design requirement is designed conceptually: The paper rigorously distinguishes recursive self-improvement from iterative improvement by requiring that the revised harness participate in producing its subsequent revision. This distinction matters because it tests whether the improved system can generate further improvements, which constitutes a core requirement for the RSI concept. The formal framework presented in Equations 9-12 and the two-layer editable surface design, comprising shared resources and core source, operationalize this requirement cleanly. The finding that 30-round extensions yield few additional accepted revisions, with GPT-5.6-Sol accepting 2 revisions in 30 rounds and Opus 5 accepting 0 revisions, suggests that sustained recursive improvement remains challenging even for frontier models. This result directly supports the paper's positioning as offering a concrete step toward measuring RSI capability rather than claiming to demonstrate sustained RSI.

Critical weaknesses
  1. The training-to-held-out gap is inadequately analyzed: The paper reports that training gains total 145.3 percentage points versus 87.0 points on held-out tasks, yielding a 60% retention ratio, yet offers limited insight into why this gap exists or how to predict it. The failure case analysis is valuable but restricted to two examples. The paper does not systematically investigate which types of harness changes are more likely to generalize, whether the gap stems from overfitting to specific task structures or from task-specific bugs in the revised harness, and whether the gap diminishes with more training tasks. The training set sizes of 15, 8, and 12 tasks per domain may simply be insufficient for generalization, but the paper does not test this hypothesis. Without understanding the retention mechanism, the benchmark provides limited guidance for improving RSI systems because it measures the gap without explaining it.

  2. The acceptance criterion is problematic for the stated goals : The training-side selection rule, which requires higher mean score, at least two improved tasks, and no severe regression on solved tasks, accepts revisions based on noisy training-rollout scores without requiring mechanistic validation. The GPT-5.5 failure case explicitly demonstrates this issue. The accepted edit's mechanism never activated in the passing rollout, yet aggregate fluctuation drove acceptance. The paper acknowledges this limitation but does not propose alternatives. More problematically, the rule compares the candidate's newly measured scores against the current harness's saved scores from when it was accepted rather than remeasured scores. This protocol design means that rollout noise can change selection decisions asymmetrically because a candidate benefits from favorable variance while the incumbent is evaluated against stale scores. This protocol design may be causing the training/held-out gap that the paper documents, yet the paper does not test whether remeasuring both harnesses would improve retention.

Questions for authors
  1. Retention analysis: You observe 60% retention of training gains to held-out tasks. Have you analyzed whether this ratio correlates with specific properties of the accepted revisions, such as which module was edited, the size of the diff, or whether the edit added versus modified code? Such analysis would help distinguish whether the gap stems from overfitting, task-specific bugs, or selection-noise issues.

  2. Training set size sensitivity: Your domain training sets contain 15, 15, 8, and 12 tasks. Does the retention ratio improve with larger training sets? A straightforward experiment running Terminal with 30 training tasks rather than 15 would test whether the bottleneck is insufficient signal for generalizable improvements.

  3. Selection protocol alternatives: The GPT-5.5 failure case demonstrates that rollout variance can drive acceptance of edits whose mechanisms never activate in passing rollouts. Have you experimented with requiring mechanistic validation, such as verifying that edited code paths execute in successful rollouts, alongside the score-based gate? This approach could reduce accidental acceptances.

  4. Cost-effectiveness claim clarification: You compare harness RSI cost, approximately \1,369forGPT−5.6−SolTerminalplusSWE,withestimatedpost−trainingcost,approximately1,369 for GPT-5.6-Sol Terminal plus SWE, with estimated post-training cost, approximately \\57,683 for a hypothetical 10T model. However, the post-training cost estimate does not include held-out evaluation to verify quality, whereas your RSI cost does include this component. If trajectory generation, validation, and held-out testing were added to the post-training side, how would the comparison change? The current framing may overstate the relative efficiency of harness RSI.

  5. Cross-domain transfer mechanism: Claude Opus 4.8's SWE-evolved harness improves Science by +22.22 points, yet the paper does not analyze what specific changes produced this effect. Given that SWE and Science tasks differ substantially, this result is surprising. Can you provide any mechanistic explanation, or is this an artifact of evaluation variance? The cross-domain results would be more compelling if supported by case studies demonstrating how the SWE revision assisted a Science task.

3 comments
AuthorsOpenReview ↗

(1/2) Author Responses

Dear reviewer xCAT,

We sincerely thank you for your thoughtful and positive review. We are especially grateful that you found the held-out evaluation design genuinely helpful because it exposes a critical failure mode, and that you noted how the recursive design requirement in our paper distinguishes recursive self-improvement from iterative improvement. We address each concern below.

W1 and Q1. Which accepted revisions carry over to held-out tasks

Whether a training gain carries over to held-out tasks depends on what the revision edits: revisions to the code that runs the agent carry over, and prompt revisions does not. We group the runs by the module agents edit.

Edited moduleMean training gainMean held-out gain
Agent Loop, Coding Session or Tool Runtime (the code that runs the agent)+9.72+9.94
Prompt or Prompt Loader (prompt text)+6.23+1.17

Across all runs, the runs that modify the agent code achieve a mean training gain of +9.72 and a mean held-out gain of +9.94 (with 87% revisions accepted). In contrast, the runs that modify prompts achieve a mean training gain of +6.23 but a mean held-out gain of only +1.17 (with only 18% revisions accepted).

W1 and Q2. Do smaller training sets lead to smaller RSI gains?

Our results do not show a consistent relationship between training set size and held-out RSI gains.

SWE has the fewest training tasks (8 tasks), yet achieves the highest mean held-out gain (+4.76). Its held-out gain is also 1.19 points higher than its training gain (+3.57). In contrast, Terminal has 15 training tasks but shows the largest drop from training to held-out performance: from +9.52 to +3.49 points, a difference of 6.03 points. Environment Learning also has 15 training tasks but shows a much smaller drop of 1.00 point. These results suggest that having fewer training tasks does not necessarily lead to smaller held-out gains or a larger train–held-out gap.

DomainTraining tasksMean training gain (points)Mean held-out gain (points)Training gain minus held-out gain
SWE8+3.57+4.76-1.19
Science12+6.75+4.76+1.99
Terminal15+9.52+3.49+6.03
Environment Learning15+3.66+2.66+1.00

W2. Could a candidate pass the gate by luck?

The acceptance gate keeps a candidate revision when, compared with the current harness (the parent), it raises the mean training score, improves at least two training tasks and causes no severe drop on a task the parent solved (Section 3.1). We tested one RSI round using three different seeds, and the candidate still wins under all three random seeds.

SeedTasks the parent solvesTasks the candidate solvesCandidate minus parent (points)
421/122/12+8.33
430/123/12+25.00
440/123/12+25.00

The candidate solves task sab_1 under all three seeds, and the parent fails it each time. So the revision fixes what the parent gets wrong on sab_1, and the candidate's gain is not by luck. A lucky candidate pass also cannot inflate RSI lift (the held-out gain), because held-out tasks never participate in accepting revisions or not.

AuthorsOpenReview ↗

(2/2) Author Responses

W2 and Q3. Does it help to also check that the edited code actually runs?

The acceptance gate focuses on training scores alone, so it can accept an edit whose code never runs in the future rounds. This check keeps the real improvement and rejects the lucky one.

  • Opus 5 (Appendix H.2), a real improvement: the edit adds a checkpoint message, and this message appears in the following rollout. The edited code actually ran, so the check would keep this edit.
  • GPT-5.5 (Appendix H.3), a lucky pass: the edit adds an error message to the Bash tool, but the message never appears in the 3 rollouts on the task the edit targets and appears once in all 45 training rollouts. The edited code almost never ran, so the check would reject the edit. Its training score still rose from 33/45 to 35/45 passes (+4.44 points) by chance, and its held-out score fell from 25/45 to 22/45 (-6.67 points).

So adding this check to the acceptance gate helps: the gate would still accept a real improvement, and it would reject a lucky pass that raised the training score by chance and lowered the held-out score.

Q4. Cost analysis

When we compare to the cost of post-training (including collecting its training data and for a held-out evaluation), it costs at least 8.4 times as much as harness RSI for all seven models. The RSI cost covers the whole run: proposing revisions, running training tasks and testing on held-out tasks (USD 1,369 for GPT-5.6-Sol on Terminal and SWE). Post-training needs three steps to reach the same point:

  1. Collect training data: collect at least 20 times as much SFT data as the benchmark tasks.
  2. Update the weights.
  3. Run the same held-out evaluation.

With all three steps counted, post-training costs 13.9 times as much as harness RSI for GLM-5.3, 14.0 times for DeepSeek V4 Pro, and at least 8.4 times for every one of the seven models in Table 1.

Q5. Opus 4.8 transfer to Science

In Figure 4, three frozen Opus 4.8 harnesses, from its SWE, Terminal and Environment Learning runs, are applied unchanged to the 12 held-out Science tasks, where the initial harness scores 16.67%. All three raise the score. Compared with the initial harness task by task:

Frozen Opus 4.8 harnessScience scoreTasks scored higherTasks scored lower
SWE38.89%40
Terminal33.33%40
Environment Learning30.56%31

The SWE and Terminal gains stay above zero when the 12 tasks are resampled (paired bootstrap 95% intervals [+5.56, +41.67] and [+2.78, +33.33] points). So all three Opus 4.8 harnesses improve Science, and the SWE harness improves it most. This matches Table 4, where Opus 4.8 harnesses raise the score in 8 of their 12 transfers to other domains, by +7.16 points on average.

We sincerely appreciate your thoughtful and constructive review. Our additional analyses provide four findings. First, code-level revisions generalize to held-out tasks, whereas prompt revisions show weaker generalization. Second, a code-execution check would have prevented the acceptance of a candidate that passed by chance. Third, post-training costs at least 8.4 times as much as RSI. We hope these additional results address your concerns.

Dear Reviewer xCAT,

Thank you again for your positive and thoughtful review. As the discussion period draws to a close, we would appreciate your thoughts on our responses, particularly our analysis of code versus prompt generalization, the code-execution check, the updated post-training cost comparison, and the bootstrap results for cross-domain transfer. If these additions address your remaining concerns, we would be very grateful if you could consider raising your score. Please let us know if we can clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.