- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RSIBench, a benchmark for evaluating recursive harness modification using frozen LLMs. Starting from the Pi harness, an agent uses training trajectories to edit its prompts, tools, and execution logic. Accepted edits are inherited for both solving future tasks and generating subsequent edits. The benchmark spans 100 tasks (50/50 train/test split) across four domains, using "RSI lift" (held-out score difference between final and initial harnesses) as the main metric. The authors evaluate seven models, analyze cross-domain transfer, ablate editable modules, and run 30-round extensions, alongside detailed trajectory case studies.
Critical strengths
An important question and concrete setup.
Testing whether an agent can turn interaction experience into lasting system improvements is important for stable post-deployment adaptation. RSIBench provides a practical testbed with a broad editable surface, strict held-out evaluation, and an inheritance loop where updated harnesses drive future edits.
Insightful failure-mode analysis.
Appendix H nicely separates fixing a real bug from fixing the actual bottleneck. In the failure case, the agent patches a genuine Bash-runtime issue, yet that code branch is never even triggered during the diagnosed task's rollouts. Contrasting this with a successful stopping-rule fix clearly illustrates why valid code edits and higher training scores don't automatically translate into generalizable improvements.
Critical weaknesses
Main concern: RSI lift doesn't test the recursive part.
RSI lift compares the held-out scores of the first and last harness. A proposer that never changes could produce the same lift, so the metric can't tell us whether the system got better at improving itself. Edits are inherited, but the paper never checks whether inheriting them matters. This is the central question, since Evo-Bench and SEAGym already cover harness evolution and transfer, and the recursive loop is what RSIBench adds. Two controls would settle it: freeze the proposer at round 0 while keeping the editable surface, feedback and budget unchanged, or start the initial and evolved proposers from the same harness and compare their edits.
I also suspect the greedy gate. It keeps only edits that raise training scores right away, so an edit that only makes later diagnosis or patching better gets thrown out. That may be why 30 rounds stall, and the paper doesn't trace where the stall comes from. Given DGM, HGM and Hyperagents, I'd expect a benchmark in this space to isolate meta-improvement more directly.
Second concern: the numbers are too noisy to rank.
In Table 2, Opus 5 Agent Loop looks like +13.33 against the main baseline, but its own matched initial-to-final score drops from 53.33 to 46.67. Appendix E.2 reports a 15. I appreciate the transparency, but at this level of noise I can't trust even the sign of many module and transfer gains. These claims need multiple runs with error bars.
Questions for authors
-
Did you test freezing the proposer at its initial state while keeping the full editable surface, feedback, and budget? Or did you compare initial vs. evolved proposers starting from the same harness to see if the evolved one generates better edits?
-
Did you observe rejected edits that improved diagnostics or code generation without immediately raising training scores? Could this strict gate explain the 30-round stagnation?
-
Which module and transfer claims still hold after accounting for baseline noise? Can you share matched initial-vs-final scores or error bars for the main claims?
3 comments
(2/2) Author Responses
W1 and Q2. Why the 30-round run stops improving
The acceptance gate keeps a candidate revision when, compared with the current harness, it raises the mean training score, improves at least two training tasks and causes no severe drop on a solved task. In the 30-round GPT-5.6-Sol run on Terminal tasks (Table 6), the training score stops rising after round 4 because no later candidate scored higher. Table 6 records each round's scores and decisions: none of the 26 candidates after the round-4 acceptance scored above the current harness (80.00). After round-4, the harness already has good performance and further self-improvement becomes more difficult.
W2 and Q3. Which claims hold after accounting for noise
Two runs of the same harness can differ. RSI lift (final minus initial held-out score) across Table 1, the Opus 4.8 transfer gain on Science and the model dependence of transfer gain all hold after accounting for this noise.
1. RSI lift
If RSI lift were noisy, scores would rise and fall equally often, however large the noise. We analyze the RSI lift below, and the direction of RSI lift holds across models and domains.
| 7 models × 4 domains = 28 results | Result |
|---|---|
| Results that change between initial and final harness | 20 |
| Rises among these 20 | 16 |
| Falls | 4 |
| Mean change over the 28 results | +3.92 points, 95% bootstrap interval [+2.1, +5.8] |
| Models with positive mean RSI lift | 7 of 7 |
| Domains with positive mean RSI lift | 4 of 4 |
2. Transfer acoss domains
Figure 4 applies three models' frozen final harnesses to the other three domains. Resampling the held-out tasks (paired bootstrap, 10,000 resamples) are provided below.
| Frozen harness and target | (points) | 95% interval |
|---|---|---|
| Opus 4.8 SWE harness → Science | +22.22 (16.67 → 38.89) | [+5.56, +41.67] |
| Opus 4.8 Terminal harness → Science | +16.67 (16.67 → 33.33) | [+2.78, +33.33] |
Both Opus 4.8 harnesses, one from SWE and one from Terminal, raise the Science score, and the low end of each interval stays above zero (+5.56 and +2.78 points). So the Opus 4.8 transfer gain on Science holds under task resampling.
3. Modules
RSIBench checks every module claim in two ways. First, Table 2 compares each final harness with the main-experiment baseline. Second, for a single edit, the stored rollouts, including the messages the harness inserts into them, show whether the edited code actually ran. In this way, we can observe RSI lift when editing specific harness modules.
- Opus 5 (Appendix H.2): the edit adds a checkpoint message, and this message appears in the following rollout. The edited code actually ran, so the edit ran and the gain belongs to it.
- GPT-5.5 (Appendix H.3): the edited code runs in 1 of the 45 training rollouts, so its +4.44-point training gain came from rollout noise, not from the edit. Its training score rose from 33/45 to 35/45 passes (+4.44 points) by chance.
So RSIBench credits a RSI gain to a module edit when the edited code ran, and catches gains that come from noise.
We sincerely appreciate your detailed and thoughtful review. Our additional experiments show that a recursive harness proposer achieves twice the training score of a frozen proposer. We also find that rejected diagnostic edits still help subsequent harness proposals build and run successfully. Furthermore, both the RSI lift and the domain transfer gain remain robust after accounting for rollout noise. We hope these additional results address your concerns, and we would appreciate your reconsideration of our paper.
[1] Zhang, Jenny, et al. "Darwin Gödel machine: Open-ended evolution of self-improving agents." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Wang, Wenyi, et al. "Huxley-g" odel machine: Human-level coding agent development by an approximation of the optimal self-improving machine." International Conference on Learning Representations. Vol. 2026. 2026.
[3] Zhang, Jenny, et al. "Hyperagents." arXiv preprint arXiv:2603.19461 (2026).
[4] Zhang, Hangfan, et al. "Self-harness: Harnesses that improve themselves." arXiv preprint arXiv:2606.09498 (2026).
[5] Lee, Hyunin, et al. "Recursive Harness Self-Improvement." arXiv preprint arXiv:2607.15524 (2026).
[6] Huang, Lisheng, et al. "Evo-Bench: Can language models improve agent harness?." arXiv preprint arXiv:2608.09096 (2026).
[7] Zheng, Congjie, et al. "SEAGym: An Evaluation Environment for Self-Evolving LLM Agents." arXiv preprint arXiv:2606.17546 (2026).
Dear Reviewer JJzC,
Thank you again for your detailed and constructive feedback. As the discussion period wraps up, we would appreciate your thoughts on our responses, including the frozen-proposer control experiment, the analysis of rejected diagnostic edits, and the bootstrap confidence intervals confirming that our RSI lift and transfer gains remain robust after accounting for rollout noise. If these results resolve your concerns, we would sincerely appreciate it if you could consider raising your score.
(1/2) Author Responses
Dear reviewer JJzC,
We sincerely thank you for your detailed and constructive review. We are especially grateful that you recognized the importance of turning interaction experience into long-lasting system improvements, our strict held-out evaluation and the design of inheritance loop where updated harnesses do future edits, and the failure-mode analysis that separates fixing a real bug from fixing the actual bottleneck. We address each concern below.
W1 and Q1. Is the recursive part tested?
Yes. The proposer (the agent writing each revision) in round t runs on the current harness Ht (Section 3.1), so after an accepted revision, an edited harness proposes the next one. We test whether this helps with per-run proposer records (subsection 1) and provides an experiment about frozen-proposer (subsection 2).
Edits accepted in earlier rounds also run inside later proposers: GPT-5.5 Terminal accepted a Bash tool revision during round-3 that appends the duration of long-running commands, and the round-4 and round-5 proposers' own build checks end with notes such as "[Command duration: 14.1s]".
1. Initial proposer versus evolved proposer
In the table below, both H0 and an edited harness proposed accepted revisions. Each row compares the two proposers under the same model, tasks and acceptance rule. The edited harness proposes the revision with larger gains.
2. Frozen-proposer experiment
We also ran an experiment comparing recursive self-improvement (where the proposer runs on the current, edited harness) with iterative improvement (where the proposer always runs on the initial harness H0). We evaluate GPT-5.6-Sol on Science tasks, and both runs use the same rounds and editable modules. The recursive run ends at a training score of 16.67% and the iterative run at 8.33%.
3. What RSIBench adds
The table below adds the recursive self-improvement methods to the comparison of Table 12. RSIBench focuses on recursive self-improvement, where the evolved harness proposes the next revision.
Q2. Can a rejected diagnostic edit still help later rounds?
Yes. A diagnostic edit makes failures easier to see without fixing them, so it may not raise the training score, and the acceptance gate then rejects it. In a GPT-5.6-Sol run on Terminal tasks, the gate rejected diagnostic edits in rounds 2 and 5. We ran the proposer three times on each of these two harnesses and three times on the initial harness, and checked whether each proposed patch passes the local build and command-line check. So a diagnostic edit that the gate rejects still helps later proposals build and run.