- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a fascinating problem: when an agent repeatedly uses the same set of tests to select modified versions, it might choose a version that "coincidentally improves the score on this set of questions but doesn't actually enhance the agent's ability." The authors propose REUSE, which encloses the evaluation set and only returns to the agent whether to accept the modification and which candidate to accept, using statistical tests to control the probability of incorrectly accepting the modification. The paper provides theoretical guarantees and also conducts experiments on the task of modifying machine learning pipeline configurations using LLM.
Critical strengths
The problem is significant but easily overlooked by existing RSI methods. Agents that continuously modify themselves based on test results may gradually adapt to the testing framework rather than gaining better general capabilities. This paper addresses the risks of selecting multiple candidates in the same round, repeated testing, and continued modification based on historical results separately, and provides error probability guarantees for the entire process under explicit conditions.
The authors also examined reliability and final performance. In the Covertype experiment, REUSE maintains a final improvement close to that of directly selecting the highest-scoring candidate, while no errors were observed being corrected. The paper also provides a lower bound for the cumulative improvement.
Critical weaknesses
The experimental scenario is relatively narrow and differs from real-world production environments. The experiment uses only a classification dataset, and the LLM primarily modifies the model type, hyperparameters, and preprocessing configuration; it doesn't validate more complex coding or coworker scenarios.
Furthermore, the theory requires an independent and closed evaluation set, where the agent cannot see detailed scores and failure examples. This assumption is unrealistic; in reality, many agents are modified based on test errors.
Questions for authors
see weaknesses
2 comments
Response to Reviewer 2FTf (2/2)
W1 (continued)
Experiment 2 evaluates code evolution on the online bin packing task studied in FunSearch, a combinatorial optimization problem (Romera-Paredes et al., 2024; Novikov et al., 2025; https://github.com/google-deepmind/funsearch). We use a binary score if the program uses at most bins on instance , where is the total item size divided by 100 and rounded up. Otherwise, including errors and timeouts, . In each round, the proposer model (Qwen2.5-7B-Instruct) generates candidate modifications to the program implementing the packing policy, evolving over rounds. We use an evaluation set with instances and a held-out set of 50,000 instances. We run the eight methods from Table 1 with 10 seeds each. The results are summarized in the table below.
| n = 2,000 | n = 10,000 | |||||
|---|---|---|---|---|---|---|
| Method | Pop. Impr. (pp) ↑ | Prom. | False ↓ | Pop. Impr. (pp) ↑ | Prom. | False ↓ |
| Empirical best-of-K | 18.37 ± 5.34 | 15.6 | 43 | 29.60 ± 5.36 | 24.5 | 54 |
| SICA-style | 16.43 ± 4.38 | 15.6 | 33 | 14.84 ± 6.25 | 17.2 | 45 |
| DGM-style | 17.88 ± 5.61 | 7.8 | 12 | 14.09 ± 4.29 | 7.6 | 8 |
| Elite | 26.75 ± 5.22 | 23.3 | 67 | 24.27 ± 5.79 | 28.5 | 63 |
| Niche-elite | 29.37 ± 3.81 | 9.4 | 14 | 19.86 ± 5.48 | 8.5 | 12 |
| PACE-style | 19.57 ± 4.66 | 6.3 | 2 | 28.39 ± 5.39 | 12.9 | 5 |
| Bonferroni-style | 23.97 ± 5.74 | 2.1 | 0 | 16.19 ± 6.18 | 3.2 | 0 |
| REUSE (ours) | 35.27 ± 2.37 | 5.4 | 0 | 31.40 ± 4.78 | 7.7 | 0 |
Experiment 2, code evolution on online bin packing task in FunSearch, rounds. Pop. Impr. is the population improvement over the initial version (mean standard error over seeds), Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.
The results demonstrate that REUSE successfully controls false promotions while reaching the highest true population improvement. Specifically, at , our method achieves an average population improvement of 35.27 pp, which is the highest among all methods, while committing zero false promotions. In contrast, the other six baselines (excluding Bonferroni-style) all have false promotions and achieve lower population improvements. Although Bonferroni-style does not have false promotions in this, it is overly conservative and exhibits lower power, resulting in a much lower number of promotions (2.1) and lower population improvement (23.97 pp) compared with our method. The same pattern holds true at .
Revision. We have incorporated both experiments into Section 3 with full protocols and results in Appendix B, and have uploaded a revised PDF with revisions highlighted in blue.
W2. The closed evaluation set
Response. We thank the reviewer for raising this point.
We would like to clarify that only the evaluation set used for promotion decisions is closed. The agent can still be modified based on its errors, as detailed scores and failure examples from independent development data can be freely used to guide modifications. Assumption 2 only restricts feedback from to promotion decisions, which can still guide subsequent proposals. Revealing detailed scores and failure examples from could cause the proposer to overfit the evaluation set, compromising the validity of repeated evaluations. Our experimental results, including the two new experiments, demonstrate that REUSE can still achieve substantial true population improvements using development feedback and promotion decisions without accessing detailed results from .
Response to Reviewer 2FTf (1/2)
We thank the reviewer for the positive assessment and for the clear summary of the three sources of false promotion that REUSE controls. We address each point below.
W1. Narrow experimental scenario
Response. We thank the reviewer for this valuable suggestion.
We would like to briefly clarify that REUSE is an evaluation framework that is agnostic to the task and proposer. Rigorously evaluating such a framework requires estimating the population performance of every promoted modification to accurately identify false promotions. This involves repeated evaluations on a sufficiently large evaluation set S and a separate sufficiently large H for accurate population performance evaluation, resulting in large computational costs. These requirements motivated our original choice of experimental setting
We fully agree that validating more complex coding or coworker scenarios is crucial. We have therefore conducted additional experiments covering existing agent self-evolution tasks. The first new experiment evaluates a self-improving coding agent, and the second new experiment focuses on a code evolution task. The results demonstrate that REUSE can still reduce false promotions while maintaining high true population performance.
Experiment 1 evaluates a self-improving coding agent on CodeContests (Li et al., 2022; https://huggingface.co/datasets/deepmind/code_contests). The initial agent is adapted from that of DGM (Zhang et al., 2025). We use Qwen3-Coder-30B-A3B-Instruct as the solver model. In each round, Claude Opus 5.5 serves as the proposer and generates one candidate modification to the agent. The proposer can rewrite any part of the agent's source code or add new files, including changes to its prompts, tools, control loop, problem-solving workflow, and interactions with the solver model. We use n=800 problems in S and 750 in H, and run the eight methods of Table 1 for $T=$10 rounds with 3 seeds each. The results are summarized in the table below.
Experiment 1, self-improving coding agent on CodeContests, T=10 rounds. Pop. Impr. is the population improvement in solve rate over the initial agent (mean ± standard error over three seeds). Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.
The results show that REUSE achieves competitive population improvement while avoiding false promotions in this self-improving coding agent setting. Specifically, REUSE achieves an average population improvement of 14.44 pp, comparable to the best-performing baselines, but with zero false promotions, whereas those baselines have false promotions. Although Bonferroni-style also avoids false promotions, it is much more conservative, resulting in fewer promotions and lower population improvement. These results demonstrate that REUSE can maintain useful progress while reliably controlling false promotions in agent self-improvement tasks.