acceptodds
Under review as a conference paper at ICLR 2027

Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks

Abstract

As recursive self-improvement (RSI) rapidly advances, reliable evaluation becomes critical for guiding adaptive search. RSI typically relies on finite evaluation resources, such as fixed benchmarks, to determine which modifications are retained and what is proposed next. However, when these finite resources are repeatedly reused, new candidates are proposed based on feedback from the same evaluation set, so the search trajectory can adaptively overfit and empirical improvement may not reflect genuine population improvement on the underlying task distribution. Some existing methods account for multiple comparisons but assume that candidates are chosen independently of the evaluation set, and therefore do not control this adaptive dependence. To address this, we propose REUSE (Risk-controlled Evaluation Under Sequential Evolution), a certified evaluation and promotion framework that allows a fixed evaluation set to support repeated adaptive decisions while providing statistical guarantees. Specifically, for a user-specified error level , with probability at least , every promoted modification is a genuine population improvement on the underlying task distribution. REUSE achieves this by strictly limiting the evaluation feedback returned to the search process and accounting for possible promotion histories within the error budget. We develop a detailed statistical theory for RSI evaluation in this setting, including simultaneous error control, valid lower bounds on cumulative improvement, and a characterization of the fundamental limits of adaptive evaluation reuse. In live self-improvement experiments, REUSE commits substantially fewer false promotions than evaluation frameworks from current RSI systems and error-controlled baselines, reducing the proportion of promotions that are false from up to 20.7% to 0%, while achieving final true population performance comparable to the best baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-improving Agents, Benchmark, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2FTfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a fascinating problem: when an agent repeatedly uses the same set of tests to select modified versions, it might choose a version that "coincidentally improves the score on this set of questions but doesn't actually enhance the agent's ability." The authors propose REUSE, which encloses the evaluation set and only returns to the agent whether to accept the modification and which candidate to accept, using statistical tests to control the probability of incorrectly accepting the modification. The paper provides theoretical guarantees and also conducts experiments on the task of modifying machine learning pipeline configurations using LLM.

Critical strengths

The problem is significant but easily overlooked by existing RSI methods. Agents that continuously modify themselves based on test results may gradually adapt to the testing framework rather than gaining better general capabilities. This paper addresses the risks of selecting multiple candidates in the same round, repeated testing, and continued modification based on historical results separately, and provides error probability guarantees for the entire process under explicit conditions.

The authors also examined reliability and final performance. In the Covertype experiment, REUSE maintains a final improvement close to that of directly selecting the highest-scoring candidate, while no errors were observed being corrected. The paper also provides a lower bound for the cumulative improvement.

Critical weaknesses

The experimental scenario is relatively narrow and differs from real-world production environments. The experiment uses only a classification dataset, and the LLM primarily modifies the model type, hyperparameters, and preprocessing configuration; it doesn't validate more complex coding or coworker scenarios.

Furthermore, the theory requires an independent and closed evaluation set, where the agent cannot see detailed scores and failure examples. This assumption is unrealistic; in reality, many agents are modified based on test errors.

Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2FTf (1/2)

We thank the reviewer for the positive assessment and for the clear summary of the three sources of false promotion that REUSE controls. We address each point below.

W1. Narrow experimental scenario

Response. We thank the reviewer for this valuable suggestion.

We would like to briefly clarify that REUSE is an evaluation framework that is agnostic to the task and proposer. Rigorously evaluating such a framework requires estimating the population performance of every promoted modification to accurately identify false promotions. This involves repeated evaluations on a sufficiently large evaluation set SS and a separate sufficiently large HH for accurate population performance evaluation, resulting in large computational costs. These requirements motivated our original choice of experimental setting

We fully agree that validating more complex coding or coworker scenarios is crucial. We have therefore conducted additional experiments covering existing agent self-evolution tasks. The first new experiment evaluates a self-improving coding agent, and the second new experiment focuses on a code evolution task. The results demonstrate that REUSE can still reduce false promotions while maintaining high true population performance.

Experiment 1 evaluates a self-improving coding agent on CodeContests (Li et al., 2022; https://huggingface.co/datasets/deepmind/code_contests). The initial agent is adapted from that of DGM (Zhang et al., 2025). We use Qwen3-Coder-30B-A3B-Instruct as the solver model. In each round, Claude Opus 5.5 serves as the proposer and generates one candidate modification to the agent. The proposer can rewrite any part of the agent's source code or add new files, including changes to its prompts, tools, control loop, problem-solving workflow, and interactions with the solver model. We use n=800n=800 problems in SS and 750 in HH, and run the eight methods of Table 1 for $T=$10 rounds with 3 seeds each. The results are summarized in the table below.

MethodPop. Impr. (pp) ↑Prom.False ↓
Empirical best-of-K14.93 ± 0.287.05
SICA-style14.04 ± 0.122.72
DGM-style12.58 ± 0.225.37
Elite14.40 ± 0.283.74
Niche-elite15.73 ± 0.233.32
PACE-style11.69 ± 0.251.31
Bonferroni-style11.42 ± 0.241.00
REUSE (ours)14.44 ± 0.312.30

Experiment 1, self-improving coding agent on CodeContests, T=10T=10 rounds. Pop. Impr. is the population improvement in solve rate over the initial agent (mean ±\pm standard error over three seeds). Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.

The results show that REUSE achieves competitive population improvement while avoiding false promotions in this self-improving coding agent setting. Specifically, REUSE achieves an average population improvement of 14.44 pp, comparable to the best-performing baselines, but with zero false promotions, whereas those baselines have false promotions. Although Bonferroni-style also avoids false promotions, it is much more conservative, resulting in fewer promotions and lower population improvement. These results demonstrate that REUSE can maintain useful progress while reliably controlling false promotions in agent self-improvement tasks.

AuthorsOpenReview ↗

Response to Reviewer 2FTf (2/2)

W1 (continued)

Experiment 2 evaluates code evolution on the online bin packing task studied in FunSearch, a combinatorial optimization problem (Romera-Paredes et al., 2024; Novikov et al., 2025; https://github.com/google-deepmind/funsearch). We use a binary score s(A,W)=1s(A,W)=1 if the program uses at most L1(W)+4L_1(W)+4 bins on instance WW, where L1(W)L_1(W) is the total item size divided by 100 and rounded up. Otherwise, including errors and timeouts, s(A,W)=0s(A,W)=0. In each round, the proposer model (Qwen2.5-7B-Instruct) generates candidate modifications to the program implementing the packing policy, evolving over T=200T=200 rounds. We use an evaluation set SS with n∈{2,000, 10,000}n\in\lbrace 2{,}000,\thinspace 10{,}000\rbrace instances and a held-out set HH of 50,000 instances. We run the eight methods from Table 1 with 10 seeds each. The results are summarized in the table below.

n = 2,000n = 10,000
MethodPop. Impr. (pp) ↑Prom.False ↓Pop. Impr. (pp) ↑Prom.False ↓
Empirical best-of-K18.37 ± 5.3415.64329.60 ± 5.3624.554
SICA-style16.43 ± 4.3815.63314.84 ± 6.2517.245
DGM-style17.88 ± 5.617.81214.09 ± 4.297.68
Elite26.75 ± 5.2223.36724.27 ± 5.7928.563
Niche-elite29.37 ± 3.819.41419.86 ± 5.488.512
PACE-style19.57 ± 4.666.3228.39 ± 5.3912.95
Bonferroni-style23.97 ± 5.742.1016.19 ± 6.183.20
REUSE (ours)35.27 ± 2.375.4031.40 ± 4.787.70

Experiment 2, code evolution on online bin packing task in FunSearch, T=200T=200 rounds. Pop. Impr. is the population improvement over the initial version (mean ±\pm standard error over seeds), Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.

The results demonstrate that REUSE successfully controls false promotions while reaching the highest true population improvement. Specifically, at n=2,000n=2{,}000, our method achieves an average population improvement of 35.27 pp, which is the highest among all methods, while committing zero false promotions. In contrast, the other six baselines (excluding Bonferroni-style) all have false promotions and achieve lower population improvements. Although Bonferroni-style does not have false promotions in this, it is overly conservative and exhibits lower power, resulting in a much lower number of promotions (2.1) and lower population improvement (23.97 pp) compared with our method. The same pattern holds true at n=10,000n=10{,}000.

Revision. We have incorporated both experiments into Section 3 with full protocols and results in Appendix B, and have uploaded a revised PDF with revisions highlighted in blue.

W2. The closed evaluation set

Response. We thank the reviewer for raising this point.

We would like to clarify that only the evaluation set SS used for promotion decisions is closed. The agent can still be modified based on its errors, as detailed scores and failure examples from independent development data can be freely used to guide modifications. Assumption 2 only restricts feedback from SS to promotion decisions, which can still guide subsequent proposals. Revealing detailed scores and failure examples from SS could cause the proposer to overfit the evaluation set, compromising the validity of repeated evaluations. Our experimental results, including the two new experiments, demonstrate that REUSE can still achieve substantial true population improvements using development feedback and promotion decisions without accessing detailed results from SS.

Reviewer_nwgqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces REUSE to control false promotions when self-improving systems repeatedly use a fixed evaluation set. Candidate generation receives only promotion decisions as feedback. The method accounts for all possible decision histories in a global error budget. Its guarantee assumes that the evaluation set is initially independent of development and that later feedback comes only through promotion decisions. Under these conditions, REUSE provides a probabilistic guarantee that every accepted update improves population performance. In the experiments, an LLM is used to modify the covertype classification pipeline, with an independent holdout set used to estimate population gains.

Critical strengths

What I find most interesting is the recognition that accept or reject feedback still leaks information. The round in which an update is accepted and the candidate that gets accepted both affect what is generated next. Later candidates can therefore still adapt to the same evaluation set. REUSE explicitly includes these possible decision histories in its error budget. This gives decision-only feedback a clear statistical justification. It addresses repeated benchmark use more fully than just limiting score feedback or correcting for the number of tests. I also like that the main experiments track both final performance and the gain from each accepted update. A system can end up performing well without every update along the way being worth accepting. The step by step promotion results in the main paper show this distinction. They also make REUSE’s contribution easier to assess. It aims to reduce the acceptance of changes that are mistaken for progress during evolution. Considering these results alongside final performance fits the paper’s question better than reporting final accuracy alone.

Critical weaknesses

I find the experimental coverage fairly narrow. It does not yet convince me that REUSE has the same practical value across a wider range of self-evolution tasks. All empirical validation centers on optimizing classification pipelines on covertype. The paper discusses SICA, DGM, and related work, and borrows their search strategies to build baselines. I was therefore a little surprised that the authors did not use one of those task environments for further validation. This matters to me because REUSE changes the evaluation feedback available to candidate generation. In the current setup, the LLM modifies predefined scikit learn configurations. It can use development-set accuracy and error analysis to guide the search. The situation may change when the updates target an agent’s tool code, prompts, or execution strategy. I am not sure whether development feedback would still be enough to support the search. I also want to know how much the improvements achieved would be affected by returning only promotion decisions. The current results show that reliability and good final performance can go together. But they do not fully address my concerns about other self-evolution settings.

I also noticed that the authors list coding and model post-training as future directions. But since the paper focuses on self-evolving systems, I think it needs at least one additional, existing agent self-evolution task. The evaluation should follow REUSE’s feedback restrictions and measure both false promotions and final task performance. This would provide a more direct test of whether the statistical mechanism leads to practical gains. For me, this is the most important missing validation in the empirical section.

Questions for authors

Could you include at least one existing agent self-evolution task, such as iterative optimization of tool code or execution strategies? The experiments in the main paper focus on covertype. I would like to see REUSE tested on other tasks under its evaluation feedback restrictions. Can it still reduce false promotions while maintaining final task performance? I would be happy to raise my score if additional results address this concern.

4 comments
AuthorsOpenReview ↗

Response to Reviewer nwgq (1/2)

We thank the reviewer for the thoughtful review, and we are glad that the reviewer found the information leaked by accept/reject decisions and the step-wise promotion results useful. We address each point below.

W1, W2, W4 and Q1. An existing agent self-evolution task

Response. We thank the reviewer for this valuable suggestion.

We would like to briefly clarify that REUSE is an evaluation framework that is agnostic to the task and proposer. Rigorously evaluating such a framework requires estimating the population performance of every promoted modification to accurately identify false promotions. This involves repeated evaluations on a sufficiently large evaluation set SS and a separate sufficiently large HH for accurate population performance evaluation, resulting in large computational costs. These requirements motivated our original choice of experimental setting.

We fully agree that evaluating an existing agent self-evolution task is a crucial validation step. We have therefore conducted additional experiments covering existing agent self-evolution tasks. The first new experiment evaluates a self-improving coding agent, and the second new experiment focuses on a code evolution task. The results demonstrate that REUSE can still reduce false promotions while maintaining final task true population performance.

Experiment 1 evaluates a self-improving coding agent on CodeContests (Li et al., 2022; https://huggingface.co/datasets/deepmind/code_contests). The initial agent is adapted from that of DGM (Zhang et al., 2025). We use Qwen3-Coder-30B-A3B-Instruct as the solver model. In each round, Claude Opus 5.5 serves as the proposer and generates one candidate modification to the agent. The proposer can rewrite any part of the agent's source code or add new files, including changes to its prompts, tools, control loop, problem-solving workflow, and interactions with the solver model. We use n=800n=800 problems in SS and 750 in HH, and run the eight methods of Table 1 for $T=$10 rounds with 3 seeds each. The results are summarized in the table below.

MethodPop. Impr. (pp) ↑Prom.False ↓
Empirical best-of-K14.93 ± 0.287.05
SICA-style14.04 ± 0.122.72
DGM-style12.58 ± 0.225.37
Elite14.40 ± 0.283.74
Niche-elite15.73 ± 0.233.32
PACE-style11.69 ± 0.251.31
Bonferroni-style11.42 ± 0.241.00
REUSE (ours)14.44 ± 0.312.30

Experiment 1, self-improving coding agent on CodeContests, T=10T=10 rounds. Pop. Impr. is the population improvement in solve rate over the initial agent (mean ±\pm standard error over three seeds). Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.

The results show that REUSE achieves competitive population improvement while avoiding false promotions in this self-improving coding agent setting. Specifically, REUSE achieves an average population improvement of 14.44 pp, comparable to the best-performing baselines, but with zero false promotions, whereas those baselines have false promotions. Although Bonferroni-style also avoids false promotions, it is much more conservative, resulting in fewer promotions and lower population improvement. These results demonstrate that REUSE can maintain useful progress while reliably controlling false promotions in agent self-improvement tasks.

AuthorsOpenReview ↗

Response to Reviewer nwgq (2/2)

W1, W2, W4 and Q1 (continued)

Experiment 2 evaluates code evolution on the online bin packing task studied in FunSearch, a combinatorial optimization problem (Romera-Paredes et al., 2024; Novikov et al., 2025; https://github.com/google-deepmind/funsearch). We use a binary score s(A,W)=1s(A,W)=1 if the program uses at most L1(W)+4L_1(W)+4 bins on instance WW, where L1(W)L_1(W) is the total item size divided by 100 and rounded up. Otherwise, including errors and timeouts, s(A,W)=0s(A,W)=0. In each round, the proposer model (Qwen2.5-7B-Instruct) generates candidate modifications to the program implementing the packing policy, evolving over T=200T=200 rounds. We use an evaluation set SS with n∈{2,000, 10,000}n\in\lbrace 2{,}000,\thinspace 10{,}000\rbrace instances and HH of 50,000 instances. We run the eight methods from Table 1 with 10 seeds each. The results are summarized in the table below.

n = 2,000n = 10,000
MethodPop. Impr. (pp) ↑Prom.False ↓Pop. Impr. (pp) ↑Prom.False ↓
Empirical best-of-K18.37 ± 5.3415.64329.60 ± 5.3624.554
SICA-style16.43 ± 4.3815.63314.84 ± 6.2517.245
DGM-style17.88 ± 5.617.81214.09 ± 4.297.68
Elite26.75 ± 5.2223.36724.27 ± 5.7928.563
Niche-elite29.37 ± 3.819.41419.86 ± 5.488.512
PACE-style19.57 ± 4.666.3228.39 ± 5.3912.95
Bonferroni-style23.97 ± 5.742.1016.19 ± 6.183.20
REUSE (ours)35.27 ± 2.375.4031.40 ± 4.787.70

Experiment 2, code evolution on online bin packing task in FunSearch, T=200T=200 rounds. Pop. Impr. is the population improvement over the initial version (mean ±\pm standard error over seeds), Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.

The results demonstrate that REUSE successfully controls false promotions while reaching the highest true population improvement. Specifically, at n=2,000n=2{,}000, our method achieves an average population improvement of 35.27 pp, which is the highest among all methods, while committing zero false promotions. In contrast, the other six baselines (excluding Bonferroni-style) all have false promotions and achieve lower population improvements. Although Bonferroni-style does not have false promotions in this, it is overly conservative and exhibits lower power, resulting in a much lower number of promotions (2.1) and lower population improvement (23.97 pp) compared with our method. The same pattern holds true at n=10,000n=10{,}000.

Revision. We have incorporated both experiments into Section 3 with full protocols and results in new Appendix B. We have uploaded a revised PDF with revisions highlighted in blue.

W3. Effect of returning only promotion decisions

Response. We thank the reviewer for raising this important point.

To address this concern and evaluate the effect of returning only promotion decisions, we conducted an ablation study that changes only the feedback channel. We keep the REUSE promotion rule identical and compare the decision-only feedback of REUSE against the standard feedback commonly used in other RSI algorithms that provides scores and failure breakdown on SS. Under the setting of Experiment 2 at n=2,000n=2{,}000, the table below summarizes the results over 10 seeds, reporting the mean final population improvement (±\pm standard error), average number of promotions, and total number of false promotions.

Feedback ChannelPop. Impr. in pp ↑Prom.False ↓
Decision-only feedback (REUSE)35.27 ± 2.375.40
Standard feedback (scores & failures)21.01 ± 5.323.30

The results show that returning only promotion decisions improves true population performance. Specifically, REUSE achieves a population improvement of 35.27 percentage points compared with 21.01 percentage points under standard feedback. The lower population improvement under standard feedback may arise because detailed evaluation scores and failure breakdown cause the proposer to adaptively overfit the evaluation set. Although both feedback settings yield zero false promotions in this experiment, revealing scores and failure breakdown from SS violates Assumption 2 in the original main text, so the theoretical guarantee of false promotion control no longer applies to the standard feedback setting.

Revision. We have incorporated this ablation into Appendix B.4.1 and have uploaded a revised PDF with revisions highlighted in blue.

Thank you for the additional experiments. The experiment addresses my main concern about validation beyond classification-pipeline optimization, and the feedback ablation addresses my concern about restricting evaluation feedback. I therefore lean toward raising my rating to weak accept.

Thank you very much for taking the time to review our additional results! We are glad they address your concerns and appreciate your positive feedback.

Reviewer_4Kw3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces REUSE, a certified evaluation and promotion framework that controls false promotions when an adaptive self-improvement process repeatedly reuses a fixed evaluation set.

Critical strengths
  • This paper addresses the important problem of determining whether performance gains measured on a limited, repeatedly reused evaluation set generalize to unseen data.
Critical weaknesses
  • Experiments cover only one binary classification task and one proposer, Qwen2.5-7B-Instruct, searching over scikit-learn configurations. This provides limited evidence for applicability to broader self-improving agents.

  • The technical contribution beyond reusable holdout methods [1, 2] remains unclear, particularly regarding restricted feedback and error control over possible decision histories.

  • REUSE also changes proposer feedback, development-based screening, and duplicate filtering. The comparisons therefore do not isolate the benefit of its error allocation.

[1] Cynthia Dwork et al. Generalization in Adaptive Data Analysis and Holdout Reuse. NeurIPS, 2015.

[2] Preetum Nakkiran and Jarosław Błasiok. The Generic Holdout: Preventing False-Discoveries in Adaptive Data Science. arXiv, 2018.

Questions for authors
  • Does REUSE preserve useful progress on coding or tool-use tasks, with different proposer models and stochastic evaluations?

  • Which guarantees go beyond applying existing reusable holdout methods to candidate–incumbent comparisons, and at what evaluation-data cost?

  • Does REUSE outperform statistical baselines when feedback, screening, duplicate filtering, and evaluation-query schedules are matched?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4Kw3 (1/4)

We thank the reviewer for the detailed comments and for recognizing the importance of the problem. We address each point below.

W1 and Q1. Coding and tool-use tasks, other proposer models and stochastic evaluations

Response. We thank the reviewer for raising this point.

We would like to briefly clarify that REUSE is an evaluation framework that is agnostic to the task and proposer. Rigorously evaluating such a framework requires estimating the population performance of every promoted modification. This involves repeated evaluations on a sufficiently large evaluation set SS and a separate sufficiently large set HH for accurate population performance evaluation, resulting in substantially large computational costs. These requirements motivated our original choice of experimental setting.

We fully agree that evaluating REUSE in broader self-improvement settings is important. We have therefore conducted two additional experiments covering coding and tool-use tasks, different proposer models, and stochastic evaluations. The first new experiment evaluates a self-improving coding agent, and the second new experiment focuses on a code evolution task. The results demonstrate that REUSE preserves useful progress across various tasks, matching and even exceeding the performance observed in our original manuscript.

Experiment 1 evaluates a self-improving coding agent on CodeContests (Li et al., 2022; https://huggingface.co/datasets/deepmind/code_contests). The initial agent is adapted from that of DGM (Zhang et al., 2025). We use Qwen3-Coder-30B-A3B-Instruct as the solver model. In each round, Claude Opus 5.5 serves as the proposer and generates one candidate modification to the agent. The proposer can rewrite any part of the agent's source code or add new files, including changes to its prompts, tools, control loop, problem-solving workflow, and interactions with the solver model. We use n=800n=800 problems in SS and 750 in HH, and run the eight methods of Table 1 for $T=$10 rounds with 3 seeds each. The results are summarized in the table below.

MethodPop. Impr. (pp) ↑Prom.False ↓
Empirical best-of-K14.93 ± 0.287.05
SICA-style14.04 ± 0.122.72
DGM-style12.58 ± 0.225.37
Elite14.40 ± 0.283.74
Niche-elite15.73 ± 0.233.32
PACE-style11.69 ± 0.251.31
Bonferroni-style11.42 ± 0.241.00
REUSE (ours)14.44 ± 0.312.30

Experiment 1, self-improving coding agent on CodeContests, T=10T=10 rounds. Pop. Impr. is the population improvement in solve rate over the initial agent (mean ±\pm standard error over three seeds). Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.

The results show that REUSE achieves competitive population improvement while avoiding false promotions in this self-improving coding agent setting. Specifically, REUSE achieves an average population improvement of 14.44 pp, comparable to the best-performing baselines, but with zero false promotions, whereas those baselines have false promotions. Although Bonferroni-style also does not have false promotions, it is much more conservative, resulting in fewer promotions and lower population improvement. These results demonstrate that REUSE can maintain useful progress while reliably controlling false promotions in agent self-improvement tasks.

AuthorsOpenReview ↗

Response to Reviewer 4Kw3 (2/4)

W1 and Q1 (continued)

Experiment 2 evaluates code evolution on the online bin packing task studied in FunSearch, a combinatorial optimization problem (Romera-Paredes et al., 2024; Novikov et al., 2025; https://github.com/google-deepmind/funsearch). We use a binary score s(A,W)=1s(A,W)=1 if the program uses at most L1(W)+4L_1(W)+4 bins on instance WW, where L1(W)L_1(W) is the total item size divided by 100 and rounded up. Otherwise, including errors and timeouts, s(A,W)=0s(A,W)=0. In each round, the proposer model (Qwen2.5-7B-Instruct) generates candidate modifications to the program implementing the packing policy, evolving over T=200T=200 rounds. We use an evaluation set SS with n∈{2,000, 10,000}n\in\lbrace 2{,}000,\thinspace 10{,}000\rbrace instances and HH of 50,000 instances. We run the eight methods from Table 1 with 10 seeds each. The results are summarized in the table below.

n = 2,000n = 10,000
MethodPop. Impr. (pp) ↑Prom.False ↓Pop. Impr. (pp) ↑Prom.False ↓
Empirical best-of-K18.37 ± 5.3415.64329.60 ± 5.3624.554
SICA-style16.43 ± 4.3815.63314.84 ± 6.2517.245
DGM-style17.88 ± 5.617.81214.09 ± 4.297.68
Elite26.75 ± 5.2223.36724.27 ± 5.7928.563
Niche-elite29.37 ± 3.819.41419.86 ± 5.488.512
PACE-style19.57 ± 4.666.3228.39 ± 5.3912.95
Bonferroni-style23.97 ± 5.742.1016.19 ± 6.183.20
REUSE (ours)35.27 ± 2.375.4031.40 ± 4.787.70

Experiment 2, code evolution on online bin packing task in FunSearch, T=200T=200 rounds. Pop. Impr. is the population improvement over the initial version (mean ±\pm standard error over seeds), Prom. is the average number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over runs.

The results demonstrate that REUSE successfully controls false promotions while reaching the highest true population improvement. Specifically, at n=2,000n=2{,}000, our method achieves an average population improvement of 35.27 pp, which is the highest among all methods, while committing zero false promotions. In contrast, the other six baselines (excluding Bonferroni-style) all have false promotions and achieve lower population improvements. Although Bonferroni-style does not have false promotions in this, it is overly conservative and exhibits lower power, resulting in a much lower number of promotions (2.1) and lower population improvement (23.97 pp) compared with our method. The same pattern holds true at n=10,000n=10{,}000.

Revision. We have incorporated both experiments into Section 3 with full protocols and results in Appendix B, and have upload a revised PDF with revisions highlighted in blue.

AuthorsOpenReview ↗

Response to Reviewer 4Kw3 (3/4)

W2 and Q2. Relation to reusable holdout methods and evaluation-data cost

Response. We thank the reviewer for this thoughtful question. We address both parts of your question. First, we explain three ways in which our guarantees and theoretical results go beyond those of [1, 2]. Second, we compare the evaluation-data costs theoretically and through experiments, showing that REUSE requires fewer evaluation examples to detect true population improvements of the same magnitude.

Methods in [1] and [2] address different evaluation problems from ours. In [1], Thresholdout estimates adaptively chosen statistics and SparseValidate validates them. In [2], Generic Holdout tests adaptively chosen hypotheses controlling false discoveries. In our setting of self-improving systems, each accepted candidate becomes the baseline for future comparisons, and each promotion decision influences which candidates are proposed next. Evaluation must therefore support a continuously evolving sequence of systems, rather than only individual statistical estimates or hypothesis tests. Our goal is to develop a systematic framework for evaluating self-improving systems that controls false promotions while maintaining the ability to detect true population improvements efficiently.

Although [1] and [2] provide tools for adaptive testing, applying them to our setting has three limitations. We compare these limitations with the corresponding guarantees and theoretical results of REUSE to clarify what our framework provides beyond applying [1, 2] to candidate–incumbent comparisons.

  1. Predefined budgets and anytime validity.

    Limitation of [1, 2]. These methods require a budget to be specified before the analysis begins. Once these budgets are exhausted, their guarantees no longer cover continued reuse of the holdout set, which is impractical for self-improving systems.

    What REUSE provides. Theorem 1 guarantees that, with probability at least 1−α1-\alpha, every promotion yields a true population improvement. This guarantee holds without a predefined limit on the number of rounds or promotions as REUSE allocates the error budget across possible candidate comparisons and decision histories.

  2. Evaluation-data cost and efficiency.

    Limitation of [1, 2]. Their statistical guarantees are calibrated to predefined budgets rather than the actual search history. When these budgets are set to guarantee statistical validity, more evaluation data are required to detect the same true population improvement, resulting in lower detection power at a fixed sample size.

    What REUSE provides. REUSE determines testing level δt,p\delta_{t,p} based on the current round tt and the number of previous promotions pp. This avoids paying for an unnecessarily large budget when few promotions occur, allowing true population improvements to be detected with fewer evaluation examples. Proposition 1 quantifies this efficiency through the minimum detectable improvement Δmin⁡\Delta_{\min}. Theorem A1 further shows that, for a class of promotion rules, such history-dependent statistical costs are necessary to control adaptive overfitting.

  3. Lower bounds on cumulative improvement.

    Limitation of [1, 2]. Their guarantees do not directly provide a statistically valid lower bound on cumulative true population improvement. This lower bound is important since it could establish how much the final system has improved over the initial system with statistical validity.

    What REUSE provides. Beyond controlling false promotions, REUSE provides statistically valid lower bounds on the cumulative true population improvement of the final system over the initial system. Specifically, Theorem 1 and Proposition A1 establish these lower bounds with statistical guarantees.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank all reviewers for their thoughtful comments and constructive suggestions. We appreciate the reviewers' recognition of the importance of reliable evaluation for self-improving AI systems.

The reviewers raised several important concerns. We have carefully considered these concerns and revised our manuscript accordingly. The main revisions are highlighted in blue in the revised PDF and are summarized below.

1. Additional experiments on broader self-improvement tasks.

A primary question raised by the reviewers concerns additional experiments. We conducted two new experiments involving a self-improving coding agent and code evolution. The results are consistent with our original experiments and demonstrate that REUSE can maintain high population improvement while controlling false promotions. We have added these experiments to Section 3, with detailed protocols in Appendix B.

2. Additional ablation studies.

We conducted further analyses to examine the contributions of REUSE's error allocation and decision-only feedback. These results are included in Appendix B.4.1.

3. Clarification of theoretical contributions.

We expanded the discussion of reusable holdout methods in Section 4 and provided additional comparisons of statistical guarantees and evaluation-data efficiency in our responses.

We hope these additional results and revisions address the reviewers' concerns. We have also provided detailed responses to each reviewer's specific questions. We sincerely appreciate the reviewers' time and feedback and welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.