acceptodds
Under review as a conference paper at ICLR 2027

Trustworthy Recursive Self-Improvement with Statistical Guarantees

Abstract

Recursive self-improvement offers a promising path toward AI agents that improve through experience by refining their own harnesses without retraining their underlying large language models (LLMs). Making this self-improvement process reliable, however, requires more than generating new updates: agents must decide which changes are worth keeping. A greedy gate that accepts updates based solely on higher validation scores can mistake an observed gain for genuine progress and lead to performance degeneration, whereas overly conservative statistical gates may reject beneficial updates and hinder performance improvement. We introduce Trustworthy Recursive Updates with Statistical Testing (TRUST), a framework that adapts an online multiple hypothesis testing procedure to control the rate of false evolution across rounds, with an LLM analyzing historical context to help determine how much evidence is required to update the incumbent. Under interpretable assumptions, we prove that TRUST limits the proportion of accepted updates that result in unacceptable performance loss. We further derive a lower bound on the final performance gain. Across the benchmarks HotpotQA, DROP, and APPS with the models DeepSeek-V4-Flash and Qwen-3.8-Flash, TRUST reduces the average ratio of accepted inferior updates from the greedy gate's 45.59% to 2.78%, while achieving higher final scores than the state-of-the-art statistical gate baselines for recursive self-improvement. These findings demonstrate that TRUST strikes a balance between statistical reliability and performance gain in recursive self-improvement.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
recursive self-improvement, harness, trustworthy gate, online multiple hypothesis testing, false evolution rate, e-value.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sm9mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the acceptance gate problem in recursive self-improvement (RSI) loops, and aims to design a reliable method to decide whether candidate agent‑harness updates should be kept. The paper proposes TRUST, an online multiple hypothesis testing procedure to control the rate of false evolution across sequential RSI rounds. TRUST defines False Evolution Rate (FER) to measure the expected fraction of accepted updates causing unacceptable performance degradation, and combines an e-value testing process with an overlap correction for reused validation examples, and an LLM-based context‑aware error‑budget allocator to adjust testing levels.

The paper provides a set theoretical guarantees, including FER control under an overlap-bias assumption, consistency for accepting truly beneficial candidates, and a lower bound on final population‑level performance gain. Experiments are performed on three benchmarks (HotpotQA, DROP, APPS) over two open‑source LLMs (DeepSeek‑V4‑Flash, Qwen‑3.8‑Flash). It results show that TRUST achieves the highest final gain, and reduces the mean FER and the mean maximum drawdown dramaticallly.

Critical strengths
  1. Well-motived, relatively underexplored problems: Not just proposing harness edits, but deciding whether an improved candidate should replace the existing one. The paper clearly describes the problems of existing approaches: greedy gates suffer high inferior‑update acceptance, and existing FWER‑based statistical gates are too conservative and produce zero improvement.

  2. Suitable formulation and theoretical treatment for closed‑loop adaptive RSI: The introduction of FER is a nice adaptation of false discovery reasoning to this setting, and it distinguishes unacceptable loss, tolerated loss, and strict improvement via the margin. Under certain assumptions, the paper proves FER control, TER consistency, and a lower bound on final deployment gain. The whole method is modular and interpretable.

  3. Meaningful and solid experimental results: Experiments run in a closed‑loop editable agent harness with four components, including reasoning, routing, memory, and workflow. Evaluation separates validation data (used for gate decisions) from held‑out test data (only for reporting metrics). Eexperiments are conducted with multiple seeds, clear baseline comparisons, comprehensive ablations, and illuminating case studies.

Critical weaknesses
  1. Guarantee depends on a very strong assumption about adaptation reuse: The guarantee in Theorem 1 and 2 depends on Assumption 1, which is an uncalibrated and potentially very strong assumption. There, \beta represents the strength of the correction for data reuse, but the paper does not provide a procedure for estimating a valid \beta. Because a small fraction of reused examples can still yield substantial adaptation bias if their outcomes are memorized, or used to modify a global harness. While the proof is a valid if Assumption 1 holds, the paper does not currently establish that the deployed procedure satisfies the premise needed for the advertised “trustworthy” protection.

  2. Tolerated losses accumulate across rounds: Each accepted update is compared only with the previous one. Tolerated losses up to \epsilon may accumulate over many accepted updates, and Theorem 4 does not provide conditions under which these terms are smaller than the positive-effect term P_T. Thus, while the result is a useful decomposition of "missed gain" cost and "accepted loss" cost, it does not establish the conclusion that TRUST yields a nonnegative final gain. The paper should address this issue, for example, by testing against the best incumbent using a cumulative margin budget.

Questions for authors

Small validation pool size and narrow experiment coverage: All experiments use a fixed validation pool consisting of only 200 examples. Real‑world agent systems may have access to much larger or streaming validation data. How do TRUST’s safety power tradeoffs behave as the number of validation samples increases? In addition, the scope of experiments have not established that the method transfers across agent architectures, richer tool-use environments, longer horizons, or settings where updates to harness are more complex.

2 comments

[Response to Weakness 1]

We appreciate that the reviewers raised questions about the choice of β\beta and sample size, which led us to reconsider these settings. Drawing on the different perspectives offered by all three reviewers, we have developed a more detailed response.

We acknowledge that the choice of β\beta is a limitation of this work. Our work aims to improve model safety through multiple hypothesis testing instead of adaptive data analysis. The parameters qq, ε\varepsilon, and β\beta each have an interpretable role and fit naturally into the framework. We use β=0.01\beta=0.01 as a shared default hyperparameter. To explain this choice and our experimental setup, we provide the following three analyses.

  • Empirical reuse bias. We updated Assumption 1 (p. 6) to bound the conditional reuse bias directly. This bias is half the difference between the expected gain on reused validation samples and the true gain, consistent with the intuition that validation reuse can lead to overly optimistic gain estimates. For every draw with ζt,i>0\zeta_{t,i} > 0, it requires 12E[Dt,i−Δt∣Ht,i−1,Qt,i=1]≤β\tfrac12\mathbb E[D_{t,i}-\Delta_t\mid\mathcal H_{t,i-1},Q_{t,i}=1]\le\beta. Lemma 1 (Appendix C, p. 25) derives the null mean bound we used in the former version.

    Our data span different histories, with too few observations under any one history to estimate the conditional bias directly. We therefore report the average reuse bias in each setting as a rough indication of its scale.

    b^=1∣S∣∑(s,t)∈SΔ^s,tvali−Δ^s,ttest2,\widehat b =\frac{1}{|\mathcal S|} \sum_{(s,t)\in\mathcal S} \frac{\widehat\Delta^{\mathrm{vali}}_{s,t}-\widehat\Delta^{\mathrm{test}}_{s,t}}{2},

    Δ^s,tvali=∑i=1τs,tQs,t,iDs,t,i∑i=1τs,tQs,t,i,Δ^s,ttest=∑j=1ntestDs,t,jtestntest,\widehat\Delta^{\mathrm{vali}}_{s,t} =\frac{\sum_{i=1}^{\tau_{s,t}}Q_{s,t,i}D_{s,t,i}} {\sum_{i=1}^{\tau_{s,t}}Q_{s,t,i}}, \widehat\Delta^{\mathrm{test}}_{s,t} =\frac{\sum_{j=1}^{n_{\mathrm{test}}}D^{\mathrm{test}}_{s,t,j}}{n_{\mathrm{test}}},

    Here, the validation mean averages the paired gains over reused samples (Qs,t,i=1Q_{s,t,i}=1) among the τs,t\tau_{s,t} recorded draws. The set S\mathcal S contains all seed and round pairs (s,t)(s,t) with at least one reused validation sample. The test mean uses the paired gains Ds,t,jtestD^{\mathrm{test}}_{s,t,j} on the ntestn_{\mathrm{test}} samples in the independent test set.

HotpotQA/DeepSeekHotpotQA/QwenDROP/DeepSeekDROP/QwenAPPS/DeepSeekAPPS/Qwen
0.0077710.0050700.0052450.004052-0.004919-0.001315

All six averages fall below β=0.01\beta=0.01, giving magnitude support for using this value as default. Remark. This calculation uses test set information and was performed after the experiments, which was not used to select the parameters before the experiments.

  • Mechanistic studies of β\beta. In Appendix B.2, we study how β\beta relates to the size of the validation pool and the amount of historical feedback supplied to the proposer. Table 5 (p. 17) compares N=200N=200 and 300300. Mean FER0\mathrm{FER}_0 falls from 23.3323.33\\% to 20.0020.00\\% at β=0\beta=0 and from 10.0010.00\\% to 00\\% at β=0.01\beta=0.01. It stays zero at β=0.03,0.05\beta=0.03,0.05. This indicates that the test becomes more conservative as NN increases. In other words, as NN decreases, sample reuse increases and the test becomes more aggressive. We also compare feedback from the most recent round with feedback from all completed rounds (Table 6, p. 18). Using the most recent round leads to one fewer accepted update across five seeds, suggesting slightly more conservative testing when less historical feedback is used. These patterns are consistent with the idea that reusing validation samples introduces optimistic bias, leading to more aggressive testing. So we need the overlap correction ζt,iβ\zeta_{t,i}\beta. They provide support for the connection between overlap and β\beta.

  • Performance and evaluation cost. Our choice of NN, nn, and TT also reflects evaluation cost. The validation resource comparisons are now shown together in Figure 7(d–f) (p. 10). Across these additional settings, TRUST achieves higher mean final gains than all four baselines, while SGM-B/H make no updates. The final gains of all five methods change only slightly relative to the main experiment when we increase TT, nn, or NN separately. Additional TRUST resource sensitivity trajectories remain in Figure 10 (Appendix B.4, p. 19). We therefore choose N=200,n=40,T=30N=200,n=40,T=30 to balance performance and evaluation cost.

[Response to Weakness 2]

We appreciate the reviewer’s thoughtful analysis and discussion of our theoretical decomposition and the further insights they provide.

Thus, while the result is a useful decomposition of "missed gain" cost and "accepted loss" cost, it does not establish the conclusion that TRUST yields a nonnegative final gain.

We have added Corollary 2 in Appendix F.2. Under its stated conditions, the lower bound on the round sample size nn in Equation (28) ensures G(T)>0G(T) > 0, as shown in Equation (29).

The paper should address this issue, for example, by testing against the best incumbent using a cumulative margin budget.

We thank the reviewer for suggesting this solution. We interpret the reviewer’s reference to “the best incumbent” in two ways.

In each round, our proposer generates three candidates. We use historical samples to select the best candidate and then test it against the currently deployed incumbent. The deployment question is whether this candidate can replace the system's running solver with controlled risk under our assumptions. The gate serves this decision and can complement methods for generating stronger candidates through feedback and reflection.

The true best historical incumbent is not directly observable during self-improvement: test results are reserved for reporting. To test against the best past version, we would have to keep a full copy of each past harness version. However, keeping and retesting all past harness versions would become costly as the run progresses.

[Response to Question 1]

All experiments use a fixed validation pool consisting of only 200 examples. Real‑world agent systems may have access to much larger or streaming validation data.

We report additional results for a larger validation pool in Figure 7(f) and for fresh validation in Figure 7(c) (p. 10). At N=300N=300, TRUST achieves the highest mean final gain. When each round uses fresh validation examples, TRUST's final gain is 11.79±0.6411.79\pm0.64 pp and its observed FER0\mathrm{FER}_{0} is 00\\%.

How do TRUST’s safety power tradeoffs behave as the number of validation samples increases?

We then jointly increased the validation pool size NN and the round sample size nn on HotpotQA/DeepSeek in Appendix B.4. Table 8 (p. 19) compares gain, TER, and FER0\mathrm{FER}_{0} across these settings. As (N,n)(N,n) increases, TER rises from 18.1118.11\\% to 24.5124.51\\% and 48.5948.59\\%, which is consistent with the convergence result in Theorem 3 under Assumption 2, while FER0\mathrm{FER}_{0} also rises from 10.0010.00\\% to 31.3331.33\\%. These settings reveal a trade-off between accepting a larger fraction of improving candidates and a higher false acceptance rate. Figure 12 (Appendix B.4, p. 20) additionally compares four noninferiority margins under the two larger resource settings. This suggests that achieving a higher gain also requires an appropriate choice of ε\varepsilon.

In addition, the scope of experiments have not established that the method transfers across agent architectures, richer tool-use environments, longer horizons, or settings where updates to harness are more complex.

Whether agents solving problems in complex environments face risks during self-improvement is an important question. We have made efforts to evaluate the gates on ToolHop [1], a complex tool-use benchmark that requires agents to interact with the environment. Figure 14 (Appendix B.7, p. 22) shows that TRUST achieves a mean final gain of 24.73±0.6924.73\pm0.69 pp, close to Greedy's 24.93±0.6924.93\pm0.69 pp and above PACE's 17.33±0.4217.33\pm0.42 pp. TRUST and PACE both have an observed mean FER0\mathrm{FER}_{0} of 00\\%, compared with 33.9033.90\\% for Greedy. SGM-B and SGM-H accept no updates, with zero gain. In this complex setting, we again observe a high risk of performance degradation with Greedy.

References

[1] Junjie Ye et al. ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use. ACL, 2025.

Reviewer_Vqg4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRUST, a statistical acceptance gate for recursive self-improvement agents. Instead of accepting a candidate update, the authors frame candidate acceptance as an online multiple-testing problem and introduces the FER as a cross-round safety criterion. TRUST combines e-value accumulation, noninferiority testing with a tolerated loss margin, overlap correction for reused validation samples, and an LLM-based allocator that assigns testing levels under a global error budget. Experiments show that TRUST achieves the highest average final gain of 24.84 pp across settings and reduces mean FER from 45.59% to 2.78% and mean maximum drawdown from 4.963 pp to 0.047 pp relative to Greedy, while remaining within a low-drawdown safe region.

Critical strengths
  1. The paper formulates recursive candidate acceptance as an online multiple-testing problem and introduces a meaningful cross-round safety criterion. TRUST integrates e-value testing, noninferiority, overlap correction, and LLM-guided budget allocation while preserving statistical guarantees; notably, LLM guidance does not invalidate FER control.

  2. The empirical evaluation is broad and convincing: across multiple datasets and models, TRUST improves final gain while greatly reducing false acceptances and drawdowns relative to greedy and statistical baselines. Ablations and case studies further clarify the roles of key parameters and LLM allocation.

Critical weaknesses
  1. The FER guarantee depends on Assumption 1 and the choice of beta, but beta is tuned by ablation rather than estimated from data. The asymptotic conditions in Theorems 3–4 (N and n growing, n/N tending to 0) also do not match the small validation pool used in experiments, so it is unclear whether the guarantee is operative in the reported regime.

  2. The experiments report FER(0), not the theoretical FER(epsilon), so the central FER guarantee is not directly tested. TRUST’s advantage over PACE is also modest in some settings and has large variance, making the empirical case less decisive than the headline numbers suggest.

Questions for authors
  1. Can you report the LLM-allocator ablation across more settings with variance, and discuss the additional LLM cost and latency?
2 comments

[Response to Weakness 1]

We appreciate that the reviewers raised questions about the choice of β\beta and sample size, which led us to reconsider these settings. Drawing on the different perspectives offered by all three reviewers, we have developed a more detailed response.

We acknowledge that the choice of β\beta is a limitation of this work. Our work aims to improve model safety through multiple hypothesis testing instead of adaptive data analysis. The parameters qq, ε\varepsilon, and β\beta each have an interpretable role and fit naturally into the framework. We use β=0.01\beta=0.01 as a shared default hyperparameter. To explain this choice and our experimental setup, we provide the following three analyses.

  • Empirical reuse bias. We updated Assumption 1 (p. 6) to bound the conditional reuse bias directly. This bias is half the difference between the expected gain on reused validation samples and the true gain, consistent with the intuition that validation reuse can lead to overly optimistic gain estimates. For every draw with ζt,i>0\zeta_{t,i} > 0, it requires 12E[Dt,i−Δt∣Ht,i−1,Qt,i=1]≤β\tfrac12\mathbb E[D_{t,i}-\Delta_t\mid\mathcal H_{t,i-1},Q_{t,i}=1]\le\beta. Lemma 1 (Appendix C, p. 25) derives the null mean bound we used in the former version.

    Our data span different histories, with too few observations under any one history to estimate the conditional bias directly. We therefore report the average reuse bias in each setting as a rough indication of its scale.

    b^=1∣S∣∑(s,t)∈SΔ^s,tvali−Δ^s,ttest2,\widehat b =\frac{1}{|\mathcal S|} \sum_{(s,t)\in\mathcal S} \frac{\widehat\Delta^{\mathrm{vali}}_{s,t}-\widehat\Delta^{\mathrm{test}}_{s,t}}{2},

    Δ^s,tvali=∑i=1τs,tQs,t,iDs,t,i∑i=1τs,tQs,t,i,Δ^s,ttest=∑j=1ntestDs,t,jtestntest,\widehat\Delta^{\mathrm{vali}}_{s,t} =\frac{\sum_{i=1}^{\tau_{s,t}}Q_{s,t,i}D_{s,t,i}} {\sum_{i=1}^{\tau_{s,t}}Q_{s,t,i}}, \widehat\Delta^{\mathrm{test}}_{s,t} =\frac{\sum_{j=1}^{n_{\mathrm{test}}}D^{\mathrm{test}}_{s,t,j}}{n_{\mathrm{test}}},

    Here, the validation mean averages the paired gains over reused samples (Qs,t,i=1Q_{s,t,i}=1) among the τs,t\tau_{s,t} recorded draws. The set S\mathcal S contains all seed and round pairs (s,t)(s,t) with at least one reused validation sample. The test mean uses the paired gains Ds,t,jtestD^{\mathrm{test}}_{s,t,j} on the ntestn_{\mathrm{test}} samples in the independent test set.

HotpotQA/DeepSeekHotpotQA/QwenDROP/DeepSeekDROP/QwenAPPS/DeepSeekAPPS/Qwen
0.0077710.0050700.0052450.004052-0.004919-0.001315

All six averages fall below β=0.01\beta=0.01, giving magnitude support for using this value as default. Remark. This calculation uses test set information and was performed after the experiments, which was not used to select the parameters before the experiments.

  • Mechanistic studies of β\beta. In Appendix B.2, we study how β\beta relates to the size of the validation pool and the amount of historical feedback supplied to the proposer. Table 5 (p. 17) compares N=200N=200 and 300300. Mean FER0\mathrm{FER}_0 falls from 23.3323.33\\% to 20.0020.00\\% at β=0\beta=0 and from 10.0010.00\\% to 00\\% at β=0.01\beta=0.01. It stays zero at β=0.03,0.05\beta=0.03,0.05. This indicates that the test becomes more conservative as NN increases. In other words, as NN decreases, sample reuse increases and the test becomes more aggressive. We also compare feedback from the most recent round with feedback from all completed rounds (Table 6, p. 18). Using the most recent round leads to one fewer accepted update across five seeds, suggesting slightly more conservative testing when less historical feedback is used. These patterns are consistent with the idea that reusing validation samples introduces optimistic bias, leading to more aggressive testing. So we need the overlap correction ζt,iβ\zeta_{t,i}\beta. They provide support for the connection between overlap and β\beta.

  • Performance and evaluation cost. Our choice of NN, nn, and TT also reflects evaluation cost. The validation resource comparisons are now shown together in Figure 7(d–f) (p. 10). Across these additional settings, TRUST achieves higher mean final gains than all four baselines, while SGM-B/H make no updates. The final gains of all five methods change only slightly relative to the main experiment when we increase TT, nn, or NN separately. Additional TRUST resource sensitivity trajectories remain in Figure 10 (Appendix B.4, p. 19). We therefore choose N=200,n=40,T=30N=200,n=40,T=30 to balance performance and evaluation cost.

[Response to Weakness 2]

The experiments report FER(0), not the theoretical FER(epsilon), so the central FER guarantee is not directly tested.

FER0\mathrm{FER}_{0} is a stricter diagnostic because it counts all non-improving accepted updates, including losses tolerated by FERε\mathrm{FER}_{\varepsilon}. For the same acceptance decisions AtA_t, write RT=∑t≤TAtR_T=\sum_{t\le T}A_t. For any ε≥0\varepsilon\ge0,

FTε=∑t≤TAt1Δt≤−ε≤∑t≤TAt1Δt≤0=FT0,F_T^{\varepsilon} =\sum_{t\le T}A_t\mathbf{1}\\{\Delta_t\le-\varepsilon\\} \le\sum_{t\le T}A_t\mathbf{1}\\{\Delta_t\le0\\}=F_T^0,

FERε(T)=E![FTεRT∨1]≤E![FT0RT∨1]=FER0(T).\mathrm{FER}_{\varepsilon}(T) =\mathbb{E}\\!\left[\frac{F_T^{\varepsilon}}{R_T\vee1}\right] \le\mathbb{E}\\!\left[\frac{F_T^0}{R_T\vee1}\right] =\mathrm{FER}_0(T).

The same ordering holds for empirical estimates: FER^ε≤FER^0\widehat{\mathrm{FER}}_{\varepsilon}\le\widehat{\mathrm{FER}}_0, so FER^0<q\widehat{\mathrm{FER}}_0 < q implies FER^ε<q\widehat{\mathrm{FER}}_{\varepsilon} < q. In the paper's Table 1, TRUST's mean empirical FER0\mathrm{FER}_{0} ranges from 00 to 1010\\%, below q=30q=30\\%. This stricter metric therefore provides support for Theorem 2.

TRUST’s advantage over PACE is also modest in some settings and has large variance, making the empirical case less decisive than the headline numbers suggest.

We agree that TRUST's advantage over PACE is not uniformly large and that the estimates have uncertainty. All methods share the proposer configuration, so the role of the different gates is to screen for good candidates as effectively as possible, with no guarantee of a better final result. Table 1 reports higher mean gain and TER for TRUST than PACE in all six settings. On APPS/DeepSeek, gain is 34.55±1.0434.55\pm1.04 versus 28.68±2.6628.68\pm2.66 pp. On DROP/Qwen, it is 17.22±1.4917.22\pm1.49 versus 13.22±3.7713.22\pm3.77 pp. These are relative increases of approximately 20.520.5\\% and 30.230.2\\%. TER is also higher: 11.6411.64\\% versus 5.195.19\\% and 10.7610.76\\% versus 5.235.23\\%, respectively. Additionally, TRUST performs better than PACE in almost all supplementary experiments.

[Response to Question 1]

Can you report the LLM-allocator ablation across more settings with variance, and discuss the additional LLM cost and latency?

We expanded the LLM-allocator ablation to HotpotQA/DROP ×\times DeepSeek/Qwen. Figure 15 (Appendix B, p. 22) shows the complete trajectories. The LLM allocator yields higher mean final gains in three settings (+1.13+1.13, +1.10+1.10, and +0.57+0.57 pp) and ties on HotpotQA/Qwen.

We call the LLM allocator once per round, so a full self-improvement run of TT rounds requires TT calls. Table 10 (Appendix B.8, p. 22) reports mean total token usage and time per call across HotpotQA, DROP, and APPS with DeepSeek and Qwen. The cost per call depends on how much context we pass to the LLM. The reported measurements show that the allocator adds only modest cost in these settings.

Reviewer_LyeXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes TRUST, a statistical way to decide whether an agent should accept each update during recursive self-improvement. It limits wrong updates, e.g., ones with a higher validation score but worse real performance, by controlling the proportion of harmful updates among all accepted ones, rather than forbidding any mistake, which makes existing statistical gates too conservative. Theoretically, it proves that this proportion stays below a user-specified level, that truly improving updates are accepted with probability approaching one as validation samples grow, and gives a lower bound on the final performance gain. Across three benchmarks and two LLMs, TRUST reduces the ratio of accepted inferior updates from 45.59% (greedy gate) to 2.78%, while achieving the highest final gain.

Critical strengths

Although I am not very familiar with online multiple hypothesis testing, I find the main idea intuitive and well motivated. The paper formulates each acceptance decision as a statistical test and aims to control the proportion of harmful updates across rounds. This is also a practically important problem for self-improving agents.

Critical weaknesses

I did not identify any critical weaknesses that would warrant outright rejection, but I have a few concerns and questions that I hope the authors can address.

  1. Understanding the source of improvement. TRUST uses the same validation feedback as the greedy gate but introduces statistical testing and adaptive thresholds for acceptance decisions. I wonder whether part of the improvement could also be achieved by simpler, more conservative acceptance rules, such as greedy acceptance with a fixed margin. Such a comparison might help better isolate the contribution of the proposed framework. Additionally, Table 8 shows that TRUST accepts no updates when ε = 0, which raises the question of how sensitive its advantage over PACE is to the choice of the noninferiority margin. It would also be helpful to understand the additional computational overhead, if any, introduced by the framework.

  2. Understanding Assumption 1. I am not very familiar with the literature on online multiple hypothesis testing, so I may be missing some standard arguments here. My understanding is that the theoretical guarantee relies on Assumption 1, which bounds the bias arising from adaptive reuse of validation data by β. Given that the validation pool contains 200 examples and is reused over 30 rounds, I wonder how reasonable this assumption is in the experimental setting. In particular, could the authors provide more intuition or justification for choosing β = 0.01? Table 9 also suggests that the choice of β can affect performance, so some additional discussion would be helpful.

  3. Generalizability of the empirical results. The experiments show that TRUST can substantially reduce harmful acceptances while maintaining competitive final performance. However, the improvement in final scores over the greedy baseline appears relatively modest in some settings, and greedy achieves higher final scores in two of the six settings in Table 1, although with higher FER₀. I wonder whether the authors could further discuss this trade-off. Additionally, Appendix G.1 describes prespecified weaknesses in the initial harnesses. It would be interesting to understand whether similar benefits can be expected in more realistic settings without such predefined weaknesses. Finally, since the experiments are limited to 30 rounds, evaluating or discussing longer self-improvement runs could help strengthen the empirical evidence.

Questions for authors

Please refer to the Weaknesses section.

4 comments

We thank the reviewer for recognizing the central idea of our framework and suggesting additional experiments to examine the different mechanisms.

[Response to Weakness 1]

I wonder whether part of the improvement could also be achieved by simpler, more conservative acceptance rules, such as greedy acceptance with a fixed margin. Such a comparison might help better isolate the contribution of the proposed framework.

We added three Greedy variants with different acceptance margins. Table 7 (Appendix B.3, p. 19) summarizes their results, and Figure 9(a) (Appendix B.3, p. 18) shows their trajectories. None achieves a higher final gain than TRUST. These Greedy variants offer some safety benefits, but their fixed margins cannot adapt to the validation samples observed at different rounds.

Additionally, Table 8 shows that TRUST accepts no updates when ε=0\varepsilon=0, which raises the question of how sensitive its advantage over PACE is to the choice of the noninferiority margin.

We also report a PACE variant in Figure 9(b) (Appendix B.3, p. 18). Following PACE's construction of the test statistic, we change the win condition from Dt,i>0D_{t,i}>0 to Dt,i>−εD_{t,i}>-\varepsilon with ε=0.10\varepsilon=0.10. This variant reaches a final gain of 6.18±1.496.18\pm1.49 pp, with FER0=52.67%\mathrm{FER}_0=52.67\%. PACE tests win probability, and the test statistic construction with ε=0.10\varepsilon=0.10 considers ties as wins. When most observations are ties and therefore treated as wins, the acceptance can be aggressive and unstable. Our framework uses score differences directly without converting them into win/loss labels, and naturally incorporates a noninferiority margin ε\varepsilon to improve statistical power.

It would also be helpful to understand the additional computational overhead, if any, introduced by the framework.

We call the LLM allocator once per round, so a whole self-improvement run of TT rounds requires TT additional calls for TRUST. Table 10 (Appendix B.8, p. 22) reports mean total token cost and time per call. The cost per call depends on how much context we pass to the LLM, and the time per call depends on network latency and other factors. The reported measurements show that the allocator adds modest cost in these settings.

[Response to Weakness 3]

The trade-off between safety and performance is an important issue in our work.

However, the improvement in final scores over the greedy baseline appears relatively modest in some settings, and greedy achieves higher final scores in two of the six settings in Table 1, although with higher FER₀. I wonder whether the authors could further discuss this trade-off.

In Table 1, TRUST finishes with a higher mean gain than Greedy in four of the six settings and a lower FER0\mathrm{FER}_0 in all six. Greedy achieves higher final gains on DROP/DeepSeek and APPS/Qwen. TRUST satisfies the safety guarantee at any stopping time τ≤30\tau\le30 adapted to the observed evolution history. Figure 13 (Appendix B.5, p. 21) shows that stopping Greedy during a downturn can leave the deployed solver below an earlier performance level. On DROP/DeepSeek, for example, Greedy's mean gain is below TRUST's in rounds 3–5. FER0\mathrm{FER}_0 measures the share of accepted updates with zero or negative gains, so a high FER0\mathrm{FER}_0 does not necessarily imply a poor final gain. It reflects one aspect of risk. Additionally, Greedy has a larger maximum drawdown in all six settings, with 5.385.38 and 3.593.59 percentage points on DROP/DeepSeek and APPS/Qwen, respectively.

Additionally, Appendix G.1 describes prespecified weaknesses in the initial harnesses. It would be interesting to understand whether similar benefits can be expected in more realistic settings without such predefined weaknesses.

The prespecified initial weaknesses give the proposer clear improvement signals. We also evaluate all five gates with a natural initialization obtained by removing the explicitly flawed instructions. Table 9 (Appendix B.6, p. 21) and Figure 7(b) (p. 10) report the results. Greedy has a final gain of −6.91±1.04-6.91\pm1.04 pp, with FER0=67.95%\mathrm{FER}_0=67.95\%. TRUST, PACE, and SGM-B/H preserve the initial performance with FER0=0\mathrm{FER}_0=0. This experiment reveals the risk associated with Greedy.

Finally, since the experiments are limited to 30 rounds, evaluating or discussing longer self-improvement runs could help strengthen the empirical evidence.

At the longer horizon T=50T=50 (Figure 7(d) (p. 10)), TRUST maintains a final gain of 12.14±0.8012.14\pm0.80 pp, compared with 10.46±0.9910.46\pm0.99 pp for PACE and 7.45±0.527.45\pm0.52 pp for Greedy. Figure 7(c) (p. 10) also examines a setting where the available validation pool equals TnTn, allowing each round to use fresh examples without reuse. TRUST also achieves the highest final gain. In addition, Table 8 and Figures 11–12 (Appendix B.4, pp. 19–20) report experiments that increase NN and nn to examine TRUST. Here, we validate the asymptotic property in Theorem 3.

[Response to Weakness 2]

We appreciate that the reviewers raised questions about the choice of β\beta and sample size, which led us to reconsider these settings. Drawing on the different perspectives offered by all three reviewers, we have developed a more detailed response.

We acknowledge that the choice of β\beta is a limitation of this work. Our work aims to improve model safety through multiple hypothesis testing instead of adaptive data analysis. The parameters qq, ε\varepsilon, and β\beta each have an interpretable role and fit naturally into the framework. We use β=0.01\beta=0.01 as a shared default hyperparameter. To explain this choice and our experimental setup, we provide the following three analyses.

  • Empirical reuse bias. We updated Assumption 1 (p. 6) to bound the conditional reuse bias directly. This bias is half the difference between the expected gain on reused validation samples and the true gain, consistent with the intuition that validation reuse can lead to overly optimistic gain estimates. For every draw with ζt,i>0\zeta_{t,i} > 0, it requires 12E[Dt,i−Δt∣Ht,i−1,Qt,i=1]≤β\tfrac12\mathbb E[D_{t,i}-\Delta_t\mid\mathcal H_{t,i-1},Q_{t,i}=1]\le\beta. Lemma 1 (Appendix C, p. 25) derives the null mean bound we used in the former version.

    Our data span different histories, with too few observations under any one history to estimate the conditional bias directly. We therefore report the average reuse bias in each setting as a rough indication of its scale.

    b^=1∣S∣∑(s,t)∈SΔ^s,tvali−Δ^s,ttest2,\widehat b =\frac{1}{|\mathcal S|} \sum_{(s,t)\in\mathcal S} \frac{\widehat\Delta^{\mathrm{vali}}_{s,t}-\widehat\Delta^{\mathrm{test}}_{s,t}}{2},

    Δ^s,tvali=∑i=1τs,tQs,t,iDs,t,i∑i=1τs,tQs,t,i,Δ^s,ttest=∑j=1ntestDs,t,jtestntest,\widehat\Delta^{\mathrm{vali}}_{s,t} =\frac{\sum_{i=1}^{\tau_{s,t}}Q_{s,t,i}D_{s,t,i}} {\sum_{i=1}^{\tau_{s,t}}Q_{s,t,i}}, \widehat\Delta^{\mathrm{test}}_{s,t} =\frac{\sum_{j=1}^{n_{\mathrm{test}}}D^{\mathrm{test}}_{s,t,j}}{n_{\mathrm{test}}},

    Here, the validation mean averages the paired gains over reused samples (Qs,t,i=1Q_{s,t,i}=1) among the τs,t\tau_{s,t} recorded draws. The set S\mathcal S contains all seed and round pairs (s,t)(s,t) with at least one reused validation sample. The test mean uses the paired gains Ds,t,jtestD^{\mathrm{test}}_{s,t,j} on the ntestn_{\mathrm{test}} samples in the independent test set.

HotpotQA/DeepSeekHotpotQA/QwenDROP/DeepSeekDROP/QwenAPPS/DeepSeekAPPS/Qwen
0.0077710.0050700.0052450.004052-0.004919-0.001315

All six averages fall below β=0.01\beta=0.01, giving magnitude support for using this value as default. Remark. This calculation uses test set information and was performed after the experiments, which was not used to select the parameters before the experiments.

  • Mechanistic studies of β\beta. In Appendix B.2, we study how β\beta relates to the size of the validation pool and the amount of historical feedback supplied to the proposer. Table 5 (p. 17) compares N=200N=200 and 300300. Mean FER0\mathrm{FER}_0 falls from 23.3323.33\\% to 20.0020.00\\% at β=0\beta=0 and from 10.0010.00\\% to 00\\% at β=0.01\beta=0.01. It stays zero at β=0.03,0.05\beta=0.03,0.05. This indicates that the test becomes more conservative as NN increases. In other words, as NN decreases, sample reuse increases and the test becomes more aggressive. We also compare feedback from the most recent round with feedback from all completed rounds (Table 6, p. 18). Using the most recent round leads to one fewer accepted update across five seeds, suggesting slightly more conservative testing when less historical feedback is used. These patterns are consistent with the idea that reusing validation samples introduces optimistic bias, leading to more aggressive testing. So we need the overlap correction ζt,iβ\zeta_{t,i}\beta. They provide support for the connection between overlap and β\beta.

  • Performance and evaluation cost. Our choice of NN, nn, and TT also reflects evaluation cost. The validation resource comparisons are now shown together in Figure 7(d–f) (p. 10). Across these additional settings, TRUST achieves higher mean final gains than all four baselines, while SGM-B/H make no updates. The final gains of all five methods change only slightly relative to the main experiment when we increase TT, nn, or NN separately. Additional TRUST resource sensitivity trajectories remain in Figure 10 (Appendix B.4, p. 19). We therefore choose N=200,n=40,T=30N=200,n=40,T=30 to balance performance and evaluation cost.

I thank the authors for their response and clarifications. They have addressed my major concerns, and I am now happy to support the paper. I will raise my score accordingly.

Thank you for taking the time to consider our response and revised manuscript. We’re glad our clarifications addressed your main concerns, and we appreciate your support. Your feedback helped us improve the manuscript.

Comments on the paper

2 comments

Dear reviewers,

Thank you for your insightful comments. We have provided separate responses for each review.

In response to your questions, we supplemented the manuscript with a range of experiments in Appendix B, including Mechanistic study of β\beta, Greedy and PACE variants, Validation resource sensitivity, Deployment trajectories across six settings, Natural initialization, Tool-use scenarios, and LLM allocator analysis. Figure 7 (p. 10) reports more results for Natural initialization, Fresh validation, Longer horizon, Larger sample, and Larger pool.

We have uploaded the revised manuscript, with changes marked in light blue. We would be grateful if you could take a moment to read our responses and supplementary experiments. Thank you again for your time and constructive feedback.

We are grateful to the reviewers for their constructive feedback. The reviews identified the following strengths:

The candidate solver deployment is an important and well motivated problem in RSI (LyeX, Vqg4, sm9m). The online multiple hypothesis testing is suitable for gate in RSI and the FER is a meaningful safety criterion (LyeX, Vqg4, sm9m). TRUST provided theoretical guarantees (Vqg4, sm9m). LLM guidance does not invalidate FER control notably (Vqg4). The experiments across seeds, tasks, and models, including ablations, are broad and solid (Vqg4, sm9m).

The reviewers also raised several important questions below, which we have responded with additional experiments or detailed explanations.

(1)Validation reuse and the choice of β. We revised Assumption 1 to state the conditional reuse bias bound more directly and intuitively. We provided scale evidence of reuse bias with β = 0.01, added experiments to clarify how β works, and explained the sample size choice.

(2)LLM allocator. We compared TRUST with Greedy and PACE variants to show TRUST’s adaptive benefits. We added experiments with natural initialization, fresh validation, larger validation pools, more samples per round, longer runs, and a more complex tool-use setting.

(3)Additional baselines and evaluation settings. We expanded the LLM allocator ablation to more settings and report token usage and latency across six settings.

(4)When final gain is positive. Building on Theorem 4, we derived a threshold of sample size per round for positive final gain (Corollary 2, Appendix F.2).

We thank the reviewers again for the time and effort they devoted to reviewing our paper. Their feedback helped us strengthen the quality of the manuscript. We gently ask whether any questions or concerns remain unresolved and we would be pleased to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.