acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Gated Research: Statistically Controlled Model Adoption in Adaptive Search

Abstract

An accepted model becomes the reference for later proposals in adaptive search, so an erroneous replacement can alter hypotheses that have not yet been generated. Evidence-Gated Research (EGR) studies model adoption when improvement must hold across a predeclared set of environments. EGR freezes each challenger before decision evidence is examined, constructs anytime-valid environment-specific evidence under predictable sampling, and passes their intersection–union evidence to an online adoption controller. Under a conditional mean bound for each sampled evidence unit, the procedure controls false discovery rate for the declared all-environment adoption target even when earlier adoptions change later challengers and null hypotheses. In a 5,000-trajectory closed-loop benchmark, development-only e-LOND has persistent-target FDR 0.621, whereas no persistent false-adoption path is observed for the audited EGR variants in that finite run. In matched replay over 600 challenger–incumbent pairs, Stagewise EGR and fixed-anytime IUT have identical adoption events by construction; at \(h = 20\), Stagewise uses 56.1% less decision evidence by stopping resolved components and avoiding later stages after failure. EGR-ALLOC further reallocates saved evidence to unresolved environments. Three-environment public-data and controlled neural studies show similar reductions in decision-evidence cost. These results characterize when sequential evidence can support statistically controlled model replacement in an adaptive search.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
adaptive model selection, sequential inference, anytime-valid inference, false discovery rate, online multiple testing, model adoption, adaptive search, reliable machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BxVdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Evidence-Gated Research (EGR), a framework as a gate to control model replacement in adaptive search. EGR attempts to ensure that improvements by each model iteration are consistent across multiple environments, and replacements are not misled by repeated testing. To achieve this objective, EGR proposes multiple modules in the control process, such as candidate freezing (before accessing decision evidence) and adaptive testing strategy (to avoid false positives), and further proposes several efficiency improvements, such as ALLOC and Balance, to reduce the cost of testing. Experiments on multiple tasks demonstrate the effectiveness and efficiency of EGR.

Critical strengths
  • Clear problem formulation: the paper targets and mitigates the important problems of cross-environment generalization and false positives in adaptive search.
  • Strong theoretical proof: the method offers clear proof and is mathematically grounded.
Critical weaknesses
  • Unclear methodological novelty: despite the theoretical support, most modules in the method are adopted from existing methods. The contribution of the method is more in the careful combination of these modules.
  • Lack of experiments in real-world complex scenarios: the experiments use synthetic searches, digits, and controlled MLP banks, but do not involve real-world complex tasks or models (e.g., LLMs on more challenging language tasks).
Questions for authors
  • Can the authors demonstrate the effectiveness of the method under more realistic task settings, such as LLMs for language tasks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BxVd

We thank the reviewer for the positive evaluation. We took the request for a realistic LLM experiment seriously, and the revised paper now includes one (§5.2, App. D.5 and E.9; changes in blue).

Setup. We use the per-instance AlpacaEval 2.0 judgments of 26 open-weight chat models from 11 families (including Qwen, Llama 2/3/3.1, Mistral and Mixtral, Yi, DBRX, Tülu, Vicuna, Gemma and Phi), ranging from 1.8B to 141B parameters. The 805 instructions come from five sources, which we use as environments: Self-Instruct for development, OASST and Anthropic helpful-base for audit, and Koala and Vicuna held out. In each of 15 rounds, six candidate models are scored on a small development scout set, the best is frozen and tested against the current model, and if adopted it becomes the model that all later candidates are compared with.

Results over 2,000 runs (Table 4).

MethodPersistent FDRRuns with a false replacementPowerPaired evaluations per candidate
Dev mean.2641139.985400
Dev e-LOND.063266.817371
Multi-env mean.052239.9431200
Fixed-anytime IUT00.7201200
Stagewise EGR00.720392
EGR-ALLOC.00031.738398

Gating on development data alone leads to a false replacement in more than half of the runs with a simple mean test, and in 13% with e-LOND. EGR makes at most one false replacement in 2,000 runs, and uses about a third of the evidence of fixed IUT at the same power. Over 24 additional settings (search pressure, α\alpha, budget, margin, five data splits, binary utility, spending schedules, proposer type and starting model), EGR's empirical FDR never exceeds .0010, while the development-only mean gate ranges from .165 to .390. Removing any one model family (Qwen, Llama or Mixtral) leaves Stagewise power between .726 and .780 (Table 12).

One limitation, which we state in the paper: the loop is replayed on existing models and precomputed judgments rather than run by an agent that trains new ones. This is what lets us know the true answer for every comparison and measure FDR exactly.

On novelty, we agree that the building blocks exist. The revised Table 1 separates what is inherited from what is new: the treatment of nulls that depend on earlier adoptions, validity under adaptive routing and asynchronous stopping across environments (Theorem 1, Lemma 1), and counterexamples showing which common shortcuts break the guarantee, such as reading a sealed holdout without replacement (Prop. 4).

Thank you again for your support. We are happy to answer any further questions, and we hope the new results strengthen your confidence in the paper.

The experiments have addressed my related concerns. I maintain my original rating (3).

AuthorsOpenReview ↗

Thank you, and two further checks on the LLM study

Dear Reviewer BxVd,

Thank you for considering our earlier response and the newly added LLM experiments. We have since performed two further checks that strengthen the empirical evaluation, with the corresponding revisions highlighted in blue.

1. Reliability on previously unread instructions.

The empirical FDR reported in Table 4 is evaluated relative to the declared decision pools. We have now added an out-of-pool persistence diagnostic (Appendix E.10, Table 22), which re-evaluates every adoption using instructions that were never read during decision sampling.

On these instructions, Stagewise EGR and EGR-ALLOC achieve empirical FDR values of 0.014 and 0.022, respectively, compared with 0.267 for the development-mean gate.

These results provide additional evidence that the reliability improvements are not confined to the particular decision samples, although they do not establish a theoretical guarantee for unseen deployment populations.

2. Verification of the LLM benchmark data.

We compared the 26-model utility bank against the public AlpacaEval annotation files. All 20,926 available judgments match exactly, and we identified the four missing judgment positions.

Assigning either 0 or 1 to these missing values leaves the audited empirical FDR unchanged (Appendix E.6, Table 13).

We also redesigned Figure 1 to make the adoption loop and incumbent updates more explicit, and redrew Figure 2 to improve readability at print size.

We hope these additional checks further strengthen the empirical evidence and address the remaining presentation concerns. We would be happy to clarify any outstanding questions.

The authors

Reviewer_zWPUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evidence-Gated Research (EGR), a framework for deciding whether a challenger model should replace the current model during an adaptive model search. EGR requires improvement across a predeclared set of environments, freezes each challenger before examining decision evidence, constructs anytime-valid evidence for each environment, and combines this evidence using an intersection–union test. The resulting candidate-level evidence is passed to e-LOND to control the false discovery rate across repeated, history-dependent replacement decisions. The paper also proposes Stagewise EGR, EGR-ALLOC, and EGR-Balance to reduce or reallocate evidence collection. The authors provide theoretical guarantees under a conditional validity assumption and evaluate the framework using synthetic experiments, a public dataset, and controlled neural network experiments.

Critical strengths
  • The paper studies an important problem in adaptive model development: an incorrect model replacement changes the incumbent and can therefore affect future candidates and hypotheses. The theoretical framework accounts for these history-dependent replacement decisions and provides FDR and FWER guarantees under clearly stated conditions.

  • The experiments examine several different failure modes, including target mismatch, repeated testing, dependent evidence units, and reuse of evidence after candidate selection. The matched replay experiments also help separate evidence-collection efficiency from changes in the search path. These experiments provide useful evidence about when the statistical guarantees can and cannot be expected to hold.

Critical weaknesses
  • The methodological novelty beyond combining existing tools is not fully clear. EGR uses existing e-processes, intersection–union tests, and e-LOND, and the paper explicitly states that it does not introduce a new online testing controller. Although the moving-incumbent setting is useful, it is unclear which technical contribution goes beyond applying existing statistical tools to this setting and how substantial that contribution is.

  • The practical usefulness of the guarantees remains unclear. Assumption 1 may be difficult to verify in realistic ML systems where datasets are reused and evaluation results affect later development. The guarantees also depend on the selected environments and improvement thresholds, but the paper provides limited guidance for making these choices. In addition, the reported power is relatively low under the evaluated EGR procedures and depends strongly on the downstream online controller. These issues may limit the practical usefulness of the framework even when its formal guarantees hold.

Questions for authors
  • Which theoretical result would not follow from directly combining existing anytime-valid intersection–union evidence with e-LOND? Please clarify the specific technical contribution introduced by EGR beyond applying these existing tools to a moving-incumbent model-search setting.

  • How can a practitioner verify Assumption 1 in a realistic ML pipeline where training, model selection, and evaluation may use related or repeatedly accessed datasets? Is there an implementable procedure for determining whether an evidence unit remains conditionally valid?

  • How should users choose the declared environments and improvement thresholds? How sensitive are the adoption decisions and power to these choices and to the choice of downstream online controller?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zWPU (1/2)

We thank the reviewer for a thoughtful review and for very concrete questions. All changes are marked in blue in the revised PDF.

Q1. What does not follow from combining existing tools? The FDR bound itself does follow from e-LOND once each candidate comes with a valid e-value, and Theorem 2 now says so. What does not follow is that the e-value we construct is valid when the hypothesis was produced by the search. Three points need an argument.

(i) The null indicator is random. Whether a challenger is a null depends on the incumbent, which depends on earlier decisions (Appendix A: the same second challenger is a null or not depending on D1D_1). Validity must therefore be stated with the indicator inside the conditional expectation, E[νtEt∣Ft−1]≤νt\mathbb{E}[\nu_t E_t \mid \mathcal F_{t-1}] \le \nu_t, and freezing the comparison before decision evidence is what makes this attainable.

(ii) The intersection–union step is not the textbook one. Standard IUT arguments fix which component is null and use fixed sample sizes. Here the null component depends on the history, evidence is routed adaptively across environments, and each component stops at a different time. Lemma 1 (conditioning on the first true component, which is fixed before sampling) and Theorem 1 (routed products on a joint filtration) show that the minimum remains valid, with no independence across environments.

(iii) Natural relaxations fail. Prop. 4 gives five counterexamples. Two are easy to fall into in practice: reusing evidence after selecting among variants, and reading a sealed holdout without replacement, which gives E[Eτ]=1+λ(1−λ)/2>1\mathbb{E}[E_\tau] = 1+\lambda(1-\lambda)/2 > 1.

None of these is a deep new inequality, and we no longer present them as one. Our claim is that they are the pieces needed to use existing sequential tests and online controllers inside a search that changes its own comparisons, together with a clear statement of where this breaks down.

Q2. Verifying Assumption 1. We do not think it can be verified from the data alone, because it is a property of how evidence is drawn, not of the observed values. What a practitioner can do is enforce it by design and then audit:

  1. Hash the candidate and fix the environments and margins before any decision data are read.
  2. Draw decision indices with randomness the proposer cannot see: fresh samples (Protocol A), with-replacement indexing of a fixed pool (Protocol B), or the without-replacement e-process for a sealed holdout (Protocol C). Use clusters as units when rows are correlated.
  3. Log the routing, the sampling scheme and the controller state. Table 7 lists each record and what it can and cannot show.
  4. Run a falsification check: shift each pool to the boundary null and confirm that the mean stopped e-value is at most 1 and the crossing rate at most 1/h1/h. We run this check on the real LLM pools (Table 19) and for Protocol C (Table 11). A failed check reveals a problem; a passed check does not prove validity.

If decision evidence for a candidate has been seen, that comparison is no longer covered and needs a new index with fresh evidence. Using a pool during proposal is fine under Protocol B: a proposer that ranks candidates on the development decision pool keeps audited FDR ≤ .0003 (Table 18).

(continued below)

AuthorsOpenReview ↗

Response to Reviewer zWPU (2/2)

Q3. Choice of environments, margins and controller. Our recommendation (§4.4, App. C) is to declare every slice on which a regression would be unacceptable, and to set δe\delta_e to the smallest gain that would justify switching, both before the search starts. The guarantee covers exactly the declared environments, and the new sensitivity results show what happens otherwise. On matched LLM streams at h=20h=20 (Table 14):

Declared environmentsHarmful pairs adoptedNull adoption rate (3-env.)
dev only38.8% of 49.0497
dev + OASST24.2% of 26.0164
dev + helpful-base5.9% of 12.0021
all threenone exist.00014

For the margin, raising δ\delta from 0 to .04 reduces the number of true improvements from 291 to 207, while Stagewise power on them varies only between .686 and .727 (Table 15); in the closed loop, audited FDR stays ≤ .0005 over the margins tested (Table 20). For the controller, we fed identical candidate e-values to four procedures (§5.3, T=500T=500). Power is .619 for e-LOND, .890 for e-LORD, .888 for e-SAFFRON and .415 for an e-FWER rule, all with empirical FDR at or below .00055. Our guarantee is proved for e-LOND only, but the comparison shows that much of the power loss comes from the spending rule rather than from the evidence construction.

W2. Practical usefulness and power. You are right that power is lower. In the LLM loop, Stagewise has power .720 against .817 for development-only e-LOND, but the latter makes a false replacement in 13.3% of runs. Power grows with the budget (.516 to .789 as BB goes from 100 to 800) and with α\alpha (.698 to .745 from .05 to .20) while audited FDR stays ≤ .0005, and EGR-ALLOC adds another .02 to .05. The audited rules end with slightly lower held-out utility (.352 vs. .380) but higher worst-environment utility (.284 vs. .270). We also added a short paragraph in §6 on when EGR is not the right tool: if trade-offs between environments are acceptable, or a replacement is cheap to undo, a development gate plus monitoring is probably the better choice.

On novelty, please see Q1 above and the revised Table 1.

We hope these answers and the revision resolve your concerns, and we would be grateful if you would reconsider your score.

AuthorsOpenReview ↗

Follow-up: further revisions since 11 Nov

Dear Reviewer zWPU,

Since our response on 11 Nov, we have made several further revisions, with the new changes highlighted in blue. These additions directly address your concerns regarding the practical usefulness of the statistical guarantees.

W2. Practical usefulness beyond the declared evaluation pools.

Theorem 2 guarantees error control for the declared decision pools. However, practitioners also care about whether the resulting decisions remain reliable on instructions not used during testing.

To investigate this question, we added Appendix E.10 and Table 22, which re-evaluate every adoption in the default LLM closed-loop experiment on instructions that were never accessed by decision sampling.

The empirical FDR on these unread instructions is 0.014 for Stagewise EGR and 0.022 for EGR-ALLOC, compared with 0.267 for the development-mean gate.

On the audit-source scout splits, which were accessed by neither the proposer nor the decision procedure, the corresponding FDR values are 0.009, 0.014, and 0.234.

These findings suggest that the reliability improvements can persist beyond the decision pools in this experiment. Nevertheless, they are empirical diagnostics, not additional theoretical guarantees. Establishing validity for an unseen deployment population would require an appropriate fresh-sampling protocol, such as Protocol A.

Q2. Practical implementation and verification of Assumption 1.

The revised Figure 1 now illustrates the evidence-access procedure more explicitly: the candidate comparison is frozen before decision evidence is read, and the declared environments are evaluated through controlled sampling from their decision pools.

We also strengthened the data provenance audit. All 20,926 available AlpacaEval judgments match the utility bank exactly, and the four missing judgment positions have been identified. Imputing these values at either 0 or 1 leaves the audited empirical FDR unchanged (Appendix E.6, Table 13).

The supplementary README now specifies the instruction-hashing procedure and provides the corresponding verification script.

These checks verify the recorded data and acquisition procedure; they do not independently prove the conditional expectation assumption. The boundary-null falsification diagnostic remains available in Table 20.

Q3. Environment selection, improvement margins, and controller sensitivity.

The previously reported sensitivity results remain unchanged. Following the insertion of Table 13, environment-declaration sensitivity is now reported in Table 15, margin sensitivity in Table 16, and closed-loop reliability–utility–cost sensitivity in Table 21.

We also corrected Appendix D.5 so that the documented sweep configurations agree with the reported experiments.

Power and evaluation cost.

Figure 2 has been redrawn at full text width to improve readability. Panel (c) presents the power–evaluation-cost trade-off as the per-environment evidence budget varies.

We hope these additional analyses make the practical scope, limitations, and usefulness of EGR clearer. If one of our responses concerning the validity assumption, environment selection, or methodological contribution remains insufficient, we would appreciate knowing which issue is most important to resolve.

The authors

Reviewer_KnvwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of model adoption in adaptive search, where an accepted model becomes the baseline for future candidates. The authors propose Evidence-Gated Research (EGR), which combines environment-specific e-processes, intersection–union testing, and online FDR control to make statistically valid adoption decisions. They also introduce several evidence allocation strategies to reduce evaluation costs. Theoretical guarantees are provided under certain assumptions, and experiments on synthetic and controlled benchmarks show improvements in evidence efficiency.

Critical strengths
  1. The paper studies an interesting problem. I agree with the authors that an incorrect model adoption can affect subsequent search steps, which makes statistical reliability particularly important for automated research systems.

  2. The distinction between candidate-level evidence validity, across-candidate error control, and evidence-acquisition cost is useful. The leakage and dependence diagnostics help illustrate why these distinctions matter.

Critical weaknesses
  1. My main concern is the novelty of the proposed framework. Most of the key components, including e-processes, IUT, and e-LOND, are existing techniques. From my understanding, the main contribution is to combine them for model adoption in adaptive search. While I agree that this is a meaningful setting, I am not fully convinced that the paper introduces a sufficiently new methodology. It would be helpful if the authors could explain more clearly what is technically new compared with existing online multiple-testing methods.

  2. The experimental setting seems somewhat limited compared with the motivation. The paper is motivated by AI research agents and automated scientific discovery, but the experiments mainly involve synthetic search and relatively small or controlled model families. I would have liked to see at least one experiment involving a more realistic automated model development pipeline. It is unclear to me whether the proposed method would be equally useful when candidates involve different architectures, training procedures, and substantial computational costs.

  3. I have some concerns about how the efficiency improvement is presented. In Section 5.5, Stagewise EGR achieves the same adoption events as fixed-anytime IUT by construction, while using 56.1% less decision evidence. This is a useful result, but the saving seems to come largely from early stopping and avoiding evaluations of later environments when an earlier test fails. I wonder how much of this improvement should be attributed to a new method, rather than a more efficient implementation of an existing testing procedure.

Questions for authors
  1. Could the authors clarify the main technical novelty of EGR beyond combining existing e-processes, IUT, and e-LOND? In particular, what is the key theoretical difficulty introduced by the moving incumbent that is not already covered by existing online testing frameworks?

  2. Since Stagewise EGR and fixed-anytime IUT have identical adoption events in the matched replay experiment, could the authors clarify which part of the efficiency improvement goes beyond standard sequential testing and early stopping?

  3. Is it possible to evaluate EGR in a more realistic automated research setting, even at a relatively small scale? I think such an experiment would make the paper substantially more convincing.

  4. How should practitioners construct valid evidence units when evaluation data are dependent or repeatedly accessed during model development? Are there practical recommendations beyond restarting the test with fresh evidence?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Knvw (1/2)

We thank the reviewer for a careful review. We revised the paper substantially in response; all changes are marked in blue in the updated PDF.

Q1. Technical novelty. We agree with the premise. E-processes, the IUT minimum and e-LOND are existing tools, and the revised paper now says this explicitly (Table 1 lists what is inherited and what is added; see also §6). The FDR step in Theorem 2 is the e-LOND counting argument, and we no longer suggest otherwise.

What these tools do not give is a valid input when the hypotheses are produced by the search itself. Three things change in the moving-incumbent setting.

First, the null itself is random. Whether challenger tt is a null depends on the incumbent it is compared with, and the incumbent depends on earlier decisions. Appendix A gives a two-step example in which the same second challenger is a true null if the first decision rejects and a false null if it adopts (ν2=1−D1\nu_2 = 1-D_1). e-LOND takes as given e-values that are valid under their nulls; here one has to say what "valid" means when the null indicator is a function of the history. This is why the input condition is written as E[νtEt∣Ft−1]≤νt\mathbb{E}[\nu_t E_t \mid \mathcal F_{t-1}] \le \nu_t.

Second, the environments of one candidate are not tested on separate clocks. Evidence is routed adaptively across environments, each component stops at its own time on a single joint history, and which component is the true null also depends on the history. Lemma 1 and Theorem 1 show that the minimum remains valid under these conditions, without independence across environments.

Third, several things practitioners do routinely break the condition. The one we found least obvious is reading a sealed holdout without replacement in random order: the stopped betting e-value can have mean 1+λ(1−λ)/2>11+\lambda(1-\lambda)/2 > 1 even though the pool satisfies the null (Prop. 4(b)). Prop. 4 and the new Table 2 collect five such failures, each paired with an empirical check. On the LLM data, for example, declaring only the development environment lets through 38.8% of the pairs that hurt an audit environment (Table 14).

The contribution we claim is therefore an interface and its limits: a sufficient condition that can be checked comparison by comparison, proofs that it holds under adaptive routing and stopping, and counterexamples showing where it fails. It is not a new online FDR procedure.

Q2. Where the efficiency comes from. We agree that the original presentation over-credited EGR here. The 56.1% figure has been removed from the abstract and the contributions and now appears only as a separate diagnostic in §5.6. In its place, Table 5 breaks the saving down on 650 frozen LLM pairs replayed on identical evidence streams (h=20h=20, B=400B=400):

RuleEvidence per pairPower
Fixed-anytime IUT1200.0.727
+ stop each environment at crossing857.5.727
+ stop after a failed stage (Stagewise)369.4.727
+ exact curtailment (Prop. 2)330.0.727
EGR-ALLOC395.2.776

The first four rows make exactly the same adoption decisions, so the differences between them are pure cost. Component stopping accounts for 41% of the saving and short-circuiting for 59%; short-circuiting matters most because most comparisons are nulls. Both are standard, and the paper now labels them that way. What is specific to this paper is the curtailment rule of Prop. 2, which is exact for the betting mixture and changes no decision, and EGR-ALLOC, which reuses the saved budget and is the only variant that changes power (+.048, 95% CI .036 to .061). We now present these numbers as the cost of enforcing the all-environment target, not as a new stopping method.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer Knvw (2/2)

Q3. A more realistic setting. This was the main gap in the original submission, and §5.2 is new. We use the per-instance AlpacaEval 2.0 judgments of 26 open-weight LLMs from 11 families, ranging from 1.8B to 141B parameters. The candidates therefore differ in the ways you mention: dense and mixture-of-experts architectures (e.g. Mixtral-8x22B, DBRX), and different post-training (SFT, DPO, KTO). The instruction sources serve as environments: Self-Instruct for development, OASST and Anthropic helpful-base for audit, and Koala and Vicuna held out. In each of 15 rounds, six candidates are scored on development scout data, the best is frozen and tested, and an adoption changes the incumbent for everything that follows.

Over 2,000 runs (Table 4), gating on development evidence produces at least one false replacement in 57% of runs with a mean test and in 13.3% with e-LOND (persistent FDR .264 and .063). EGR has at most one such run out of 2,000 (FDR ≤ .0003), and stays at or below .0010 in all 24 sweeps we ran. Stagewise uses 392 paired evaluations per candidate where fixed IUT uses 1,200, at the same power.

One limitation should be stated plainly. The loop is a replay over existing checkpoints and precomputed judgments, not an agent training new models. We chose this design because it gives the true null status of every comparison, so FDR can be measured exactly; a live training run does not allow this. EGR also reduces only evaluation cost, not the cost of training candidates. When training is expensive, the more important benefit is avoiding a wrong replacement on which every later candidate is then built.

Q4. Dependent or reused evaluation data. Assumption 1 concerns how evidence is sampled given the history, not the dataset itself, so the practical answer depends on how the data are accessed. The revised paper covers four cases (§4.1, App. B.11, App. C).

  • Correlated rows (same document, user or prompt template): use the cluster as the evidence unit. At within-cluster correlation .95, treating rows as independent gives a null crossing rate of .728; cluster units give .0127 (§5.4).
  • A fixed pool accessed repeatedly: index it with replacement (Protocol B). Each draw is then independent of the history, so the pool can even be used to rank proposals. In the LLM loop, a proposer that ranks challengers on the development decision pool still has audited FDR ≤ .0003 (Table 18).
  • A sealed holdout read once: use a without-replacement e-process (Protocol C) instead of the i.i.d. one. On the LLM pools its mean stopped null e-value is 1.000 (Table 11).
  • A restart is needed only when decision evidence was used to choose or modify that particular candidate. On the real pools such reuse raises the null crossing rate 2.8-fold even with about 3.5 candidates per incumbent (Table 19).

Table 7 adds a checklist that separates what logs can document from what they cannot establish.

Finally, you rated soundness 1. If there is a specific step or assumption you believe is wrong, we would very much like to know which one, so that we can address it during the discussion.

We believe the revision addresses the main points of your review, and we would be grateful if you would reconsider your score in light of it.

AuthorsOpenReview ↗

Follow-up: further revisions since 11 Nov

Dear Reviewer Knvw,

Since our response on 11 Nov, we have prepared a further revision, with the new changes highlighted in blue. We summarize the additional evidence below and indicate how it relates to your questions.

Q3. Realistic evaluation beyond the decision pools.

Theorem 2 provides guarantees relative to the declared decision pools. To examine whether the observed reliability extends beyond these pools, we added an out-of-pool persistence diagnostic (Appendix E.10, Table 22).

We re-evaluate every adoption in the default closed-loop experiment on instructions that were never read during decision sampling. On these instructions, the empirical FDR is 0.014 for Stagewise EGR and 0.022 for EGR-ALLOC, compared with 0.267 for the development-mean gate.

On the two audit-source scout splits, which were accessed by neither the proposer nor the decision procedure, the corresponding rates are 0.009, 0.014, and 0.234.

These results provide additional empirical evidence of reliability beyond the decision pools. Importantly, we do not claim that the theoretical guarantee extends to these unseen instructions; such a guarantee would require an appropriate sampling protocol for the target population.

Additional verification of the LLM evaluation data.

We identified the positions of the four missing judgments in the public AlpacaEval annotation files. All remaining 20,926 entries match our utility bank exactly.

Setting the four missing values to either 0 or 1 leaves the empirical FDR of every audited method unchanged and changes power by at most 0.003 (Appendix E.6, Table 13).

The supplementary materials now include the missing-cell mask, verification script, and instruction-hashing procedure, allowing these checks to be reproduced independently.

Q1. Clarifying the moving-incumbent mechanism.

We redesigned Figure 1 as a flowchart explicitly showing both outcomes of each adoption decision.

If the challenger is adopted, the incumbent is replaced and the adoption count increases; otherwise, the incumbent remains unchanged. The next comparison is then constructed using the resulting search state.

This illustrates why later null hypotheses depend on previous adoption decisions and why Theorem 2 uses the null-weighted conditional validity requirement rather than treating all hypotheses as fixed in advance.

Q2 and Q4. Previous clarifications and updated references.

Our previous responses concerning efficiency attribution and evidence validity remain applicable. The new Table 13 shifts several appendix table numbers: environment-declaration sensitivity is now Table 15, the decision-pool proposer experiment is in Table 19, and the evidence-reuse diagnostic is in Table 20.

Remaining soundness concern.

We would particularly appreciate clarification of the specific argument underlying your soundness assessment. If the concern involves the conditional e-value bound, Appendix B.3 and Appendix B.8 provide the detailed arguments for Lemma 1 and Theorem 2.

If any particular assumption or proof step remains unconvincing, we would welcome the opportunity to address it during the remaining discussion period.

The authors

Reviewer_KnvwOpenReview ↗

Followup comment

I thank the authors for their detailed rebuttal. Some of my concerns (Q2 and Q3) have been addressed. However, others (W1 and Q1) remain unresolved. Given that addressing these concerns would require substantial revisions, I would recommend resubmitting the paper after these issues have been adequately addressed.

Comments on the paper

1 comment

Hello Reviewers,

As the discussion period is approaching its end, please take a moment to review the authors’ responses and share any remaining thoughts (either to authors or privately to other reviewers and me) or updates to your assessment.

Thanks for your thoughtful reviews and discussion!

Bests, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.