- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Evidence-Gated Research (EGR), a framework as a gate to control model replacement in adaptive search. EGR attempts to ensure that improvements by each model iteration are consistent across multiple environments, and replacements are not misled by repeated testing. To achieve this objective, EGR proposes multiple modules in the control process, such as candidate freezing (before accessing decision evidence) and adaptive testing strategy (to avoid false positives), and further proposes several efficiency improvements, such as ALLOC and Balance, to reduce the cost of testing. Experiments on multiple tasks demonstrate the effectiveness and efficiency of EGR.
Critical strengths
- Clear problem formulation: the paper targets and mitigates the important problems of cross-environment generalization and false positives in adaptive search.
- Strong theoretical proof: the method offers clear proof and is mathematically grounded.
Critical weaknesses
- Unclear methodological novelty: despite the theoretical support, most modules in the method are adopted from existing methods. The contribution of the method is more in the careful combination of these modules.
- Lack of experiments in real-world complex scenarios: the experiments use synthetic searches, digits, and controlled MLP banks, but do not involve real-world complex tasks or models (e.g., LLMs on more challenging language tasks).
Questions for authors
- Can the authors demonstrate the effectiveness of the method under more realistic task settings, such as LLMs for language tasks?
3 comments
The experiments have addressed my related concerns. I maintain my original rating (3).
Thank you, and two further checks on the LLM study
Dear Reviewer BxVd,
Thank you for considering our earlier response and the newly added LLM experiments. We have since performed two further checks that strengthen the empirical evaluation, with the corresponding revisions highlighted in blue.
1. Reliability on previously unread instructions.
The empirical FDR reported in Table 4 is evaluated relative to the declared decision pools. We have now added an out-of-pool persistence diagnostic (Appendix E.10, Table 22), which re-evaluates every adoption using instructions that were never read during decision sampling.
On these instructions, Stagewise EGR and EGR-ALLOC achieve empirical FDR values of 0.014 and 0.022, respectively, compared with 0.267 for the development-mean gate.
These results provide additional evidence that the reliability improvements are not confined to the particular decision samples, although they do not establish a theoretical guarantee for unseen deployment populations.
2. Verification of the LLM benchmark data.
We compared the 26-model utility bank against the public AlpacaEval annotation files. All 20,926 available judgments match exactly, and we identified the four missing judgment positions.
Assigning either 0 or 1 to these missing values leaves the audited empirical FDR unchanged (Appendix E.6, Table 13).
We also redesigned Figure 1 to make the adoption loop and incumbent updates more explicit, and redrew Figure 2 to improve readability at print size.
We hope these additional checks further strengthen the empirical evidence and address the remaining presentation concerns. We would be happy to clarify any outstanding questions.
The authors
Response to Reviewer BxVd
We thank the reviewer for the positive evaluation. We took the request for a realistic LLM experiment seriously, and the revised paper now includes one (§5.2, App. D.5 and E.9; changes in blue).
Setup. We use the per-instance AlpacaEval 2.0 judgments of 26 open-weight chat models from 11 families (including Qwen, Llama 2/3/3.1, Mistral and Mixtral, Yi, DBRX, Tülu, Vicuna, Gemma and Phi), ranging from 1.8B to 141B parameters. The 805 instructions come from five sources, which we use as environments: Self-Instruct for development, OASST and Anthropic helpful-base for audit, and Koala and Vicuna held out. In each of 15 rounds, six candidate models are scored on a small development scout set, the best is frozen and tested against the current model, and if adopted it becomes the model that all later candidates are compared with.
Results over 2,000 runs (Table 4).
Gating on development data alone leads to a false replacement in more than half of the runs with a simple mean test, and in 13% with e-LOND. EGR makes at most one false replacement in 2,000 runs, and uses about a third of the evidence of fixed IUT at the same power. Over 24 additional settings (search pressure, α, budget, margin, five data splits, binary utility, spending schedules, proposer type and starting model), EGR's empirical FDR never exceeds .0010, while the development-only mean gate ranges from .165 to .390. Removing any one model family (Qwen, Llama or Mixtral) leaves Stagewise power between .726 and .780 (Table 12).
One limitation, which we state in the paper: the loop is replayed on existing models and precomputed judgments rather than run by an agent that trains new ones. This is what lets us know the true answer for every comparison and measure FDR exactly.
On novelty, we agree that the building blocks exist. The revised Table 1 separates what is inherited from what is new: the treatment of nulls that depend on earlier adoptions, validity under adaptive routing and asynchronous stopping across environments (Theorem 1, Lemma 1), and counterexamples showing which common shortcuts break the guarantee, such as reading a sealed holdout without replacement (Prop. 4).
Thank you again for your support. We are happy to answer any further questions, and we hope the new results strengthen your confidence in the paper.