acceptodds
Under review as a conference paper at ICLR 2027

CONFIRM: Exact Post-Nomination Gating for Test-Time Model Replacement

Abstract

Ranking adapted models does not determine whether the best-ranked candidate should replace the incumbent: every candidate may be worse, and labels used for adaptive nomination cannot be naively reused for authorization. We formulate limited-label replacement on an enumerated target pool and introduce CONFIRM, a selector-agnostic exact gate. Conditioning on the realized nomination history turns observed nominee-relevant outcomes into a fixed incumbent-relative offset; only a fresh uniform sample without replacement from unresolved disagreements remains random. The resulting paired finite-population p-value is conditionally valid, and Holm correction controls family-wise false replacement across predeclared deployment units. We characterize finite-budget certifiability through the offset, unresolved win, loss, and tie composition, and remaining budget, showing that equal accuracy gains can have different and even budget-reversing certifiability. We evaluate CONFIRM across TTA, CLIP adaptation, refusal-dominated transfer, and quantum error correction. Compared with valid finite-population alternatives, CONFIRM authorizes more beneficial replacements under partial, low-budget confirmation. In the predeclared DomainBed study, CONFIRM makes no harmful replacement across 174,080 repeated gated decisions; ungated harm ranges from 21.9% to 84.1%, including a random-nomination control.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
test-time adaptation, safe model replacement, post-selection inference, finite-population inference, distribution shift, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EAWZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CONFIRM, a finite-sample-valid framework for test-time model replacement under limited labeling budgets. The key idea is to condition on adaptive nomination history, incorporating previously observed outcomes as a fixed offset, while using fresh samples from unresolved prediction disagreements to construct an exact hypergeometric test. This enables valid reuse of historical evidence and controls false replacements without restricting the nomination algorithm. The paper further introduces a finite-budget certifiability framework, showing that authorization power depends not only on accuracy gains but also on paired-outcome composition, historical evidence, and labeling budgets. Experiments across TTA, CLIP adaptation, CINIC-10, and quantum error correction demonstrate effective false replacement control and improved authorization power over several valid alternatives.

Critical strengths
  1. The statistical framework is sound. The conditional inference construction addresses the challenges of adaptive model selection and label reuse, providing finite-sample guarantees while remaining compatible with arbitrary adaptive nomination rules over fixed predictions.
  2. The certifiability analysis is informative, which reveals nontrivial relationships between accuracy gains, evidence composition, and labeling budgets, including budget-dependent ordering reversals. Experiments across diverse prediction systems further support the framework's generality.
Critical weaknesses
  1. The application is restrcitive, since the guarantees rely on a pre-enumerated finite target pool and frozen model predictions, limiting their direct applicability to realistic online adaptation scenarios involving evolving models or distribution shifts.
  2. The empirical gains are modest: I observe that the improvements over other valid confirmation procedures are concentrated in low-budget settings, with dataset-level confidence intervals often including zero
  3. The methodological implications are limited, e.g., the certifiability analysis remains largely descriptive and does not lead to an optimal acquisition or budget-allocation strategy.
Questions for authors
  1. How could CONFIRM be extended beyond the fixed finite-pool setting to scenarios involving continuously arriving data, distribution shifts, or ongoing model adaptation? Which aspects of the current validity guarantees could be preserved?
  2. Given the modest gains over valid baselines and the limited contribution of historical offsets under the primary setting, what is the main source of CONFIRM's empirical advantage?
  3. Could the certifiability analysis guide a more effective allocation of labeling budgets between nomination and confirmation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer EAWZ : Scope and Mechanism (1/2)

Thank you for recognizing the soundness of the framework and the certifiability insights.

W1/Q1. What guarantees can be preserved beyond a fixed pool?

Answer. The original guarantee can be preserved per buffered episode with frozen predictions; extending it to an evolving stream requires additional theory.

Original evidence. Section 3 and Theorem 1 (Appendix A) define and prove conditional finite pool validity for arbitrary adaptive nomination with fresh uniform sampling. The original QEC rotation illustrates repeated decisions across 112 tasks (Section 5.5, Appendix G). Models and pool composition may differ across units.

New scope clarification. Sections 1, 7 and H.10 now describe buffered episodes: freeze predictions, finalize nomination history, then sample unresolved disagreements uniformly without replacement. Earlier episodes may inform nomination, with relevant revealed labels accounted for in the history and offset. For Holm across a predeclared family, each confirmation draw must be uniform conditional on all nomination histories. Finalizing all histories before fresh randomization independent of them suffices; independence among draws is not needed. Feedback from previous confirmation outcomes into later units requires separate justification.

Anytime valid confidence sequences for sampling without replacement (Waudby-Smith & Ramdas, 2020) could support optional stopping, with a new proof for the offset adjusted count. Sequential alpha spending under suitable conditional validity could address repeated episodes. A future data claim additionally needs a sampling model for the future population and a superpopulation term. We do not claim these extensions as established results. The current guarantee concerns the buffered batch, not arbitrary ongoing drift.

W2/Q2. What drives the empirical advantage if favorable history contributes little?

Answer. CONFIRM's advantage persists at exactly zero nomination offset. Informative count conditioning and retaining nomination evidence play distinct roles.

Original evidence. In Section 4.2, conditioning on the nomination history fixes d0d_0, enabling valid reuse; conditioning on nm=Wm+Lmn_m=W_m+L_m separates the amount of informative evidence from its direction. Theorem 2 explicitly gives the nested hypergeometric decomposition (Section 4.3, Eq. 5); Proposition 4 establishes offset monotonicity. The original PACS ablation demonstrates the value of history: at a 25% nomination share, offset reuse adds 10–50 deployment pp over adversarial history exact (Section 5.1, Figure 4; accuracy gains of 1.20–3.94 pp, Appendix C.1, revised Table 6). In contrast, we agree that favorable history contributes little under the original CLIP 5% setting: capping favorable offsets changes mean accuracy by just 0.013 pp (Table 2).

New zero offset diagnostic. To directly isolate the other mechanism, the nominee is selected with 512 separate development labels per dataset; the evaluation pool has Q=∅Q=\varnothing and d0=0d_0=0. All gates then share the nominee, fresh sample, realized cost and threshold. At b=50:

Gate (8 datasets)Accuracy (%)CONFIRM gain (pp), 95% CI
CONFIRM54.235
Unconditional paired exact53.341+0.894 [0.122, 1.665]
Bonferroni marginal53.107+1.129 [0.323, 1.934]

Source: Section 5.3, Table 3; K.5, Table 46 (240 paired decisions; 50 fresh labels per gate). All three have zero observed harmful/false replacements. This post hoc diagnostic excludes development labels from cost and is not an equal total cost selector comparison. Gains diminish at b=400 (Table 46).

The unconditional gate maximizes a ternary tail without conditioning on the observed informative count; marginal bounds handle wins/losses separately. The zero offset result supports a contribution from conditioning on informative count, even with no favorable history. Consistently, the new original CLIP fraction audit finds that capping positive offsets leaves all CONFIRM decisions unchanged at f=5% and 10%, while the gains vs marginal/unconditional remain +0.40/+0.43 and +0.97/+0.91 pp (H.3, Table 31).

Further evidence on new images. We also tested the locked Shared policy on 1,024 disjoint images from each of the same 8 datasets (K.2, K.5).

Fresh pool, b=200Accuracy (%)Beneficial approvals
CONFIRM56.210176
Unconditional exact54.907143

Source: Table 44 (same history, fresh evidence, 198.108 mean labels; 240 decisions). Gain +1.303 pp [−0.344, 2.950]; both gates have zero observed harm/false. This is a secondary same evidence contrast on new images, not a new domain or future pool guarantee.

Part 2 addresses evidence retention and practical allocation.

AuthorsOpenReview ↗

Response to Reviewer EAWZ: Certifiability and Allocation (2/2)

W2/Q2 (continued): Does retaining nomination history matter separately?

The original Proposition 4 (Section 4.3) and PACS offset ablation (Section 5.1, Figure 4) show that observed favorable evidence can increase power. The small effect at the original 5% share is not a general limitation of evidence reuse.

New experiment. On four new datasets, we compare CONFIRM and adversarial history exact under the same Stage 1 stopping policy, using the same nomination and fresh evidence:

b=400Accuracy (%)
Retain incumbent62.646
History discarded, stopping62.646
CONFIRM, stopping71.313
Full budget may keep71.322

Source: H.4, Table 32 (120 repeated decisions; zero observed harm in these rows). Discarding the observed history under stopping retains every incumbent, whereas CONFIRM reaches nearly full budget accuracy. The new original CLIP share audit further shows that at b=400 the history discarding gate falls from 65.50% to 60.44% as nomination increases from 10% to 40%, while CONFIRM varies by at most 0.86 pp at that budget (H.5). These results complement the zero offset diagnostic: both roles of conditioning are valuable, but in different regimes.

W3/Q3. Can certifiability guide more effective label allocation?

Answer. Yes, through exact power based planning and a simple observable disagreement stopping rule, without changing the original gate. We distinguish this from an optimal acquisition theorem.

Original evidence. Theorem 2 gives the conditional certifiability profile ΠH(m,t)\Pi_{\mathcal H}(m,t) from the offset, unresolved win/loss/tie composition and remaining sample size (Section 4.3, Appendix A). Propositions 2–3 and Figure 3 show that equal gains can have different, even budget reversing, approval probabilities. Corollary 1 bounds Holm authorization between profiles at α/L\alpha/L and α\alpha. Thus the theory already provides a way to quantify how much evidence a hypothesized improvement needs. Section 4.1 also bounds feasible confirmation by the observable number of unresolved disagreements.

New power based planning. Section 4.3 now recommends choosing the smallest feasible m such that

ΠH(m,t)≥1−β,\Pi_{\mathcal H}(m,t)\ge 1-\beta,

for a plausible improving composition and realized offset. For L units, t=α/Lt=\alpha/L gives a conservative Holm target. Historical or simulated compositions can guide the calculation, while the actual gate uses exact fresh evidence. Estimated power need not equal actual power.

New practical allocation rule. Stage 1 continues while b−∣Q∣>∣Dj^∖Q∣b-|Q|>|D_{\hat j}\setminus Q|, stopping when the budget fits or one label remains. This uses only frozen predictions and the current leader, not the unknown outcome composition (Section 5.2, H.1). It is inspired by the feasible evidence limit, rather than an optimizer of the unknown certifiability profile.

SettingFixed 5%StoppingMay keepSource
DomainBed K=16, b=40089.73990.08590.085T30
DomainBed K=32, b=40089.90590.494not reportedH.1
4 new datasets, b=40068.66271.31371.322T32

Accuracy (%); revised H.1, H.4. The DomainBed K=16 Model Selector improvement has a descriptive unit bootstrap 95% interval [0.19, 0.54] pp; all four held out domains improve. The new loss decomposition makes the mechanism of this improvement clear:

4 new datasets, b=400Fixed 5%Stopping
Selection loss (pp)2.680.02
Confirmation loss (pp)00
Mean realized labels216.9356.5

Source: H.6 (post hoc decomposition of the prespecified H.4 experiment). When confirmation is already sufficient, additional nomination can recover the lost utility. At b=50 both losses remain (3.15 and 2.59 pp), so the same allocation is not optimal everywhere. On 8 further datasets a fixed 20% share can outperform stopping (Table 37, K.1).

Validity after stopping. The rule stops Stage 1, not confirmation; the final history is fixed before the fresh uniform draw, so Theorem 1 remains applicable. The new stopping audits on DomainBed, original CLIP and CINIC-10 observe zero harmful/false replacements; the largest complete null familywise false replacement rate is 4.15% (Wilson 95% [3.36, 5.12]; H.1). These observations complement the exact theoretical guarantee.

The original certifiability result is now connected to actionable budget planning and allocation. Joint optimal acquisition remains open because selection changes both the offset and the unknown composition. We thank you for motivating this practical extension of the original theory.

I thank the authors for their detailed responses and substantial additional experiments. While I appreciate these efforts, I have decided to maintain my original rating for the following reasons.

First, the applicability remains limited to fixed pools with frozen predictions. The proposed buffered-episode formulation clarifies the scope but does not fundamentally address the limitations in online adaptation settings.

Second, the new zero-offset experiment provides useful evidence for informative-count conditioning, but also raises questions about how much of CONFIRM's empirical advantage is specifically attributable to its central contribution of adaptive history reuse. Moreover, the gains over valid alternatives remain modest, with key dataset-level confidence intervals still including zero.

Third, the proposed stopping rule improves budget utilization, but does not directly optimize certifiability or resolve the fundamental trade-off between nomination and confirmation.

Overall, I acknowledge the improved empirical evidence and clearer presentation, but believe the core concerns regarding practical generality, incremental methodological novelty, and the connection between certifiability theory and actionable allocation remain insufficiently addressed to warrant a higher rating.

AuthorsOpenReview ↗

Follow-up on the Review Questions and Revised Evidence

Thank you for your further assessment. In response to the original questions, we added separate mechanism studies, allocation experiments and guidance, and a concrete batch use case.

Practical scope. The batch inspection example illustrates a setting where the finite pool is the operational deliverable. CONFIRM checks whether to authorize a candidate’s predictions for that batch, giving its guarantee a direct role within an adaptation workflow.

Contribution of history reuse. Reuse has a small effect in the original CLIP configuration but substantial value in the matched history ablation under stopping at b=400 (Table 32). The zero-offset diagnostic separately identifies the contribution of informative-count conditioning. These results clarify when each mechanism helps, complementing the methodological contribution of exact authorization after adaptive nomination with retained selection evidence on the same pool.

Connection to allocation. The allocation study (H.6) demonstrates improved budget utilization within the same cap while preserving exact authorization. Certifiability supports power planning for specified outcome compositions and observed offsets, while the stopping rule acts on observable confirmation capacity. Together, they provide complementary tools for planning and practical allocation.

We hope the final assessment will consider both the original methodological contribution and the evidence added in response to the review questions.

Reviewer_ejQPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies when a candidate model should replace an existing model, given a fixed batch of initially unlabeled examples from the target environment. Several candidate models and a labeling budget are given. The goal is to select a promising candidate and test whether it is more accurate than the current model before replacing it. CONFIRM splits the labeling budget between candidate selection and a new confirmation phase that statistically confirms or rejects the replacement. Its central contribution is a statistical way to reuse labels collected during selection, combined with new labels sampled from examples where the selected candidate and current model disagree. The authors prove that the probability of approving a model that does not improve accuracy stays below a chosen threshold. Across three experimental settings (test-time adaptation, CLIP adaptation, and quantum error correction), they hide most ground-truth labels, select a candidate using a small label budget, and test whether CONFIRM approves a beneficial replacement or blocks a harmful one.

Critical strengths

Useful guarantee for model replacement. The paper addresses the fact that selecting the best candidate does not establish that it is better than the current model. I think requiring statistical evidence of improvement provides a stronger basis for replacement than simply deploying the candidate that performed best during selection. The authors provide such a check while reusing labels collected during selection. Since these labels have already influenced the choice of candidate, treating them as an independent evaluation can overstate the evidence for improvement. The authors account for this dependence and prove a bound on the probability of approving a model that does not improve accuracy. The guarantee holds across different selection procedures, even when earlier labels influence which examples are labeled next. I consider this a meaningful contribution because it allows a limited labeling budget to support both selection and confirmation with a clear statistical guarantee.

Critical weaknesses

1. Limited methodological advance and a narrow setting. Selecting models and statistically assessing their performance is already studied in prior work [R1, R2], and statistical tests comparing two models on the same examples are well established. The specific contribution here is a valid way to account for labels collected during adaptive selection and test the remaining disagreements on a fixed batch. I consider this useful, but the resulting advance seems relatively narrow for ICLR. Additionally, the guarantee applies only to that batch and requires fixed model predictions. It does not establish that an approved replacement improves future predictions, which seems important for the motivation of adapting to distribution shift. I would also like a more convincing explanation of the practical situations where the guarantee provides a substantial benefit.

2. Limited practical advantage and unclear baselines. The average accuracy gains are very low, eg, in Table 2, CONFIRM improves accuracy over the two closest methods by only 0.3% points, and all three make zero false replacements. The benefit is concentrated at low budgets, but still, at a budget of 50 labels, the gains are only ~1.1% points.

Additionally, CONFIRM often allocates only 5% of the labeling budget to selection. For example, at a labeling budget of 50, only two labels are used for candidate selection. This makes me concerned about selection quality: CONFIRM would reject a poorly selected candidate instead of selecting a good one in the first place. A basic baseline is therefore using the entire labeling budget for selection without confirmation. Here, the whole labeling budget can be used for selection. I found it difficult to determine whether the main-text baselines consistently use this allocation. For example, Table 1 does not clearly specify the labeling budget used by “Selector alone.” I found something explicit in Appendix E, Table 20, where the full-budget selection achieves higher average accuracy than CONFIRM at every evaluated budget, with no observed harmful replacements.

Overall, I am currently unconvinced that the demonstrated practical advantage is substantial enough.

3. Difficult presentation. The paper uses a lot of terminology for concepts that could be explained more directly. For example, “incumbent”, “nominee”, “nomination”, “authorization”, “gate”, and “certification credit/debt.” Even the basic method becomes difficult to follow because the reader needs to keep translating these terms into the "current model", "candidate selection", and "checking whether replacement is justified". I would prefer a simple explanation with a concrete example before introducing the formal terminology.

References

[R1] Laufer-Goldshtein et al. Efficiently Controlling Multiple Risks with Pareto Testing. https://arxiv.org/abs/2210.07913, 2022.

[R2] Rink and Brannath. Post-selection confidence bounds for prediction performance. https://doi.org/10.1007/s10994-024-06632-w, 2025.

Questions for authors
  1. What can we do with CONFIRM that was not possible with existing statistical methods for model selection and evaluation, especially since the practical advantage seems rather low?
  2. Can you intuitively describe a concrete scenario where the studied setting applies and explain why a guarantee restricted to the fixed batch is useful there?
  3. In Table 1, does “Selector alone” use the whole labeling budget for selection? I could not determine whether this is a full-budget baseline or simply the candidate selected using the same small share of the budget as CONFIRM.
  4. Is the full-budget baseline shown for every experiment? Does the confirmation stage have a substantial impact on accuracy?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ejQP: Methodological Contribution, Scope and Presentation (1/2)

Thank you for recognizing the contribution of valid adaptive label reuse. We clarify the original evidence and the new results prompted by your questions.

W1/Q1. What can CONFIRM do beyond existing statistical methods?

Answer. CONFIRM permits an adaptively selected replacement to be tested for strict improvement on the very same finite pool used for selection, while retaining observed selection outcomes and providing exact false replacement control without modeling the selector.

Original evidence. The construction in Sections 4.1–4.2 fixes the observed win minus loss offset d0d_0 and draws a fresh uniform sample of unresolved disagreements. Theorem 1 and Appendix A prove validity after arbitrary adaptive acquisition and nomination over frozen predictions. This is not simply a paired test applied to a selected model: Proposition 1 shows that naively pooling the two stages can reject with probability 8.4% at nominal 5% even when all four candidates tie the current model. The original complete null experiment confirms the distinction (7.20% naive reuse vs 3.85% CONFIRM; Section 5.1, Figure 4). All candidate testing is another valid option, but introduces multiplicity; discarding selection evidence is conservative (Figure 4, Appendix B.2).

The original certifiability analysis also goes beyond ranking: two populations with identical +20 pp improvement have approval probabilities 15.4% vs 2.5% at 10 labels, reversing to 22.4% vs 36.5% at 20 labels (Section 4.3, Theorem 2, Propositions 2–3, Appendix A). Figure 3 illustrates the same phenomenon in frozen CLIP predictions. Accuracy alone therefore cannot specify the evidence required for a replacement decision.

New comparisons and clarification. We expand Section 2 and Appendix B.2 to distinguish CONFIRM from the two references you identified. Pareto Testing [R1] evaluates risk constraints using held out calibration data; Rink–Brannath [R2] constructs bootstrap tilting confidence bounds for selected model performance. Neither is the same design based, same pool paired authorization problem with an observed adaptive offset and a fresh uniform draw. [R2] is already a baseline in the submission: paired MABT is our adaptation of the Rink–Brannath bound (Appendix B.2; Appendix E, revised Table 21). We do not claim that it has the same exact finite pool guarantee or that CONFIRM uniformly outperforms it. New realized cost and common cap comparisons are reported in Part 2 (J, K.4).

The construction enables flexible allocation. Under the new stopping rule on four additional datasets at b=400:

PolicyAccuracy (%)
Exact gate discarding history62.646
CONFIRM71.313
Full budget may keep71.322

Source: revised Table 32 (H.4). The history discarding gate retains every incumbent, whereas CONFIRM retains the observed selection evidence and reaches nearly full budget accuracy. This strengthens the practical value of the original exact construction without changing it.

Q2. Why is a guarantee restricted to a fixed batch useful?

Answer. When the batch predictions themselves are the deliverable, the guarantee concerns the actual decision rather than a proxy for future performance.

Original evidence. Section 3 defines the target as the enumerated pool. The original QEC source rotation audit evaluates four decoders on 112 archived tasks and never replaces a non replaceable incumbent (Section 5.5, Appendix G). An archived shot record is a concrete example of a batch that is decoded once.

New explanation. Section 1 and H.10 now describe batch quality inspection: a team buffers 1,024 production images, freezes the current and 16 adapted models' predictions, and uses 100 expert labels before releasing predictions for that lot. Exact authorization concerns that lot. Online systems can similarly reauthorize frozen checkpoints on buffered pools, subject to fresh sampling. We explicitly do not guarantee future shifted accuracy or evolving predictors (Section 7).

Q3. Does Table 1's selector use the entire budget?

Yes. In the original Table 1, Selector alone uses b labels (b−1 for SHR) and must choose a non incumbent. The original ungated Stage 1 nominee in Table 2 uses only the nomination share; these are distinct baselines (Appendices B.1, D, E). We have clarified both captions and added a stronger full budget rule that may retain the incumbent (H.2, Part 2).

W3. Presentation. We agree that the original notation made the method harder to follow. The revision introduces current model, candidate and replacement check in plain language (Section 1, Figures 1–2), and Section 3 now opens with the worked example moved up from Appendix B.1: 8 adaptive labels give d0=−1d_0=-1; 20 fresh labels (12 candidate wins, 6 current model wins, 2 ties) give p≈0.106>0.05p\approx0.106>0.05, so the current model is retained. The example precedes the formal rule.

AuthorsOpenReview ↗

Response to Reviewer ejQP: Full Budget Baselines and Practical Utility (2/2)

W2/Q4. Is full budget selection more useful, is it shown throughout, and does confirmation limit accuracy?

Answer. Full budget selection can be more accurate, especially with scarce labels. But it does not guarantee safe replacement. The new allocation and matched cost studies show that the accuracy cost of exact authorization can become small in some realistic settings.

Original evidence. We agree with your reading that the original CLIP full budget selector is more accurate at every tested budget. One detail in the cited Appendix E, Table 20 (Table 21 in the revision) is important: at b=100, the ungated full budget selector ("No gate") makes 7 harmful replacements out of 140 decisions, whereas CONFIRM makes none. Thus the original results already quantify a trade between accuracy and safety, rather than harm free full budget selection throughout. The original Table 1 also uses full budget must replace selection (b−1 for SHR); HILTTA-FP at b=400 improves from 88.797% with 880 harms to 89.867% with zero harms under CONFIRM. The original QEC full budget empirical comparator, which can retain the incumbent, makes 251/5,600 harms at b=250 (Section 5.5, Appendix G, revised Table 29). The original CLIP matched gate advantage is +14.7 useful authorization pp at b=50 (66 vs 49 of 116 beneficial nominees, Table 2, Appendix E).

New stronger baselines and allocation. Motivated by your concern about just two selection labels at b=50, we added flexible shares and a stopping rule that spends otherwise unused confirmation labels on selection (Section 5.2, H.1). We also explicitly compare full budget selection that may keep the incumbent if its sampled accuracy is not exceeded (H.2). The following summarizes the new results.

SettingCONFIRM acc.May keep acc.Harm C/MSource
8 datasets, b=50, 5%50.49455.8020/30T37
8 datasets, b=400, stop56.07258.4170/0T37
CLIP, b=100, stop65.31965.4380/7T34
DomainBed K=16, b=400, MS stop90.08590.0850/0T30
DomainBed K=16, b=400, Uniform stop89.96689.9130/187T30
4 new datasets, b=400, stop71.31371.3220/0T32

Accuracy (%); revised Tables 30 (H.1), 32 (H.4), 34 (J), 37 (K.1). Counts are out of 240, 240, 140, 1,280, 1,280, 120 repeated decisions, respectively. Table 34 matches each realized label quota; other rows compare common caps. On the 8 dataset suite, may keep remains more accurate even at b=400, so we do not claim the gap always closes. On 4 new datasets, selection loss falls from 2.68 to 0.02 pp with stopping at b=400, as mean labels rise from 216.9 to 356.5; confirmation loss is already zero (H.6). This directly addresses your concern that nomination quality can limit end to end accuracy. On DomainBed, the stopping rule brings Model Selector from 89.739% to 90.085% (Table 30).

Does this help when candidates are similar? The new same backbone experiment restricts candidates to five prompt classifiers of a single CLIP backbone, rotating the incumbent (H.7; 1,800 decisions/budget).

Model SelectorCONFIRM acc.Full budget acc.Harm C/M
b=5064.9265.660/84
b=400, stopping65.7365.770/14

Even among closely related candidates, exact authorization avoids harmful replacements for a small observed accuracy cost at b=400. This is not merely a benefit from including poor, heterogeneous candidates.

New comparisons against valid gates and paired MABT.

ComparisonGain (pp)95% CISource
4 new, f=10%, vs unconditional+1.134[−1.126, 3.395]T3
8 new, f=10%, vs unconditional+0.379[−0.023, 0.781]T35
Fresh pool, b=200, Shared vs MABT+0.524[−0.810, 1.858]T42

The first two use the same evidence/cost; the third is the prespecified common cap comparison (K.4), with 56.210% vs 55.686% and 198.108 vs 200 mean labels. All eight original CLIP realized cost MABT intervals include zero (J, Table 34). We do not claim superiority over MABT. The original split into 313 partial draws and 247 identical census decisions explains the small average matched gate gain (Section 5.3; Appendix E, revised Table 24).

Coverage. H.2 clarifies ungated comparators across PACS, DomainBed, CLIP, CINIC-10 and QEC (Tables 11, 1, 21, 28, 29). The new may keep comparisons supplement the original QEC endpoints; not every selector/budget uses the same baseline variant. Exact control follows from the sampling design (Theorem 1), not from the absence of observed harms.

The original method offers exact evidence reuse; the new experiments show how it can preserve safety with little accuracy cost in favorable regimes. We thank you for prompting these stronger and more transparent comparisons.

Thank you for the clarifications and substantial new experiments. However, I decided to retain my score aswell. The main reasons for me are that (as described in the original review and also by other reviewers) the applicable scenario, the methodological advance, and the advantages are too narrow. The safety-accuracy trade-off also does not really convince me, as the method protects against some small amount of bad replacements while making many good replacements less accurate.

AuthorsOpenReview ↗

Accuracy, Harm Reduction, and Evidence Efficiency

Thank you for your assessment. The reported results address three aspects of the safety–accuracy trade-off: accuracy cost, harmful replacements, and evidence efficiency under error control.

1. Allocation can reduce the observed accuracy cost. On four additional datasets at a budget cap of 400, the stopping rule raises CONFIRM's accuracy from 68.662% to 71.313%, compared with 71.322% for full budget selection that may retain the incumbent (Table 32). Mean label use increases from 216.9 to 356.5 within that cap (H.6). Thus, adapting allocation can substantially narrow the accuracy gap in this setting.

2. Accuracy and harm reduction can improve together. In the post hoc DomainBed comparison at K=16 and b=400, CONFIRM with stopping has higher observed mean accuracy than full budget selection that may retain the incumbent for LURE, PPAT and Uniform. Observed harmful replacements are zero for CONFIRM versus 68, 116 and 187, respectively, across 1,280 repeated decisions per selector (Table 30). These settings illustrate benefits beyond exchanging lower accuracy for fewer harmful replacements. This ungated full budget comparator does not provide exact false replacement control, whereas CONFIRM's bound holds under the stated sampling conditions.

3. Evidence efficiency matters within error control. On the original CLIP datasets at b=50, CONFIRM approves 66/116 beneficial nominees versus 49/116 for each of the marginal and unconditional exact gates, with identical evidence, label cost and error allocation (Section 5.3, Table 23). This measures how effectively the available evidence supports beneficial authorization under a common error constraint.

The revision also reports settings with larger accuracy costs (Table 37). Together, these comparisons show how accuracy costs and harmful replacement counts vary across settings, and distinguish the cost of requiring control from the efficiency of authorization within that requirement.

Reviewer_QYohOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Picking the best-looking adapted model does not necessarily justify replacing the one already deployed, especially when labels are scarce. This paper asks how to make that decision reliably when some labels have already been used to choose the candidate. CONFIRM preserves those observed outcomes as a fixed contribution, then uses a fresh random sample of the remaining disagreements to test for an accuracy improvement on the fixed target pool. Its key insight is that equally beneficial models can require very different amounts of evidence: how much a model improves and how readily we can establish that improvement are distinct questions.

Critical strengths

1.I find the distinction between model nomination and replacement authorization useful. The examples showing that equal accuracy gains can require different amounts of evidence are particularly insightful.

2.The confirmation procedure is clean and principled. It preserves the known contribution of adaptively queried labels while obtaining valid inference from fresh sampling. Comparisons with other valid gates also make the evaluation more informative than comparisons with ungated selection alone.

Critical weaknesses

1.The practical benefit depends on how the label budget is split between nomination and confirmation. Although sensitivity results are provided, guidance for choosing this split in a new setting remains limited.

2.The empirical advantage over other valid gates is promising but not yet strong across datasets. Gains concentrate at low budgets, and the seven-dataset confidence intervals include zero. The paper appropriately acknowledges this, but it limits the broader conclusions.

Questions for authors

1.How should practitioners choose the nomination budget without knowing candidate quality in advance? When would the default 5% allocation be inadequate?

2.Can the existing results clarify whether CONFIRM retains an advantage when only a small fraction of unresolved disagreements can be labeled? This would help establish where its label-efficiency benefit is most useful.

4 comments
AuthorsOpenReview ↗

Response to Reviewer QYoh: Allocation and Small Confirmation Fractions

Thank you for recognizing the value of nomination/authorization and certifiability. Your questions motivated the following new experiments.

W1/Q1. How should practitioners choose the nomination budget, and when is 5% inadequate?

Answer. Start with 5% as a practical default, extend selection if the remaining labels cannot be used for confirmation, and consider a larger share selected on development data if nomination is difficult. For admissible Stage 1 splits, the split affects power while Theorem 1 remains valid.

Original evidence. Section 4.1 already specifies m=min⁡(b−∣Q∣,∥Dj⋆∖Q∣)m=\min\left(b-|Q|,\|D_{j^\star}\setminus Q|\right): confirmation can use at most the unresolved disagreements, which are observable from frozen predictions. Theorem 1 permits adaptive Stage 1 acquisition; Theorem 2 gives exact certifiability (Section 4.3, Appendix A). On original PACS, at a 25% share, offset reuse adds 10–50 deployment pp over the history discarding gate (Section 5.1, Figure 4). The original Appendix D oracle nominee audit approves 68.6% of 740 beneficial nominees at b=50, 93.9% at b=100, and all at b≥200, with none of 540 nonpositive nominees approved (revised Table 20). Thus nomination can be the bottleneck.

New allocation experiments. We now continue Stage 1 while b−∣Q∣>∣Dj^∖Q∣b-|Q|>|D_{\hat j}\setminus Q|, where j^\hat j is the current leader, stopping when the budget fits or ∣Q∣=b−1|Q|=b-1. This requires no knowledge of true candidate quality; Stage 2 remains uniform conditional on the finalized history (Section 5.2, H.1).

SettingFixed 5%New allocationMay keepSource
DomainBed K=16, b=40089.73990.085 stop90.085T30
4 new datasets, b=40068.66271.313 stop71.322T32
8 datasets, b=5050.49451.560 dev.55.802T37
8 datasets, b=20054.22556.199 (20%)57.137T37

Accuracy (%); revised Tables 30 (H.1), 32 (H.4), 37 (K.1). All CONFIRM policies in these tables have zero observed harmful/false replacements. At b=400 on DomainBed, fixed 5% uses just 77–107 of 400 labels. On 4 new datasets, stopping increases mean label use from 216.9 to 356.5 and cuts selection loss from 2.68 to 0.02 pp, with confirmation loss already zero (H.6). On 8 datasets, stopping is not universally best; 20% or a development selected share can be better (Table 37).

The new CLIP share sweep (H.5) shows at most 1.4 pp accuracy variation for CONFIRM across 2–40% shares; at b=400 the history discarding gate falls from 65.50% to 60.44% (10–40%). Section 4.3 adds power based planning; no universally optimal split is claimed.

W2/Q2. Does the advantage persist at small confirmation fractions?

Answer. Yes, observed gains recur at 5–10% of unresolved disagreements across three dataset suites, but not at every fraction. We distinguish authorization efficiency from accuracy superiority.

Original evidence. At b=50 on the seven original CLIP datasets, CONFIRM approves 66/116 beneficial nominees versus 49/116 for each exact comparator (+14.7 pp) with identical nominee, history, fresh draw, cost and error allocation (Section 5.3, Table 2, Appendix E; revised Table 23). Accuracy gains are +1.090/+1.069 pp over marginal/unconditional exact, with dataset level intervals including zero.

New fraction sweeps and replications. We vary the targeted fraction f of unresolved disagreements, not the fraction of the whole pool. Original pool sweeps are post hoc; f=10% is prespecified in both new dataset suites.

fOriginal 7New 4Further 895% CI, further 8
1%000[0, 0]
2%00+0.825[−1.126, 2.776]
5%+0.43+0.71+0.190[−0.141, 0.521]
10%+0.91+1.13+0.379[−0.023, 0.781]
20%+0.36+0.79+0.540[0.082, 0.999]
50%+0.060+0.731[−0.405, 1.867]
100%not runnot run0[0, 0]

Accuracy gains (pp) vs unconditional exact; revised Table 31 (H.3–H.4), Tables 3, 36 (K.1). At 1–2% no original CLIP gate approves. At the prespecified 10% on 8 datasets:

GateAcc. (%)Beneficial approvalsLabelsHarm/false
CONFIRM51.48034540.30/0
Unconditional51.10153540.30/0
Marginal51.01733340.30/0

Source: Table 35 (K.1), 240 repeated decisions per gate; four dataset means improve and four tie. The primary interval includes zero; the positive secondary 20% interval does not replace it. The original 560 decisions already show that all 247 censuses coincide, while 313 partial draws yield +6.45/+8.96 useful authorization pp over marginal/unconditional (Section 5.3; Appendix E, revised Table 24).

The new allocation rule and replications clarify where CONFIRM is useful, not universally superior.

AuthorsOpenReview ↗

Request for Feedback on the Revised Evidence

Dear Reviewer QYoh,

As the discussion period draws to a close, we would like to briefly highlight the additions responding to your two questions:

  • Budget allocation: We added power based planning guidance and an adaptive Stage 1 stopping rule, with experiments showing when changing the allocation improves budget utilization (Section 4.3; H.1, H.4).
  • Small confirmation fractions: We added fraction sweeps across three dataset suites, including prespecified 10% comparisons on four and eight additional datasets (Tables 31, 35, 36). We would appreciate your feedback on whether these additions address your concerns and affect your overall assessment of the paper.

Thank you again for your time and constructive feedback.

Best regards, The authors

Thank you for the detailed response.

Overall, I think the authors have addressed my main concerns, and the revised paper is much stronger. I am willing to increase my rating from 2 to 3.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Reassessment

Dear Reviewer QYoh,

Thank you very much for your thoughtful reassessment and for recognizing the substantial improvements in our revised manuscript.

Your constructive feedback helped us strengthen both the empirical evidence and the practical relevance of our central contribution: statistically valid model replacement after adaptive selection while preserving previously observed evidence.

We sincerely appreciate your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Original Contributions and New Evidence

We thank all reviewers for recognizing the value of separating model selection from replacement authorization, the soundness of conditional inference, and the insights of finite budget certifiability. Their questions have helped us substantially strengthen the paper. The original CONFIRM gate and its exact finite pool guarantee remain unchanged.

1. What the original submission establishes. CONFIRM answers a question distinct from ranking: can we authorize an adaptively selected replacement on the same finite pool, reusing the observed selection outcomes without invalid inference? Theorem 1 establishes this guarantee for arbitrary adaptive nomination over frozen predictions (Sections 3–4, Appendix A). Proposition 1 shows that naive pooling can reject at 8.4% under a complete null at nominal 5%; the original null audit reports 7.20% for naive reuse versus 3.85% for CONFIRM (Section 5.1, Figure 4). The original CLIP audit already demonstrates a matched evidence benefit: at b=50, CONFIRM authorizes 66/116 beneficial nominees versus 49/116 for each exact comparator (+14.7 percentage points, pp; Section 5.3, Table 2, Appendix E). In DomainBed, zero harmful replacements are observed across 174,080 repeated gated decisions (Section 5.2, Table 1, Appendix D). These are repeated decisions on fixed pools, not independent deployments.

2. What the revision adds. We directly address the reviewers' questions through additional experiments, summarized here. Locations refer to the revised manuscript.

New experimentCONFIRMComparator / resultLocation
4 additional datasets, f=10%+1.134 ppvs unconditional exactTable 3; H.3–H.4
8 further datasets, f=10%45 beneficial approvals35 unconditional / 33 marginalTables 3, 35
Zero offset, b=5054.235%53.341% unconditional / 53.107% marginalTable 46
DomainBed, b=400, stopping90.085%89.739% fixed 5%; 90.085% may keepTable 30
4 new datasets, b=400, stopping71.313%68.662% fixed 5%; 71.322% may keepTable 32
CLIP, b=100, equal realized cost65.319%, 0 harmsMay keep: 65.438%, 7 harmsTable 34

Mechanism. With no current pool selection labels (d0=0d_0=0), CONFIRM still gains +0.894 [0.122, 1.665] and +1.129 [0.323, 1.934] pp over the two exact gates at b=50 (post hoc diagnostic, Table 46). This supports a contribution from informative count conditioning that does not require favorable selection evidence. This uses 512 separate development labels per dataset; it is not an equal total cost selector comparison.

Allocation. The new Stage 1 stopping rule redirects otherwise unusable confirmation budget to selection, preserving Theorem 1 (Section 5.2, H.1). On the four new datasets at b=400, selection loss falls from 2.68 to 0.02 pp and mean label use rises from 216.9 to 356.5 within the same cap; confirmation loss is already zero (H.6). Larger or development selected shares also help in some settings (Table 37).

Utility and limits. The prespecified 8 dataset f=10% contrast is +0.379 pp [−0.023, 0.781] versus unconditional exact. The 4 dataset interval also includes zero [−1.126, 3.395]. These replications support authorization efficiency rather than established accuracy superiority. At b=50 on 8 datasets, may keep has higher accuracy (55.802% vs 50.494%) but 30 vs 0 harms; at b=400 it remains more accurate with no observed harms (Table 37). Exact control, rather than zero empirical harm in every future run, is our guarantee. The new matched cost comparisons against paired MABT do not establish superiority (J, K.4; Tables 34, 40, 42).

3. Presentation and scope. We add a concrete batch use case (Section 1), move the existing worked example from Appendix B.1 to the opening of Section 3, and provide power based planning (Section 4.3) and clearer baseline definitions and limitations (Tables 1–2, Section 7). The revised appendices document the new protocols and results, distinguishing post hoc analyses from prespecified tests. CONFIRM applies to buffered pools with frozen predictions and fresh uniform confirmation, not arbitrary future drift or indefinite streams.

We believe these revisions directly address the concerns about mechanism, allocation, utility and presentation while retaining the original methodological contribution. Thank you for helping us make the evidence and its implications clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.