- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CONFIRM, a finite-sample-valid framework for test-time model replacement under limited labeling budgets. The key idea is to condition on adaptive nomination history, incorporating previously observed outcomes as a fixed offset, while using fresh samples from unresolved prediction disagreements to construct an exact hypergeometric test. This enables valid reuse of historical evidence and controls false replacements without restricting the nomination algorithm. The paper further introduces a finite-budget certifiability framework, showing that authorization power depends not only on accuracy gains but also on paired-outcome composition, historical evidence, and labeling budgets. Experiments across TTA, CLIP adaptation, CINIC-10, and quantum error correction demonstrate effective false replacement control and improved authorization power over several valid alternatives.
Critical strengths
- The statistical framework is sound. The conditional inference construction addresses the challenges of adaptive model selection and label reuse, providing finite-sample guarantees while remaining compatible with arbitrary adaptive nomination rules over fixed predictions.
- The certifiability analysis is informative, which reveals nontrivial relationships between accuracy gains, evidence composition, and labeling budgets, including budget-dependent ordering reversals. Experiments across diverse prediction systems further support the framework's generality.
Critical weaknesses
- The application is restrcitive, since the guarantees rely on a pre-enumerated finite target pool and frozen model predictions, limiting their direct applicability to realistic online adaptation scenarios involving evolving models or distribution shifts.
- The empirical gains are modest: I observe that the improvements over other valid confirmation procedures are concentrated in low-budget settings, with dataset-level confidence intervals often including zero
- The methodological implications are limited, e.g., the certifiability analysis remains largely descriptive and does not lead to an optimal acquisition or budget-allocation strategy.
Questions for authors
- How could CONFIRM be extended beyond the fixed finite-pool setting to scenarios involving continuously arriving data, distribution shifts, or ongoing model adaptation? Which aspects of the current validity guarantees could be preserved?
- Given the modest gains over valid baselines and the limited contribution of historical offsets under the primary setting, what is the main source of CONFIRM's empirical advantage?
- Could the certifiability analysis guide a more effective allocation of labeling budgets between nomination and confirmation?
4 comments
Response to Reviewer EAWZ: Certifiability and Allocation (2/2)
W2/Q2 (continued): Does retaining nomination history matter separately?
The original Proposition 4 (Section 4.3) and PACS offset ablation (Section 5.1, Figure 4) show that observed favorable evidence can increase power. The small effect at the original 5% share is not a general limitation of evidence reuse.
New experiment. On four new datasets, we compare CONFIRM and adversarial history exact under the same Stage 1 stopping policy, using the same nomination and fresh evidence:
| b=400 | Accuracy (%) |
|---|---|
| Retain incumbent | 62.646 |
| History discarded, stopping | 62.646 |
| CONFIRM, stopping | 71.313 |
| Full budget may keep | 71.322 |
Source: H.4, Table 32 (120 repeated decisions; zero observed harm in these rows). Discarding the observed history under stopping retains every incumbent, whereas CONFIRM reaches nearly full budget accuracy. The new original CLIP share audit further shows that at b=400 the history discarding gate falls from 65.50% to 60.44% as nomination increases from 10% to 40%, while CONFIRM varies by at most 0.86 pp at that budget (H.5). These results complement the zero offset diagnostic: both roles of conditioning are valuable, but in different regimes.
W3/Q3. Can certifiability guide more effective label allocation?
Answer. Yes, through exact power based planning and a simple observable disagreement stopping rule, without changing the original gate. We distinguish this from an optimal acquisition theorem.
Original evidence. Theorem 2 gives the conditional certifiability profile from the offset, unresolved win/loss/tie composition and remaining sample size (Section 4.3, Appendix A). Propositions 2–3 and Figure 3 show that equal gains can have different, even budget reversing, approval probabilities. Corollary 1 bounds Holm authorization between profiles at and . Thus the theory already provides a way to quantify how much evidence a hypothesized improvement needs. Section 4.1 also bounds feasible confirmation by the observable number of unresolved disagreements.
New power based planning. Section 4.3 now recommends choosing the smallest feasible m such that
for a plausible improving composition and realized offset. For L units, gives a conservative Holm target. Historical or simulated compositions can guide the calculation, while the actual gate uses exact fresh evidence. Estimated power need not equal actual power.
New practical allocation rule. Stage 1 continues while , stopping when the budget fits or one label remains. This uses only frozen predictions and the current leader, not the unknown outcome composition (Section 5.2, H.1). It is inspired by the feasible evidence limit, rather than an optimizer of the unknown certifiability profile.
| Setting | Fixed 5% | Stopping | May keep | Source |
|---|---|---|---|---|
| DomainBed K=16, b=400 | 89.739 | 90.085 | 90.085 | T30 |
| DomainBed K=32, b=400 | 89.905 | 90.494 | not reported | H.1 |
| 4 new datasets, b=400 | 68.662 | 71.313 | 71.322 | T32 |
Accuracy (%); revised H.1, H.4. The DomainBed K=16 Model Selector improvement has a descriptive unit bootstrap 95% interval [0.19, 0.54] pp; all four held out domains improve. The new loss decomposition makes the mechanism of this improvement clear:
| 4 new datasets, b=400 | Fixed 5% | Stopping |
|---|---|---|
| Selection loss (pp) | 2.68 | 0.02 |
| Confirmation loss (pp) | 0 | 0 |
| Mean realized labels | 216.9 | 356.5 |
Source: H.6 (post hoc decomposition of the prespecified H.4 experiment). When confirmation is already sufficient, additional nomination can recover the lost utility. At b=50 both losses remain (3.15 and 2.59 pp), so the same allocation is not optimal everywhere. On 8 further datasets a fixed 20% share can outperform stopping (Table 37, K.1).
Validity after stopping. The rule stops Stage 1, not confirmation; the final history is fixed before the fresh uniform draw, so Theorem 1 remains applicable. The new stopping audits on DomainBed, original CLIP and CINIC-10 observe zero harmful/false replacements; the largest complete null familywise false replacement rate is 4.15% (Wilson 95% [3.36, 5.12]; H.1). These observations complement the exact theoretical guarantee.
The original certifiability result is now connected to actionable budget planning and allocation. Joint optimal acquisition remains open because selection changes both the offset and the unknown composition. We thank you for motivating this practical extension of the original theory.
I thank the authors for their detailed responses and substantial additional experiments. While I appreciate these efforts, I have decided to maintain my original rating for the following reasons.
First, the applicability remains limited to fixed pools with frozen predictions. The proposed buffered-episode formulation clarifies the scope but does not fundamentally address the limitations in online adaptation settings.
Second, the new zero-offset experiment provides useful evidence for informative-count conditioning, but also raises questions about how much of CONFIRM's empirical advantage is specifically attributable to its central contribution of adaptive history reuse. Moreover, the gains over valid alternatives remain modest, with key dataset-level confidence intervals still including zero.
Third, the proposed stopping rule improves budget utilization, but does not directly optimize certifiability or resolve the fundamental trade-off between nomination and confirmation.
Overall, I acknowledge the improved empirical evidence and clearer presentation, but believe the core concerns regarding practical generality, incremental methodological novelty, and the connection between certifiability theory and actionable allocation remain insufficiently addressed to warrant a higher rating.
Follow-up on the Review Questions and Revised Evidence
Thank you for your further assessment. In response to the original questions, we added separate mechanism studies, allocation experiments and guidance, and a concrete batch use case.
Practical scope. The batch inspection example illustrates a setting where the finite pool is the operational deliverable. CONFIRM checks whether to authorize a candidate’s predictions for that batch, giving its guarantee a direct role within an adaptation workflow.
Contribution of history reuse. Reuse has a small effect in the original CLIP configuration but substantial value in the matched history ablation under stopping at b=400 (Table 32). The zero-offset diagnostic separately identifies the contribution of informative-count conditioning. These results clarify when each mechanism helps, complementing the methodological contribution of exact authorization after adaptive nomination with retained selection evidence on the same pool.
Connection to allocation. The allocation study (H.6) demonstrates improved budget utilization within the same cap while preserving exact authorization. Certifiability supports power planning for specified outcome compositions and observed offsets, while the stopping rule acts on observable confirmation capacity. Together, they provide complementary tools for planning and practical allocation.
We hope the final assessment will consider both the original methodological contribution and the evidence added in response to the review questions.
Response to Reviewer EAWZ : Scope and Mechanism (1/2)
Thank you for recognizing the soundness of the framework and the certifiability insights.
W1/Q1. What guarantees can be preserved beyond a fixed pool?
Answer. The original guarantee can be preserved per buffered episode with frozen predictions; extending it to an evolving stream requires additional theory.
Original evidence. Section 3 and Theorem 1 (Appendix A) define and prove conditional finite pool validity for arbitrary adaptive nomination with fresh uniform sampling. The original QEC rotation illustrates repeated decisions across 112 tasks (Section 5.5, Appendix G). Models and pool composition may differ across units.
New scope clarification. Sections 1, 7 and H.10 now describe buffered episodes: freeze predictions, finalize nomination history, then sample unresolved disagreements uniformly without replacement. Earlier episodes may inform nomination, with relevant revealed labels accounted for in the history and offset. For Holm across a predeclared family, each confirmation draw must be uniform conditional on all nomination histories. Finalizing all histories before fresh randomization independent of them suffices; independence among draws is not needed. Feedback from previous confirmation outcomes into later units requires separate justification.
Anytime valid confidence sequences for sampling without replacement (Waudby-Smith & Ramdas, 2020) could support optional stopping, with a new proof for the offset adjusted count. Sequential alpha spending under suitable conditional validity could address repeated episodes. A future data claim additionally needs a sampling model for the future population and a superpopulation term. We do not claim these extensions as established results. The current guarantee concerns the buffered batch, not arbitrary ongoing drift.
W2/Q2. What drives the empirical advantage if favorable history contributes little?
Answer. CONFIRM's advantage persists at exactly zero nomination offset. Informative count conditioning and retaining nomination evidence play distinct roles.
Original evidence. In Section 4.2, conditioning on the nomination history fixes d0, enabling valid reuse; conditioning on nm=Wm+Lm separates the amount of informative evidence from its direction. Theorem 2 explicitly gives the nested hypergeometric decomposition (Section 4.3, Eq. 5); Proposition 4 establishes offset monotonicity. The original PACS ablation demonstrates the value of history: at a 25% nomination share, offset reuse adds 10–50 deployment pp over adversarial history exact (Section 5.1, Figure 4; accuracy gains of 1.20–3.94 pp, Appendix C.1, revised Table 6). In contrast, we agree that favorable history contributes little under the original CLIP 5% setting: capping favorable offsets changes mean accuracy by just 0.013 pp (Table 2).
New zero offset diagnostic. To directly isolate the other mechanism, the nominee is selected with 512 separate development labels per dataset; the evaluation pool has Q=∅ and d0=0. All gates then share the nominee, fresh sample, realized cost and threshold. At b=50:
Source: Section 5.3, Table 3; K.5, Table 46 (240 paired decisions; 50 fresh labels per gate). All three have zero observed harmful/false replacements. This post hoc diagnostic excludes development labels from cost and is not an equal total cost selector comparison. Gains diminish at b=400 (Table 46).
The unconditional gate maximizes a ternary tail without conditioning on the observed informative count; marginal bounds handle wins/losses separately. The zero offset result supports a contribution from conditioning on informative count, even with no favorable history. Consistently, the new original CLIP fraction audit finds that capping positive offsets leaves all CONFIRM decisions unchanged at f=5% and 10%, while the gains vs marginal/unconditional remain +0.40/+0.43 and +0.97/+0.91 pp (H.3, Table 31).
Further evidence on new images. We also tested the locked Shared policy on 1,024 disjoint images from each of the same 8 datasets (K.2, K.5).
Source: Table 44 (same history, fresh evidence, 198.108 mean labels; 240 decisions). Gain +1.303 pp [−0.344, 2.950]; both gates have zero observed harm/false. This is a secondary same evidence contrast on new images, not a new domain or future pool guarantee.
Part 2 addresses evidence retention and practical allocation.