CONFIRM: Exact Post-Nomination Gating for Test-Time Model Replacement

Ranking adapted models does not determine whether the best-ranked candidate should replace the incumbent: every candidate may be worse, and labels used for adaptive nomination cannot be naively reused for authorization. We formulate limited-label replacement on an enumerated target pool and introduce CONFIRM, a selector-agnostic exact gate. Conditioning on the realized nomination history turns observed nominee-relevant outcomes into a fixed incumbent-relative offset; only a fresh uniform sample without replacement from unresolved disagreements remains random. The resulting paired finite-population p-value is conditionally valid, and Holm correction controls family-wise false replacement across predeclared deployment units. We characterize finite-budget certifiability through the offset, unresolved win, loss, and tie composition, and remaining budget, showing that equal accuracy gains can have different and even budget-reversing certifiability. We evaluate CONFIRM across TTA, CLIP adaptation, refusal-dominated transfer, and quantum error correction. Compared with valid finite-population alternatives, CONFIRM authorizes more beneficial replacements under partial, low-budget confirmation. In the predeclared DomainBed study, CONFIRM makes no harmful replacement across 174,080 repeated gated decisions; ungated harm ranges from 21.9% to 84.1%, including a random-nomination control.