- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Certified Rate Matching (C-RM), a framework for deciding whether to replace an existing classification model with a new one under distribution shift, when target labels are unavailable. The method first matches the number of positive predictions made by the existing model, allowing the candidate model to select different instances without increasing the review workload. It then uses historical labeled data and current unlabeled data to test whether the replacement is likely to improve F-score.
The main contributions include a decomposition of F-score regret into selection-count and ranking errors, a finite-sample statistical guarantee controlling erroneous model replacements, and a more efficient certification procedure based on disagreements between the two models.
Experiments on text and image classification benchmarks demonstrate that the method can identify beneficial model upgrades while avoiding harmful replacements. The proposed disagreement-based test also certifies substantially more upgrades than a simpler alternative.
Critical strengths
-
Practically relevant and well-defined problem setting. The paper addresses an important deployment challenge: deciding whether to adopt a new model when current labels are unavailable and operational capacity must remain fixed. The distinction between global predictive performance and performance at a particular decision threshold is meaningful and well motivated.
-
Statistical guarantees supported by targeted experiments. The finite-sample certification procedure provides explicit control over erroneous approvals under its stated assumptions. The disagreement-based construction also improves statistical power, with experiments showing 4.0–4.6 times as many certified upgrades on COCO under the same nominal error budget. This provides a convincing connection between the theoretical development and practical benefits.
Critical weaknesses
-
Restrictive assumptions underlying the certification guarantee. The main guarantee requires positive-class detection rates to remain stable between source and target distributions. This assumption may be difficult to justify or verify without target labels, especially under the distribution shifts motivating the paper. Although the authors discuss bounded drift, the practical reliability of the certificate when this assumption is violated remains unclear. This limits its applicability to realistic deployment scenarios.
-
Limited demonstrated utility and unclear advantages over simpler alternatives. The certification procedure is quite conservative, approving only 834 out of 26,250 candidate upgrades in the GoEmotions audit. While this avoids harmful replacements in the reported experiments, it also misses many potentially beneficial upgrades. Moreover, the core approach combines established ideas from rate matching, paired statistical testing, and conservative policy improvement. A clearer comparison with existing risk-control and model-selection approaches would help establish the novelty and practical advantages of the proposed framework.
Questions for authors
See weakness.
4 comments
Response 2
Response 2:
We expanded the evaluation to quantify avoided losses, missed gains, and comparisons with alternative statistical rules, while retaining the original models, thresholds, and paired evidence.
In the original GoEmotions TF-IDF-to-Qwen3 audit, unconditional replacement lowers empirical target-pool F1 in 3,578 of 26,250 comparisons (13.63%). These harmful comparisons have a mean decline of 1.59 F1 percentage points; 1,118 decline by at least 2 points, 149 by at least 5 points, and the largest decline is 7.28 points. The original certificate rejects all these harmful proposals while approving 834 beneficial replacements, whose mean F1 gain is 9.23 points. To isolate selection from the number of approvals, uniformly choosing exactly 834 comparisons would produce an expected 113.68 harmful replacements and a mean deployed gain of 0.141 points; the original gate achieves zero observed harmful replacements and a gain of 0.293 points. The random-control values are exact expectations over subsets of the fixed archive.
Using the same evidence and delta=0.05, the paper's factorized rule increases approvals to 1,634. A four-bound ratio CP refinement developed for this rebuttal further increases approvals to 2,072, retaining all 834 original approvals and adding 1,238 beneficial replacements. Mean deployed F1 rises from 0.53109 to 0.53462, compared with the reference's 0.52816. This increases the deployed gain by 120.52%, with no observed empirical-pool or evaluation-batch F1 losses under either the original or new rule. Unconditional replacement attains a higher mean F1 of 0.57264 while accepting the declines quantified above.
We evaluated 12 statistical rules at the same nominal delta, including exact paired-proportion tests, joint multinomial regions, concentration-bound alternatives, and score/Wald approximations. Principal comparisons are:
| Rule | GoEmotions: 26,250 comparisons | COCO archive: 63,000 comparisons |
|---|---|---|
| Original direct CP | 834 (0 / 0) | 352 (0 / 0) |
| Paper's factorized CP (replay) | 1,634 (0 / 0) | 1,334 (0 / 6) |
| New four-bound ratio CP | 2,072 (0 / 0) | 2,253 (0 / 9) |
| Exact McNemar-IUT | 178 (0 / 2) | 2,003 (1 / 55) |
| Joint multinomial mixture region, coupled ratio | 0 (0 / 0) | 0 (0 / 0) |
| All-beta delta-Wald, asymptotic | 6,727 (151 / 174) | 19,926 (6 / 390) |
Entries are approvals (empirical-pool / evaluation-batch F1 loss counts). COCO uses the archived 100-target-seed experiment, distinct from the main-text 31,500-comparison experiment. McNemar-IUT uses the full delta for each directional test. The joint baseline preserves dependence within fixed four-cell Dirichlet(1/2,1/2,1/2,1/2) mixture confidence regions, assigning delta/2 to each domain. Our criterion can certify upgrades where increased capture compensates for increased workload, whereas this directional McNemar-IUT requires workload reduction. Wald yields a larger GoEmotions mean gain of 1.910 points, together with the losses shown above.
We also quantify missed utility. Of 22,516 F1-improving proposals, the original rule misses 21,682 and the new rule misses 20,444; positive-gain retention increases from 6.29% to 13.87%. For the original rule, 1,060 rejected proposals improve every finite-beta objective but fail the strict sufficient condition, accounting for 0.73% of available positive F1 gain. In contrast, 19,531–19,772 rejected proposals satisfy the noiseless target-pool sufficient condition and account for 91.80%–92.63% of available positive gain. This latter gap includes confidence-bound conservatism and possible source–target discrepancies. A separate fixed-law simulation isolates sample size: increasing source-positive/target-unlabeled counts from 250/1,000 to 1,000/5,000 to 5,000/20,000 raises new-rule approval from 4.82% to 65.48% to 100% over 10,000 repetitions per setting.
The methodological contribution is the comparison available under this supervision pattern. Historical positive labels estimate paired capture differences, while current unlabeled traffic estimates paired selection rates. Their sufficient margin certifies relative population F_beta improvement simultaneously for every beta>0 without estimating target prevalence, under the stated transport assumptions. An independent matching window constructs the replacement at the reference workload, and certification accounts for subsequent workload mismatch. Learn then Test provides a general framework for valid tests; C-RM supplies a test for this particular partially supervised comparison. Conformal Risk Control, including its shift extensions, controls an expected-loss target, while HCPI/SPIBB use logged rewards or transitions to study policy returns. C-RM's contribution is the decision-transfer formulation, paired certificate, disagreement factorization with improved empirical power, and identifiability boundary for deploying a fixed classifier replacement before target labels arrive.
Thank you for the efforts, I will raise my score by 1.
Thank you for revisiting our submission and considering the additional analyses. We appreciate your updated assessment and the time you have devoted to the discussion. Your constructive feedback helped us clarify the presentation and strengthen the empirical evaluation.
Response 1
Response 1: Thank the reviewer for these valuable suggestions sincerely.Our additional analysis shows that individual capture-rate invariance is stronger than the original certificate needs. We also added controlled experiments with substantial changes in both policies' capture rates and audited the original CivilComments models under natural shift.
Let gjD denote policy j's positive-class capture rate in domain D, with S the candidate, R the reference, P the source, and Q the target. For positive source and target reference capture rates and positive target prevalence, the unchanged certificate retains its per-comparison error guarantee for simultaneous population Fβ improvement for every β>0 under the weaker condition
gRQgSQ≥gRPgSP. On the original confidence event, approval implies that the source capture ratio exceeds both 1 and the target workload ratio. The condition above transfers this inequality to the target capture ratio, making both terms in the cross-multiplied F_beta difference positive. The approval rule and confidence level remain unchanged. In particular, both capture rates may decrease by a common multiplicative factor while this guarantee remains applicable.
We tested this case using the original direct-CP implementation at delta=0.05. Candidate and reference capture rates both decrease by 20% relative to their source values: 0.70 to 0.56 and 0.55 to 0.44, respectively. This violates individual stability while preserving their ratio. The new fixed-design study covers four common multipliers, two shift mechanisms, and two sample sizes, with 10,000 repetitions per setting. For the 20% reduction, original-rule approval rates are:
The hidden-shift mechanism preserves the entire unlabeled joint decision distribution. The respective population F1 gains are 11.34 and 10.17 percentage points. These experiments measure certification power under the weaker transport condition, using the original rule.
The original-model CivilComments audit provides complementary natural-shift evidence. Across three Qwen3 training seeds, candidate capture-rate changes are -0.39, -0.37, and -0.79 percentage points, compared with -2.02 points for the TF-IDF reference. The candidate's average capture advantage increases from 6.92 source-domain points to 8.42 target-domain points. The original certificate approves 141/150 overall comparisons, increases mean deployed F1 from 0.5819 to 0.6550, and retains 94.20% of available positive gain, with no observed evaluation-batch F1 losses. Subgroup counts and pointwise intervals are also reported.
For shifts beyond this ratio condition, Proposition D.8 provides certification under declared two-policy drift budgets. Across 210 controlled distribution/sample-size settings, the maximum observed all-beta erroneous-approval rate among the 570 setting-budget combinations whose budgets cover actual drift is 0.01%, at nominal delta=5%, with no observed F1-harmful approvals. This extension also retains power: when candidate capture falls by 5 percentage points and reference capture is unchanged, a predeclared budget of (0.05,0) yields approval rates of 99.99% and 99.29% for the two mechanisms at sample sizes 3,000/20,000.
A harmful hidden-shift control makes the correction's effect explicit. With candidate capture falling by 20 percentage points, reference capture unchanged, and sample sizes 1,000/5,000, the zero-drift direct rule produces 76.20% harmful approvals. A covering budget of (0.20,0.10) reduces approval to zero, withholding that harmful replacement.
Operationally, certification uses an explicit transport condition or externally justified drift envelope. Historical or delayed-label audits can inform the envelope under an assumption linking the audited environments to the deployment window. Corollary D.9 reports the candidate-drift budget that a certificate can tolerate when reference capture is fixed, rather than estimating the unobserved drift. Proposition D.10 characterizes why unlabeled target observations alone cannot verify label-conditional drift: identical observables can correspond to opposite signs of F_beta improvement.