acceptodds
Under review as a conference paper at ICLR 2027

Certified Decision Transfer under Unlabeled Shift

Abstract

In content moderation and fraud detection, model upgrades should identify more positive cases without increasing review workload. Yet higher offline AUC does not guarantee this improvement under distribution shift, delayed target labels hinder evaluation before deployment, and changes in score scale complicate threshold reuse. We study certified decision transfer: using historical labels and current unlabeled traffic to decide whether a candidate should replace an existing policy. Rate Matching (RM) lets the reference determine how many cases to select and the candidate determine which cases fill those slots. We derive finite-batch and population regret decompositions that separate selection-count error from ranking error, and show why global AUC superiority is insufficient at a fixed review count. Certified RM (C-RM) uses paired policy disagreements on historical positives and bounds target selection-rate changes using unlabeled traffic. For pre-specified policies evaluated on independent certification samples, stable positive-case detection rates enable finite-sample control of erroneous approvals for each comparison, simultaneously for all ; insufficient evidence retains the reference. In a GoEmotions audit, 3,578 of 26,250 class-level upgrade comparisons are harmful; fully paired C-RM approves 834 with no observed population or finite-batch F1 degradation. On COCO, disagreement factorization certifies 4.0–4.6 times as many upgrades as the direct-cell test using the same data and nominal error budget. Our framework supports evidence-based model replacement before target labels become available. Code is available at https://anonymous.4open.science/r/CDT-86F8/.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
distribution shift, model replacement, risk control, statistical certification, F-measure, label-free evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mhDoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Certified Rate Matching (C-RM), a framework for deciding whether to replace an existing classification model with a new one under distribution shift, when target labels are unavailable. The method first matches the number of positive predictions made by the existing model, allowing the candidate model to select different instances without increasing the review workload. It then uses historical labeled data and current unlabeled data to test whether the replacement is likely to improve F-score.

The main contributions include a decomposition of F-score regret into selection-count and ranking errors, a finite-sample statistical guarantee controlling erroneous model replacements, and a more efficient certification procedure based on disagreements between the two models.

Experiments on text and image classification benchmarks demonstrate that the method can identify beneficial model upgrades while avoiding harmful replacements. The proposed disagreement-based test also certifies substantially more upgrades than a simpler alternative.

Critical strengths
  1. Practically relevant and well-defined problem setting. The paper addresses an important deployment challenge: deciding whether to adopt a new model when current labels are unavailable and operational capacity must remain fixed. The distinction between global predictive performance and performance at a particular decision threshold is meaningful and well motivated.

  2. Statistical guarantees supported by targeted experiments. The finite-sample certification procedure provides explicit control over erroneous approvals under its stated assumptions. The disagreement-based construction also improves statistical power, with experiments showing 4.0–4.6 times as many certified upgrades on COCO under the same nominal error budget. This provides a convincing connection between the theoretical development and practical benefits.

Critical weaknesses
  1. Restrictive assumptions underlying the certification guarantee. The main guarantee requires positive-class detection rates to remain stable between source and target distributions. This assumption may be difficult to justify or verify without target labels, especially under the distribution shifts motivating the paper. Although the authors discuss bounded drift, the practical reliability of the certificate when this assumption is violated remains unclear. This limits its applicability to realistic deployment scenarios.

  2. Limited demonstrated utility and unclear advantages over simpler alternatives. The certification procedure is quite conservative, approving only 834 out of 26,250 candidate upgrades in the GoEmotions audit. While this avoids harmful replacements in the reported experiments, it also misses many potentially beneficial upgrades. Moreover, the core approach combines established ideas from rate matching, paired statistical testing, and conservative policy improvement. A clearer comparison with existing risk-control and model-selection approaches would help establish the novelty and practical advantages of the proposed framework.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response 1

Response 1: Thank the reviewer for these valuable suggestions sincerely.Our additional analysis shows that individual capture-rate invariance is stronger than the original certificate needs. We also added controlled experiments with substantial changes in both policies' capture rates and audited the original CivilComments models under natural shift.

Let gjDg_j^D denote policy jj's positive-class capture rate in domain DD, with SS the candidate, RR the reference, PP the source, and QQ the target. For positive source and target reference capture rates and positive target prevalence, the unchanged certificate retains its per-comparison error guarantee for simultaneous population FβF_{\beta} improvement for every β>0\beta > 0 under the weaker condition

gSQgRQ≥gSPgRP.\frac{g_S^Q}{g_R^Q} \geq \frac{g_S^P}{g_R^P}. On the original confidence event, approval implies that the source capture ratio exceeds both 1 and the target workload ratio. The condition above transfers this inequality to the target capture ratio, making both terms in the cross-multiplied F_beta difference positive. The approval rule and confidence level remain unchanged. In particular, both capture rates may decrease by a common multiplicative factor while this guarantee remains applicable.

We tested this case using the original direct-CP implementation at delta=0.05. Candidate and reference capture rates both decrease by 20% relative to their source values: 0.70 to 0.56 and 0.55 to 0.44, respectively. This violates individual stability while preserving their ratio. The new fixed-design study covers four common multipliers, two shift mechanisms, and two sample sizes, with 10,000 repetitions per setting. For the 20% reduction, original-rule approval rates are:

Source-positive / target-unlabeled countsPositive-composition shiftHidden conditional shift
1,000 / 5,00078.06%76.26%
3,000 / 20,000100%100%

The hidden-shift mechanism preserves the entire unlabeled joint decision distribution. The respective population F1 gains are 11.34 and 10.17 percentage points. These experiments measure certification power under the weaker transport condition, using the original rule.

The original-model CivilComments audit provides complementary natural-shift evidence. Across three Qwen3 training seeds, candidate capture-rate changes are -0.39, -0.37, and -0.79 percentage points, compared with -2.02 points for the TF-IDF reference. The candidate's average capture advantage increases from 6.92 source-domain points to 8.42 target-domain points. The original certificate approves 141/150 overall comparisons, increases mean deployed F1 from 0.5819 to 0.6550, and retains 94.20% of available positive gain, with no observed evaluation-batch F1 losses. Subgroup counts and pointwise intervals are also reported.

For shifts beyond this ratio condition, Proposition D.8 provides certification under declared two-policy drift budgets. Across 210 controlled distribution/sample-size settings, the maximum observed all-beta erroneous-approval rate among the 570 setting-budget combinations whose budgets cover actual drift is 0.01%, at nominal delta=5%, with no observed F1-harmful approvals. This extension also retains power: when candidate capture falls by 5 percentage points and reference capture is unchanged, a predeclared budget of (0.05,0) yields approval rates of 99.99% and 99.29% for the two mechanisms at sample sizes 3,000/20,000.

A harmful hidden-shift control makes the correction's effect explicit. With candidate capture falling by 20 percentage points, reference capture unchanged, and sample sizes 1,000/5,000, the zero-drift direct rule produces 76.20% harmful approvals. A covering budget of (0.20,0.10) reduces approval to zero, withholding that harmful replacement.

Operationally, certification uses an explicit transport condition or externally justified drift envelope. Historical or delayed-label audits can inform the envelope under an assumption linking the audited environments to the deployment window. Corollary D.9 reports the candidate-drift budget that a certificate can tolerate when reference capture is fixed, rather than estimating the unobserved drift. Proposition D.10 characterizes why unlabeled target observations alone cannot verify label-conditional drift: identical observables can correspond to opposite signs of F_beta improvement.

AuthorsOpenReview ↗

Response 2

Response 2:

We expanded the evaluation to quantify avoided losses, missed gains, and comparisons with alternative statistical rules, while retaining the original models, thresholds, and paired evidence.

In the original GoEmotions TF-IDF-to-Qwen3 audit, unconditional replacement lowers empirical target-pool F1 in 3,578 of 26,250 comparisons (13.63%). These harmful comparisons have a mean decline of 1.59 F1 percentage points; 1,118 decline by at least 2 points, 149 by at least 5 points, and the largest decline is 7.28 points. The original certificate rejects all these harmful proposals while approving 834 beneficial replacements, whose mean F1 gain is 9.23 points. To isolate selection from the number of approvals, uniformly choosing exactly 834 comparisons would produce an expected 113.68 harmful replacements and a mean deployed gain of 0.141 points; the original gate achieves zero observed harmful replacements and a gain of 0.293 points. The random-control values are exact expectations over subsets of the fixed archive.

Using the same evidence and delta=0.05, the paper's factorized rule increases approvals to 1,634. A four-bound ratio CP refinement developed for this rebuttal further increases approvals to 2,072, retaining all 834 original approvals and adding 1,238 beneficial replacements. Mean deployed F1 rises from 0.53109 to 0.53462, compared with the reference's 0.52816. This increases the deployed gain by 120.52%, with no observed empirical-pool or evaluation-batch F1 losses under either the original or new rule. Unconditional replacement attains a higher mean F1 of 0.57264 while accepting the declines quantified above.

We evaluated 12 statistical rules at the same nominal delta, including exact paired-proportion tests, joint multinomial regions, concentration-bound alternatives, and score/Wald approximations. Principal comparisons are:

RuleGoEmotions: 26,250 comparisonsCOCO archive: 63,000 comparisons
Original direct CP834 (0 / 0)352 (0 / 0)
Paper's factorized CP (replay)1,634 (0 / 0)1,334 (0 / 6)
New four-bound ratio CP2,072 (0 / 0)2,253 (0 / 9)
Exact McNemar-IUT178 (0 / 2)2,003 (1 / 55)
Joint multinomial mixture region, coupled ratio0 (0 / 0)0 (0 / 0)
All-beta delta-Wald, asymptotic6,727 (151 / 174)19,926 (6 / 390)

Entries are approvals (empirical-pool / evaluation-batch F1 loss counts). COCO uses the archived 100-target-seed experiment, distinct from the main-text 31,500-comparison experiment. McNemar-IUT uses the full delta for each directional test. The joint baseline preserves dependence within fixed four-cell Dirichlet(1/2,1/2,1/2,1/2) mixture confidence regions, assigning delta/2 to each domain. Our criterion can certify upgrades where increased capture compensates for increased workload, whereas this directional McNemar-IUT requires workload reduction. Wald yields a larger GoEmotions mean gain of 1.910 points, together with the losses shown above.

We also quantify missed utility. Of 22,516 F1-improving proposals, the original rule misses 21,682 and the new rule misses 20,444; positive-gain retention increases from 6.29% to 13.87%. For the original rule, 1,060 rejected proposals improve every finite-beta objective but fail the strict sufficient condition, accounting for 0.73% of available positive F1 gain. In contrast, 19,531–19,772 rejected proposals satisfy the noiseless target-pool sufficient condition and account for 91.80%–92.63% of available positive gain. This latter gap includes confidence-bound conservatism and possible source–target discrepancies. A separate fixed-law simulation isolates sample size: increasing source-positive/target-unlabeled counts from 250/1,000 to 1,000/5,000 to 5,000/20,000 raises new-rule approval from 4.82% to 65.48% to 100% over 10,000 repetitions per setting.

The methodological contribution is the comparison available under this supervision pattern. Historical positive labels estimate paired capture differences, while current unlabeled traffic estimates paired selection rates. Their sufficient margin certifies relative population F_beta improvement simultaneously for every beta>0 without estimating target prevalence, under the stated transport assumptions. An independent matching window constructs the replacement at the reference workload, and certification accounts for subsequent workload mismatch. Learn then Test provides a general framework for valid tests; C-RM supplies a test for this particular partially supervised comparison. Conformal Risk Control, including its shift extensions, controls an expected-loss target, while HCPI/SPIBB use logged rewards or transitions to study policy returns. C-RM's contribution is the decision-transfer formulation, paired certificate, disagreement factorization with improved empirical power, and identifiability boundary for deploying a fixed classifier replacement before target labels arrive.

Thank you for the efforts, I will raise my score by 1.

Thank you for revisiting our submission and considering the additional analyses. We appreciate your updated assessment and the time you have devoted to the discussion. Your constructive feedback helped us clarify the presentation and strengthen the empirical evaluation.

Reviewer_yTGSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors investigagte whether a candidate model can safely replace an existing decision policy using historical labels and unlabeled target data during deployment. The deployment problem is important, and the paired certification construction is clearly motivated. However, I have several concerns regarding the assumptions, baselines and practical benefit demonstrated by the experiments.

Critical strengths

see above

Critical weaknesses
  1. The central guarantee depends on stability of each policy’s positive-class capture rate between source and target. In most experiments, this assumption holds by construction because the target distributions are generated through class-conditional resampling that changes prevalence. These experiments establish behavior under the assumed model but provide limited evidence about robustness when the positive population changes. The CivilComments results are encouraging, but capture-rate stability is not reported. Since target labels are available retrospectively, the authors should quantify the actual capture-rate drift and evaluate how certification behaves under controlled violations.

  2. The main audit raises concerns about practical utility. Fully paired C-RM approves only 834 of 26,000 GoEmotions comparisons. Mean deployed F1 increases from .5282 to .5311, compared with .5726 for unconditional replacement. Avoiding harmful upgrades is valuable, but zero observed losses alongside very high abstention provides limited evidence of a useful deployment trade-off. Can you discuss this?

  3. Stronger certification baselines are needed. The main power comparison compares variants of the proposed construction, and it is unclear if the factorization improves over established methods for paired proportions or joint multinomial confidence regions using the same evidence and error budget. The authors should additionally distinguish abstention caused by insufficient samples from beneficial upgrades excluded by their sufficient condition, which certifies improvement for every F-beta objective simultaneously.

Questions for authors

see above

5 comments
AuthorsOpenReview ↗

Response 3:

Reviewer concern 3:

Stronger certification baselines are needed. The main power comparison compares variants of the proposed construction, and it is unclear if the factorization improves over established methods for paired proportions or joint multinomial confidence regions using the same evidence and error budget. The authors should additionally distinguish abstention caused by insufficient samples from beneficial upgrades excluded by their sufficient condition, which certifies improvement for every F-beta objective simultaneously.

Response 3:

We sincerely thank the reviewer for these valuable suggestions. In response, we expanded the certification baseline comparisons and further analyzed the sources of abstention. The expanded comparison shows that certification power can improve substantially without changing the models, thresholds, or paired evidence. At delta=0.05, our new four-bound Clopper–Pearson (CP) certificate approves 2,072 GoEmotions comparisons versus 178 for exact McNemar-IUT, and 2,253 COCO comparisons versus 2,003. It incurs no empirical-target-pool F1 losses in either archive; evaluation-batch losses are 0 and 9, respectively. On GoEmotions, its mean deployed F1 gain increases from the original direct rule's 0.293 to 0.647 percentage points.

We evaluated 12 rules, including exact paired-proportion tests, joint multinomial confidence-region methods, empirical-Bernstein and Hoeffding–Bentkus bounds adapted to the certification criterion, and three large-sample score/Wald methods. All use identical paired evidence and nominal delta. Exact McNemar-IUT receives the full delta for each directional component under the intersection-union construction. The joint multinomial comparators use fixed four-cell Dirichlet(1/2,1/2,1/2,1/2) likelihood-mixture confidence regions, with delta/2 allocated to each domain. We evaluate both separate projections and coupled ratio optimization; the latter retains within-domain joint dependence when certifying the capture/workload ratio inequalities. Our new four-bound ratio CP rule uses four preallocated one-sided CP bounds under the original assumptions.

RuleGoEmotions: 26,250 comparisonsCOCO archive: 63,000 comparisons
Original direct CP834 (0 / 0)352 (0 / 0)
Factorized CP1,634 (0 / 0)1,334 (0 / 6)
New four-bound ratio CP2,072 (0 / 0)2,253 (0 / 9)
Exact McNemar-IUT178 (0 / 2)2,003 (1 / 55)
Joint multinomial mixture region — separate projections0 (0 / 0)0 (0 / 0)
Joint multinomial mixture region — coupled ratio0 (0 / 0)0 (0 / 0)
All-beta delta-Wald, asymptotic6,727 (151 / 174)19,926 (6 / 390)

Entries are approvals (empirical-pool F1 losses / evaluation-batch F1 losses). The COCO results use the archived 100-target-seed experiment with 63,000 comparisons; the corresponding main-text experiment contains 31,500 comparisons.

Our criterion can certify upgrades whose capture improvement compensates for increased workload, whereas the directional McNemar-IUT requires workload reduction. Directly estimating the all-beta criterion with a Wald approximation yields higher approval rates and mean gains, alongside more F1 losses. On GoEmotions, Wald achieves a 1.910-percentage-point mean deployed F1 gain, versus 0.647 for the four-bound certificate, with the losses shown above.

The improvement also extends across the three archived CivilComments candidates. On the overall test population, approvals increase from 141 to 150 for Qwen3, 90 to 148 for RoBERTa, and 108 to 147 for ModernBERT, out of 150 comparisons each, with no observed evaluation-batch F1 losses.

We also decomposed abstention for the original GoEmotions direct rule. Among unapproved F1-improving proposals, 850–1,091 do not satisfy the objective of improvement for every finite beta>0; 1,060 satisfy that objective but fail the strict sufficient condition; and 19,531–19,772 satisfy the noiseless target-pool sufficient condition but remain unapproved. The additional structural exclusions from the sufficient condition account for 0.73% of available positive F1 gain, whereas the last category accounts for 91.80%–92.63%. The ranges reflect ambiguity in recovered integer parameters. The evidence gap in the last category includes confidence-bound conservatism and possible source–target discrepancies.

A fixed-law simulation separately isolates sample size: with capture improvement and equal workload, increasing source-positive/target-unlabeled sample sizes from 250/1,000 to 1,000/5,000 to 5,000/20,000 raises four-bound approval from 4.82% to 65.48% to 100% over 10,000 repetitions per setting. Across the 24 calibration settings, its maximum observed all-beta erroneous-approval rate is 0.96% at nominal delta=5%.

AuthorsOpenReview ↗

Response 2:

Reviewer concern 2: The main audit raises concerns about practical utility. Fully paired C-RM approves only 834 of 26,000 GoEmotions comparisons. Mean deployed F1 increases from .5282 to .5311, compared with .5726 for unconditional replacement. Avoiding harmful upgrades is valuable, but zero observed losses alongside very high abstention provides limited evidence of a useful deployment trade-off. Can you discuss this?

Response 2:

We quantify this trade-off through both realized gains and the downside of rejected replacements. In the original 26,250 GoEmotions comparisons, unconditional replacement lowers empirical target-pool F1 in 3,578 cases (13.63%). These harmful cases have a mean decline of 1.59 F1 percentage points: 1,118 decline by at least 2 points, 149 by at least 5 points, and the largest decline is 7.28 points. Across all comparisons, the worst 5% have a mean decline of 3.10 points. The original certificate rejects all these harmful proposals while enabling 834 beneficial replacements. Thus the protection is measurable in both frequency and magnitude, including the lower tail.

We also increased utility using the same models, thresholds, paired evidence, and delta=0.05. The new four-bound ratio CP certificate developed for this rebuttal retains all 834 original approvals and adds 1,238 beneficial replacements, for 2,072 approvals in total. Mean deployed F1 gain increases by 120.52%, from 0.293 to 0.647 percentage points, with no observed empirical-pool or evaluation-batch F1 losses under either rule.

RuleReplacementsMean deployed F1Empirical-pool F1 declines
Always retain the reference00.528160
Original direct CP8340.531090
Existing factorized CP, supplementary replay1,6340.533240
New four-bound ratio CP2,0720.534620
Unconditional replacement26,2500.572643,578

Always retaining the reference also avoids replacement losses, but obtains no upgrade gain. The original rule instead realizes a mean improvement of 9.23 F1 points among its approved proposals. The new rule increases the share of available positive gain captured from 6.29% to 13.87%. Unconditional replacement still achieves higher average F1 while accepting the declines quantified above. The comparison therefore makes the gain-versus-downside trade-off explicit.

We report the remaining opportunity cost as well: among 22,516 F1-improving proposals, the original rule misses 21,682 and the new rule misses 20,444. Our third response decomposes these omissions and tests improved certification and increased sample sizes as routes to higher power. High abstention is also not universal: in the original CivilComments overall audit, the original rule approves 141/150 comparisons, increases mean F1 from 0.5819 to 0.6550, and retains 94.20% of available positive gain.

AuthorsOpenReview ↗

Response 1

Reviewer concern 1: The central guarantee depends on stability of each policy’s positive-class capture rate between source and target. In most experiments, this assumption holds by construction because the target distributions are generated through class-conditional resampling that changes prevalence. These experiments establish behavior under the assumed model but provide limited evidence about robustness when the positive population changes. The CivilComments results are encouraging, but capture-rate stability is not reported. Since target labels are available retrospectively, the authors should quantify the actual capture-rate drift and evaluate how certification behaves under controlled violations.

Response 1:

We added two targeted analyses: a retrospective audit of capture-rate drift for the original CivilComments models, and controlled experiments evaluating the bounded-drift extension in Proposition D.8. The original certificate retains 94.20% of available positive F1 gain on CivilComments despite measurable capture-rate changes, while the bounded-drift extension maintains substantial certification power under nonzero drift.

First, we audited the archived results of the original Qwen3 models across three training seeds and the TF-IDF reference, retaining the original thresholds and data partitions. The candidate's capture-rate changes are -0.39, -0.37, and -0.79 percentage points across the three seeds, compared with -2.02 points for the reference. The candidate's average capture-rate advantage over the reference increases from 6.92 points in the source domain to 8.42 points in the target domain.

The original certification rule approves 141 of 150 overall-population comparisons, increasing mean deployed F1 from 0.5819 to 0.6550, with no observed evaluation-batch F1 losses. These approvals retain 94.20% of the positive F1 gain available from beneficial replacements.

We also quantify subgroup heterogeneity. For example, among comments mentioning male identity, candidate capture rates decline by 5.60–7.53 percentage points across seeds, while the reference declines by 7.08 points. The supplementary audit reports subgroup counts and pointwise confidence intervals alongside the overall results.

Second, Proposition D.8 extends exact capture-rate stability to externally specified bounds on capture-rate drift. We evaluated this extension across 210 distribution and sample-size settings, including changes in positive-population composition and hidden conditional shifts that preserve the entire unlabeled joint prediction distribution. The guarantee applies when the specified budgets cover the actual drift.

In one beneficial-upgrade setting, the candidate's capture rate decreases by 5 percentage points, the reference remains unchanged, and the predeclared drift budget is (epsilon_S, epsilon_R) = (0.05, 0). Approval rates over 10,000 repetitions per setting are:

Source-positive / target-unlabeled sample sizesPositive-composition shiftHidden conditional shift
1,000 / 5,00043.15%17.00%
3,000 / 20,00099.99%99.29%

Thus, accommodating nonzero capture-rate drift can preserve high approval power when sufficient certification evidence is available.

Across the 570 tested setting–budget combinations in which the budgets cover the actual drift, the bounded-drift direct-CP rule's maximum observed erroneous-approval rate for the all-beta criterion is 0.01%, at nominal delta = 5%, with no observed F1-harmful approvals.

We additionally tested a harmful-upgrade control. Under a hidden conditional shift where the candidate's capture rate decreases by 20 percentage points and the reference remains unchanged, with 1,000 source positives and 5,000 target-unlabeled examples, the zero-drift rule produces 76.20% harmful approvals. Applying a covering budget of (0.20, 0.10) reduces the approval rate to zero, correctly withholding this harmful replacement. This control directly demonstrates the operational effect of incorporating drift into the certification rule.

Reviewer_QpoCOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the problem whether a candidate model should replace an existing reference policy when current labels are not available without increasing review workload. Rate Matching gives the candidate the selection count of the reference policy. Certified Rate Matching then uses historical positives and unlabeled current traffic to approve replacements with evidence of improved population FβF_\beta. Its finite-sample guarantee applies to fixed policies when positive-class detection rates remains stable across populations. Experiments show that higher AUC can conceal worse decisions at the operating point, and that pairing policy disagreements increases the number of upgraded certified.

Critical strengths

S1. Practical problem. The paper addresses a practical model-replacement decision without increasing the review workload. The analysis explains why AUC does not always predict which policy finds more positive cases in the review queue.

S2. Clear assumptions. The certificate clearly states its independence and stability assumptions and provides an exact error guarantee for each comparison.

S3. Good use of pair data. The disagreement-based test certifies more upgrades than the direct-cell approach using the same evidence.

S4. Useful experiments. The experiments include examples of harmful upgrades and natural-shift audits, showing both approvals and cases where the reference policy is kept.

Critical weaknesses

W1. Unverifiable assumption. Positive-class detection rate stability is central to the guarantee, but cannot be verified from unlabeled target traffic. Under unlabeled traffic configuration, one can see which transactions each model selects but cannot tell which transactions are actually fraudulent. The natural-shift audits therefore show promising results, but cannot certify that the guarantee applies there.

W2. Certification is conservative. On GoEmotions, the method approves 834 of 26,250 TF–IDF-to-Qwen comparisons despite a substantial average gain. The paper should explain more clearly how many useful upgrades this cautious approach misses.

W3. Reused test samples. The many test comparisons draw from the same underlying examples. Seeing no harmful approved switches is encouraging, but it is not the same as seeing no harm across that many independent deployments. Testing on new data collected at different times or sites would provide stronger evidence about how often harmful switches occur in practice.

W4. Future workload may vary. The fixed cutoff matches workload on the matching window, while future selection rates can differ. The certificate accounts for this statistically, but the operational size of those workload deviations deserves more attention.

Questions for authors
  1. How can someone decide how much the models’ ability to catch positive cases might change before target labels are available?

  2. How large are realized workload deviations after the cutoff is fixed, especially when traffic is small or rapidly changes?

6 comments
AuthorsOpenReview ↗

Response 1:

Response 1:

Our additional analysis broadens the conditions under which the original certificate applies and makes the evidence needed for a predeployment drift budget explicit. The natural-shift audit measures performance retrospectively; the transport condition supporting certification must come from information available before the deployment decision.

First, individual capture-rate invariance is stronger than necessary for the original gate's improvement-sign guarantee. Let gjDg_j^D denote policy jj's positive-class capture rate in domain DD, with SS the candidate, RR the reference, PP the source, and QQ the target. With positive source and target reference capture rates and positive target prevalence, the unchanged gate retains its per-comparison error guarantee for simultaneous population FβF_{\beta} improvement for every β>0\beta > 0 under the weaker condition

gSQgRQ≥gSPgRP.\frac{g_S^Q}{g_R^Q} \geq \frac{g_S^P}{g_R^P}.

Thus, both models may become worse at detecting positives while the guarantee remains applicable, provided the candidate's relative capture advantage does not deteriorate. In a new controlled study using the original direct CP rule at δ=0.05\delta=0.05, capture rates decrease from 0.70 to 0.56 for the candidate and from 0.55 to 0.44 for the reference. With 1,000 source positives and 5,000 unlabeled target examples, approval rates are 78.06% under positive-composition shift and 76.26% under a hidden shift preserving the entire unlabeled joint decision distribution. At 3,000/20,000 examples, both rates are 100%. Each setting has 10,000 repetitions. These are power measurements under the weaker transport condition.

The original-model CivilComments audit is consistent with this mechanism: across three Qwen3 training seeds, candidate capture changes by -0.39, -0.37, and -0.79 percentage points, versus -2.02 points for the reference. The average capture advantage increases from 6.92 to 8.42 points. The original gate approves 141/150 overall comparisons and retains 94.20% of available positive F1 gain, with no observed evaluation-batch F1 losses. These measurements use original candidate outputs and recovered original-reference aggregate statistics.

Second, a drift allowance before current target labels arrive requires a structural envelope or labeled historical environments linked to future deployment. For policies fixed independently of those environments, an explicit protocol combines simultaneous binomial intervals for historical capture-rate changes with an environment-level order-statistic bound. Under exchangeability of historical and future environments, the budget's failure probability is at most αenv+η\alpha_{\mathrm{env}} + \eta; certification then spends δcert\delta_{\mathrm{cert}}, giving total erroneous-approval risk at most αenv+η+δcert\alpha_{\mathrm{env}} + \eta + \delta_{\mathrm{cert}}. This guarantee is marginal over historical and future environment draws, not conditional on a particular realized target environment or matching window. This adapts the multi-environment calibration principle of Duchi et al., rather than inferring label-conditional drift from unlabeled selection rates. For example, a total 5% budget can be allocated as 2% environment uncertainty, 1% interval uncertainty, and 2% certification error; the original 5% certificate results above are a separate experiment.

Target-dependent threshold selection requires calibration simultaneous over the selectable policy family, or another separately justified selection protocol. Our historical-environment simulation tests fixed policy pairs. Its assumptions cannot automatically be transferred to arbitrary thresholds selected on the current target window.

Once an external allowance is available, Proposition D.8 directly incorporates it. For a declared 5-percentage-point candidate-capture allowance with unchanged reference capture, the original drift-adjusted gate approves 99.99% and 99.29% of the two controlled-shift settings at 3,000/20,000 examples. Corollary D.9 separately reports the maximum drift the observed certificate could tolerate. This is a robustness margin, not an estimate of future drift. Proposition D.10 explains the remaining identification boundary: identical unlabeled observations can correspond to opposite signs of improvement. Historical calibration supplies additional evidence and an explicit environmental assumption; it does not remove this boundary.

Reference: Duchi, Gupta, Jiang, and Sur, Predictive Inference in Multi-Environment Scenarios, Statistical Science (2025), https://doi.org/10.1214/24-STS973; full text https://arxiv.org/html/2403.16336v2. The historical-TPR budget construction is our adaptation, not a theorem quoted verbatim from that paper.

AuthorsOpenReview ↗

Response 2

Reviewer concern 2: Certification is conservative. On GoEmotions, the method approves 834 of 26,250 TF–IDF-to-Qwen comparisons despite a substantial average gain. The paper should explain more clearly how many useful upgrades this cautious approach misses.

Response 2:

We now quantify both the number of missed upgrades and their forgone F1 gain. In the original GoEmotions archive, 22,516 of 26,250 proposals improve empirical target-pool F1. The original direct certificate approves 834 and misses 21,682 of these beneficial proposals, retaining 6.29% of the total available positive F1 gain. Using the identical models, thresholds, paired evidence, and nominal delta=0.05, the paper's factorized construction approves 1,634. A four-bound ratio CP refinement developed for the rebuttal approves 2,072, including all 834 original approvals and 1,238 additional beneficial replacements.

RuleApprovedMissedGain retainedΔF1 (pp)
Original direct CP83421,6826.29%+0.293
Factorized CP (replay)1,63420,88210.89%+0.508
New four-bound ratio CP2,07220,44413.87%+0.647

All three rules have zero observed empirical-pool and evaluation-batch F1 losses in this archive. The new refinement increases mean deployed gain by 120.52% over the original rule.

The decomposition identifies where further gains are available. Among beneficial proposals rejected by the original rule, 850–1,091 fail the stronger objective of improving every finite-beta F_beta score; 1,060 meet that objective but fail the strict sufficient condition, accounting for 0.73% of available positive gain; and 19,531–19,772 satisfy the noiseless target-pool sufficient condition but remain unapproved, accounting for 91.80%–92.63% of available positive gain. The ranges reflect ambiguity in recovered integer parameters. The final category includes confidence-bound conservatism and source–target discrepancies; it is not solely a sample-size effect.

A separate fixed-law experiment isolates the effect of sample size. With improved capture and equal workload, increasing source-positive/target-unlabeled counts from 250/1,000 to 1,000/5,000 to 5,000/20,000 increases four-bound approval from 4.82% to 65.48% to 100%, with 10,000 repetitions per setting. Together, these results motivate tighter bounds and more positive certification examples while keeping the intended safety objective fixed.

The protection also has measurable value: unconditional replacement lowers empirical target-pool F1 in 3,578 comparisons (13.63%), with a mean decline of 1.59 points among those cases and a worst decline of 7.28 points. The original and refined gates reject all of these observed harmful proposals while obtaining the positive gains above. Unconditional replacement still achieves a higher mean F1 of 0.57264, compared with 0.53109 and 0.53462 for the original and refined gates; the reference mean is 0.52816. The comparison therefore reports both the gains obtained and the cost of withholding uncertain upgrades.

AuthorsOpenReview ↗

Response 4

Reviewer concern 4: Future workload may vary. The fixed cutoff matches workload on the matching window, while future selection rates can differ. The certificate accounts for this statistically, but the operational size of those workload deviations deserves more attention.

Related question: How large are realized workload deviations after the cutoff is fixed, especially when traffic is small or rapidly changes?

Response 4:

We added a workload audit using the original CivilComments models and frozen thresholds. Each model family has three training seeds and 50 partitions per seed; each overall-population comparison uses 53,512 matching comments and a disjoint evaluation set of 40,135 comments. The table reports selection-rate differences as actions per 1,000 comments. Positive values mean increased workload.

ModelMean shiftP5–P95Deployed shift
Qwen3-0.075[-2.778, 3.478]+0.348
RoBERTa-0.208[-3.008, 3.254]-0.028
ModernBERT+0.052[-2.785, 2.942]+0.117

All values are actions per 1,000 comments. Mean shift and P5–P95 describe the candidate's evaluation-minus-matching selection-rate difference. Deployed shift is the mean evaluation-set difference between the gate-deployed policy and the original reference. Positive values indicate increased workload.

The final column averages all 150 comparisons per model, retaining the reference in unapproved cases. Original approvals are 141, 90, and 108, respectively.

Small windows exhibit substantially greater variation. For archived partition 0, we partition the original held-out evaluation set into non-overlapping windows and report all three Qwen3 training seeds:

Comments per windowWindows per seedSeed 42: 5th–95th percentilesSeed 43Seed 44
100401[-50.21, 59.79][-50.25, 59.75][-50.27, 59.73]
1,00040[-13.36, 13.89][-16.25, 17.95][-10.37, 13.78]
5,0008[-5.04, 3.89][-5.29, 5.87][-4.49, 4.72]

Values remain actions per 1,000 comments, relative to the original matching-window candidate rate; they are empirical quantiles, not confidence limits.

We also sort those same held-out comments by timestamp while keeping the original thresholds. In 1,000-comment windows, the upper 95th-percentile deviation increases to 34.00–39.88 actions per 1,000 across the three seeds. This retrospective time stratification exposes workload variation hidden by random ordering. It is not a minute-scale burst or prospective deployment test: even 100-comment windows span a median 24.92 hours in this benchmark subset, and training/matching samples include later dates.

The audit also measures a small initial matching discrepancy from boundary ties in the archived quantized scores: at most 0.804 extra actions per 1,000 matching comments across these models.

These measurements clarify the operational interpretation. The frozen cutoff anchors workload to the matching window, while the certificate accounts for the measured candidate reference selection-rate difference on independent target traffic. It does not impose a hard capacity on every future batch. For deployment, workload monitoring is directly observable without labels; certification should be renewed on fresh target traffic when its distribution changes, with repeated decisions covered by an appropriate risk allocation. Any new cutoff is a new policy requiring renewed certification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Title: Summary of Rebuttal and Key Evidence

Dear Area Chair,

Our paper studies model replacement before target labels arrive. Rate Matching (RM) takes the incumbent's review count on a matching window and lets the candidate fill those slots. Our finite-batch and population F-beta decompositions separate errors in selection count from errors in case ranking. On GoEmotions, ModernBERT has higher source AUC than RoBERTa but lower target F1 at the reference-matched operating point.

The paper's second contribution is an approval rule. Certified RM compares the policies on historical positives where their decisions differ and uses unlabeled target traffic to account for workload changes after the cutoff is frozen. On COCO, separating how often the policies disagree from which policy wins those disagreements certifies 4.0 to 4.6 times as many replacements as the original direct test on the same evidence.

The rebuttal strengthened both the guarantee and its evaluation. We showed that the original rule does not require each policy's recall to remain constant: both recalls may fall if the candidate-to-incumbent recall ratio does not decrease. With fixed policies, positive reference recalls and target prevalence, and independent i.i.d. certification samples, the rule controls the probability of an erroneous approval per comparison, simultaneously for all beta > 0. Target scores alone cannot verify the recall assumption. When it lacks support, an independent labeled target audit provides a separate route to certification.

We compared 12 decision rules on the same evidence. On the GoEmotions TF-IDF-to-Qwen3 archive at nominal 5% risk, the four-bound refinement increased approvals from 834 to 2,072 and raised mean deployed F1 gain over keeping the incumbent from 0.29 to 0.65 percentage points. It withheld all 3,578 observed F1 declines and retained 13.9% of the available positive gain.

Under natural shift, the same rule retained 71.9% of positive gain on CivilComments-to-Wikipedia transfer and 78.9% at six held-out air-quality stations. Of 36 station proposals with higher source AUC, four lowered held-out F1; the gate withheld all four and approved 16 improvements. The four declines involved the same model pair, and each station-specific 95% week-block bootstrap interval for the F1 change included zero. These comparisons share models and observations across proposals.

QpoC confirmed that the analyses of missed upgrades and workload variability addressed W2 and W4. yTGS confirmed that the expanded baselines and abstention analysis addressed their third concern.

The practical contribution is a way to distinguish upgrades supported by the available evidence from those for which it is worth waiting for labels. Its value depends on the cost of a mistaken switch as well as the gains lost by waiting.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.