acceptodds
Under review as a conference paper at ICLR 2027

When Is a Veto Worth Its Cost? Proposer Precision, Gate Selectivity, and Requalification under Distribution Shift

Abstract

Downstream gates can reject, defer, or override proposals, but every veto can also suppress a useful action. We study requalification under distribution shift: a proposer recommends reactivating a previously rejected alternative and a downstream gate either authorizes the switch or keeps the incumbent. We operationalize the classical cost-sensitive/net-benefit break-even form using independently measured blocking rates and blocked-outcome severities, and compare learned gating with a switch-rate-matched random blocker. Across three controlled domains, calibration-only operating points predict holdout sign and magnitude better than a naive fixed-0.5 rule in the informative domains, while fresh audits show within-domain drift. A targeted severity learner fails: its low-precision benefit is mostly refusal rate, while an equal-rate oracle reveals unrealized selectivity. Degrading that oracle yields a representation target of Spearman , versus – for the learned signal. Fresh feature-tier and small-budget audits further show that better ranking and probability calibration do not necessarily improve frozen-threshold regret. A frozen external stress test on real handwritten-digit observations under constructed corruption shows the audit's own refusal condition: the break-even is weakly identified there, yet matched-rate selectivity remains detectable in three corruption levels. The contribution is an empirical operating audit that separates ranking, calibration, threshold portability, and selective action value.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
selective prediction, learning to defer, distribution shift, calibration, cost-sensitive learning, abstention, selective classification, decision making under uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Xn8AOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates requalification under distribution shift. Some alternative that was previously rejected is recommended to be reactivated by a proposer model. A gate then accepts or rejects the switch. They state a necessary and sufficient condition for when a gate decreases expected regret: if and only if the proposer's precision π is below q_w H_w / (q_w H_w + q_c H_c), where q_c and q_w are the rates at which the gate blocks correct and wrong proposals and H_c and H_w are the average costs of those blocks. The paper measures these quantities on held-out data, and compare the gate with controls like random rate-matched blockers.

Critical strengths
  1. It would be interesting to measure the cost of vetoes in some domains.
  2. The math seems correct.
Critical weaknesses
  1. The paper's focus seems unmotivated. I did not get a good sense of why this question is particularly important or interesting to study.
  2. "Under distribution shift, that balance can change abruptly because the proposer itself can move from unreliable to highly reliable." This sort of move would be surprising! If a proposer is unreliable in-distribution, it seems unlikely that it would become highly reliable out-of-distribution. It would be kind of like a child that has only been taught English and failed to learn to speak it correctly turning out to be very good at speaking Spanish without any training.
  3. The paper's experiments are all on synthetic data, but the kind of question it studies would only be interesting if it were on real data, and there's little reason to think that experiments on synthetic data are any real guide to performance on real data in this case.
  4. The controlled-precision experiment results seem trivial. Here the only thing that's changed is the mixture proportion pi, so q_c, q_w, H_c, and H_w remain the same, and the results match Δ = ρ[π q_c H_c − (1−π) q_w H_w] with sampling errors.
  5. The paper's headline effect is very small: the change in regret is -0.0034.
  6. The paper seems substantially AI-generated with little human vetting. Tells include a substantial dressing-up of mostly trivial and uninteresting results, extensive references to the history of the project ("An early version of this project," etc.), and many new coinages (QRL, ContextEvidenceML, FullValidityVeto, EvidenceRank, DelayMatchedRank, LearnedDeferred, etc.).
Questions for authors
  1. Since only π changes between calibration and testing, how could the calibration-only prediction have been falsified?
  2. Can you test on a real gating task?
2 comments

Motivation. Whenever a deployed model is retrained, the old and new models disagree on some inputs, and for each of those inputs someone must decide whether to accept the new answer. The gate is that decision rule, and the question is when it lowers regret.

"The proposer itself can move from unreliable to highly reliable." The sentence was badly put, and your analogy is fair for a fixed proposer on a new distribution. In the setting we now study the proposer is the retrained model, and its precision is the share of its disagreements with the deployed model on which it is right. That share depends on how stale the deployed model has become, and it moved a great deal within single streams: across periods from 0.22 to 0.86 on Elec2, 0.23 to 0.83 on Rialto and 0.08 to 0.66 on INSECTS-abrupt. The revision replaces the sentence with these observed ranges.

Q1. How could the calibration-only prediction have been falsified? In the submitted experiment it could not; you were right, and the controlled-precision result is now labelled an identity check. In the new experiments the predictions were posted in this forum before scoring, and many failed: label-free monitoring of the gate effect on HuffPost and arXiv (r = -0.19 and -0.42); transport of the frozen consequence estimates on HuffPost (r = 0.10); the label-budget rule on arXiv; the fallback M7 on covtype and poker; and two of the three mechanism predictions tested on INSECTS. Three later policies gave no confirmed result. Only one of our five adaptive policies met its primary criterion.

Q2. A real gating task. The new experiments use model update on public data: news topics by month (HuffPost), arXiv categories by month, and twelve further evaluable streams. The observations are real. The update schedule and the gate are ours, and several streams are in file order, not time order. It is not a deployed system, and we say so.

Effect size. We now report effects relative to the regret of deploying the update with no gate. The learned gate changed regret by +4.8% (HuffPost), +22.0% (arXiv), -51% (covtype and poker), -31% (INSECTS-abrupt), and +18% and +7% (the other two INSECTS streams). The sign changes with the regime, which is now the paper's main finding.

AI assistance and vetting. The submission declared AI assistance for writing, ideation and execution, drafting, synthetic data and proofs. The same holds for the new experiments, and the revision states it. We take responsibility for every claim. Each commitment used code hashes posted before scoring; second runs reproduced the verdicts, with small floating-point differences in 9 of 54 C3 periods. Failures are reported alongside successes. The revision removes the project-history passages and the coined names you listed, replaces them with standard descriptions, and organises the experiments around one question.

We would be grateful to know whether this addresses your concerns.

Reviewer_Xn8AOpenReview ↗

Reply

Thank you. I'm going to keep my score. It looks like the human authors aren't paying any attention to this paper at all. The replies here are entirely AI-generated, and the paper has been completely overhauled in response to the reviews. It's not worth my attention to engage with this.

Reviewer_q6TXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies when a downstream veto is worth its cost in a two-stage proposer + gate system. The task is requalification: after a distribution shift, a proposer recommends reactivating a previously rejected alternative, and a gate either authorizes the switch or keeps the incumbent. The authors decompose the gate's value into the blocking rates of correct and wrong proposals (q_c, q_w) and the mean consequences of those blocks (H_c, H_w). They estimate a break-even π* on a disjoint calibration set and compare the gate with a switch-rate-matched random blocker. Experiments control proposer precision in three synthetic domains, plus a stress test on Digits in Appendix G. The authors present the contribution as a six-step operating-audit protocol.

Critical strengths
  1. Extensive and carefully run experiments, with negative results kept. Training, validation, calibration and test identifiers are disjoint, and thresholds are frozen in advance. The failed severity gate and the overturned LearnedDeferred result are also reported.
  2. The four-quantity decomposition is interpretable, and the bio counterexample is convincing. In bio, the gate blocks wrong proposals 1.85 times as often as correct ones, yet π̂* is only 0.440 because the consequences are asymmetric. This shows that the value of a gate depends on consequence size as well as on blocking rates.
Critical weaknesses
  1. The only real-data experiment (Appendix G, Digits with Gaussian corruption) uses an artificial shift, and its proposer and gate are also built for the experiment. No naturally occurring shift is tested. Overall, the paper lacks validation in a real setting.
  2. The paper does not discuss or compare with directly related work, such as threshold adjustment under label shift, Conformal Risk Control, and Learn-then-Test. The switch-rate-matched random blocker is a weak baseline. The ten-plus sub-experiments each give a local finding, and most of them only add qualifications to earlier claims. They do not lead to a new insight that carries over to other settings.
  3. The paper uses many self-coined terms and abstract concepts, e.g., "operating audit" and "refusal condition".
Questions for authors
  1. Apart from "report ranking, calibration, threshold portability and action value separately", which conclusion of the paper holds outside the authors' own generators, and which experiment supports it? Under a natural shift where class-conditional distributions are not held fixed, can the calibration-only π̂* still predict the sign of Δ on the holdout?
  2. With the same calibration budget, how does the regret of your protocol compare with thresholds chosen by Learn-then-Test or Conformal Risk Control? How does it compare with a simple rule at the same block count, e.g., blocking by evidence gap? What information does your audit give that these methods do not?
1 comment

Thank you for the specific questions; they shaped most of the new work. C1 to C7 are the pre-outcome commitments in the general response. E1 is our earlier exploratory experiment on five drifting streams (Elec2, Weather, Airlines, Rialto, Spam), hash-fixed in a file before the runs but not externally timestamped.

Natural shift, constructed update tasks. The new experiments use real observations with shift we did not construct (news headlines and arXiv titles by month, hourly bike-sharing demand, and others). The proposer and gate are ours: a retrained model, and a gate on its disagreements with the deployed model. Several streams are in file order. It is not a deployed system.

Q1a. What holds outside our generators? (i) A gate's value depends on consequence size and on how often proposals are right, not only on blocking rates, which is your reading of bio. The INSECTS gate was most selective in the two streams where it hurt (C4), and signs were opposite on HuffPost and arXiv versus covtype and poker (C1 to C3). Where the consequence-weighted and the fixed 0.5 break-even disagreed in C3 and C4 (11 of 101 identified periods, realised precision supplied), the consequence-weighted one was right in 10. (ii) Ranking and operating point separate (Q2).

Q1b. Can the calibration-only break-even predict the sign under natural shift? Not reliably. With precision estimated label-free by EM (the label-shift adjustment of Saerens et al., 2002), sign accuracy was 0.38 to 0.50 on three E1 streams and 0.86 on Rialto, and the committed prediction failed on HuffPost and arXiv (r = -0.19 and -0.42). With realised precision supplied, a fixed 0.5 threshold was as accurate as the frozen break-even on all five E1 streams.

Q2. Same calibration budget. The locked pipeline computed each method below on the same windows and sample size, on E1 and on the five committed-run streams (C3, C4). These were not pre-declared hypotheses, and the revision reports every method. Values are regret differences as a share of no-gate regret, averaged over streams, with within-stream block-bootstrap intervals conditional on the streams.

  • Same block count (your example). Blocking the learned gate's own number of proposals by evidence gap instead raised regret in all ten streams (+0.11, CI 0.07 to 0.16); at random, +0.18. At that count the gate selected a lower-regret subset than evidence gap.
  • Where to cut. Evidence gap with its own validation-chosen rate beat the learned gate (-0.073, CI -0.117 to -0.035), and so did the cost-ratio rule of Trauble et al. (-0.093, CI -0.153 to -0.034). Most of this came from our own design: the learned gate chose its rate between 5% and 95% and could not stop gating. With 0% and 100% allowed on the same validation window, the estimated gap to CostRatio shrank to -0.004 (CI -0.051 to +0.042), not an equivalence result. A similar gain appeared in previously scored E1, chiefly in Spam, and almost all of it came from choosing 0%, that is, not to gate. Beyond that option, the score's gain over a score-free choice of deploying or not was unresolved (-0.018, CI -0.043 to +0.007).
  • LTT and CRC. Our implementations target the cost-weighted harmful-switch risk at alpha = 0.25 to 0.9 times the validation no-gate risk; LTT uses Hoeffding-Bentkus fixed-sequence testing with delta = 0.1. Both may choose no blocking. Even with alpha chosen per stream in hindsight, both had higher regret than the cost-ratio rule on four of five committed-run and four of five E1 streams. Empirical test-period risk exceeded the calibration-derived target in 5% to 39% of committed-run period-alpha evaluations. These are descriptive exceedances under shift, not direct violations of LTT's guarantee, which concerns population risk under its assumptions. Risk control and regret are different objectives, and we report them separately.
  • What the audit adds. The audit chooses the blocking rate that minimises estimated regret on a labelled window, with no gate and block-all as options; its objective is regret, and it gives no risk guarantee. Its advantage over the learned gate came from those two options, not from using more recent labels (+0.002 once the options were matched). When it fails, its decomposition says which term moved: precision, consequence ratio or selectivity.

Bio. The learned gate's values were q_w/q_c = 1.85 and break-even 0.44; an evidence-gap gate reproduces the counterexample (1.56 and 0.39) and never differs from the learned gate on the 15 submitted cells after Holm correction.

Other points. The random blocker is kept only as a reference line. Sub-experiments that only qualified earlier claims move to the appendix. "Operating audit", "refusal condition" and the coined names are removed.

We would be grateful to know whether this addresses your concerns.

Reviewer_yZPFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when a learned gate should block a proposer's recommendation to switch from the current option to a previously rejected alternative option. It adopts a classical cost-sensitive decision rule to derive a break-even threshold for when blocking reduces the regret. The authors evaluate this rule in three controlled environments and conduct additional experiments on gate selectivity, calibration, and distributional shifts. The preliminary results show that the estimated threshold can predict decision outcomes in these settings, but better prediction or calibration does not always lead to lower regret.

Critical strengths

The paper includes some useful controlled experiments, especially a random blocker with the same blocking rate as the learned gate. This helps distinguish the value of learned selectivity from the effect of blocking proposals in general. The authors also report negative results rather than presenting every experiment as a success.

Critical weaknesses

The main theoretical result is a direct application of classical cost-sensitive decision theory. The main positive experiment changes proposer precision while keeping the class-conditional distributions fixed. Under this setup, the expected regret difference is already known to be linear in proposer precision. Therefore, the experiment largely confirms an expected result rather than providing new insights into learning under distribution shift. The main experiments rely on synthetic or highly controlled environments.

Each experiment addresses a limitation of an earlier result, but the paper does not develop a clear and coherent research contribution. The writing is repetitive and makes the main findings unnecessarily difficult to follow.

Questions for authors

Not much.

2 comments

We agree with the weaknesses you identified and have rebuilt the paper around them.

The theory is classical. Yes. The break-even condition is cost-sensitive decision theory, and the revision presents it as background. The claim is now empirical: when does a veto on a model update reduce realised regret, against deploying the update and against a simple cost-ratio rule?

The main experiment could only confirm linearity in proposer precision. Yes. With the class-conditional quantities fixed, the result was known in advance. The sweep is now labelled an identity check and is no longer offered as evidence about distribution shift. In the new experiments all quantities move freely, and the outcome was open: several predictions that we posted in this forum before scoring failed (general response).

Synthetic or highly controlled environments. The new evidence comes from 14 evaluable public data streams, nine of them under commitments posted here before scoring. The learned gate raised regret on some, lowered it on others, and did both within one benchmark family. None of the tested adaptive gate policies was shown to beat its relevant fixed baseline.

No coherent contribution, and repetitive writing. The revision states one question, uses consistent notation, removes the coined names and the project history, and moves the qualifying sub-experiments to the appendix. The rate-matched random blocker you found useful is kept as a reference line, beside stronger baselines.

The outcome is less favourable to our original framing than the submission was, and we report it that way.

I kindly suggest the author carefully prepare a responsible resubmission instead of dumping AI-generated responses to the general audience and reviewers.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Results of pre-outcome commitment 4 (three INSECTS streams), all outcomes

Commitment 4 (lock ec6e20aa...) was posted at 17:09 KST (08:09 UTC) on 10 November 2026; the first recorded model run began at 08:23:33 UTC. The runner checked the locked code, protocol, and three dataset hashes (including Git blob IDs) before execution. A separate rerun from the raw files in the pinned environment (Python 3.13.18, scikit-learn 1.8.0, single-threaded BLAS) produced byte-identical final JSON and 81-period CSV outputs. All 81 test periods (27 per stream) were scored, with none excluded. These three streams belong to one benchmark family and are not independent replications.

Primary prediction (M8): confirmed under the precommitted rule. Switching from the learned gate to the CostRatio-style policy when prior calibration AUROC was below 0.55 reduced regret by 0.049 of mean no-gate regret relative to the learned gate (95% CI for M8 minus learned: -0.089 to -0.014). The effect was driven by abrupt (-0.131, confirmed; 7 switches). Gradual was inconclusive (-0.017; 4 switches), and incremental did not switch (difference 0). Relative to always using CostRatio, M8 was +0.051 (CI -0.014 to +0.112). This meets the usefulness condition we stated in advance (the interval does not lie wholly above zero). That condition is weak: it does not establish noninferiority or added value against always-CostRatio, whose point estimate is better, and we make no such claim. M8 versus no gate was -0.068 (CI -0.181 to +0.039), inconclusive.

Regimes (pre-specified descriptive readout, without a predicted sign). The learned gate helped in 20/27 abrupt periods (74%) but only 5/27 gradual and 5/27 incremental periods (19% each). Learned-minus-no-gate normalized regret was -0.310 on abrupt versus +0.180 and +0.073 on gradual and incremental. Thus, opposite operating regimes occurred within one benchmark family.

R1: R1a passed its precommitted >50% sign-persistence test: the previous labelled window correctly predicted the next-period sign in 70/81 periods (86.4%, CI 79.0%-93.8%). Keeping each stream's first sign fixed achieved 64/81 (79.0%); the difference of 7.4 percentage points (CI 0.0-16.0) was unresolved. The first sign also matched the average test-effect sign in all three streams (R1c; descriptive, not an independent test). R1b passed: using no gate after a previous window in which the gate hurt (F1, used in 54/81 periods) reduced regret versus learned by 0.112 (CI 0.063-0.165) and versus no gate by 0.131 (CI 0.061-0.204). Its difference from always-CostRatio was unresolved (-0.012, CI -0.078 to +0.052); equivalence was not established.

Mechanism predictions were only partly replicated. M0 passed: grid-edge thresholds occurred in 80% of periods, versus 95% under score permutation (Holm p < 0.001), in the same direction as earlier exploratory E1 and precommitted commitment 3. M1x failed to confirm and reversed direction (median next-window AUROC 0.70 at the edge versus 0.64 in the interior; Holm p = 0.91). M2 failed to confirm (weighted Spearman -0.14; Holm p = 0.15). M3 passed: oracle-assisted H1 and the fixed 0.5 rule disagreed in 5/51 identified periods; H1 was right in 4 of those 5. We retain the dependence caveat for the edge-period tests.

Exploratory, post hoc explanation, not an independent test of the break-even identity: median test AUROC was 0.54 on abrupt versus 0.74/0.70 on gradual/incremental, while median correct-update prevalence was 0.34 versus 0.88/0.79. The exact accounting condition is log(q_w/q_c) > log(kappa), where kappa = pi H_c / ((1-pi) H_w). Median selectivity terms (0.06, 1.15, 1.13) and median consequence/base-rate terms (-0.67, 2.21, 1.80) align with the regime differences. These are descriptive medians, not a causal decomposition or a new proof. Descriptively, the gate was most selective in the two streams where it hurt, and those were the streams where most updates were correct; selectivity alone did not indicate whether the gate would pay off. In these data, the AUROC-0.55 switch improved on learned but did not demonstrate an advantage over always-CostRatio, whereas the lagged-effect-sign rule was beneficial relative to learned. We will report all precommitted successes, failures, and unresolved results in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.