acceptodds
Withdrawn from ICLR 2027

Sharp Selective-Risk Envelopes under Normalized Density-Ratio Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selective prediction controls error by abstaining on uncertain inputs, but an operating point calibrated on source data can lose its guarantee under distribution shift. We characterize the exact worst-case selective risk under a normalized joint density-ratio bound, . For binary loss, it depends only on the source coverage and the accepted error rate. The familiar error-odds envelope is sharp precisely when rejected examples can absorb the normalization imbalance; at high coverage, it can discard valid operating points. We derive a closed-form, coverage-dependent error target and use it in , which combines independent score-only coverage estimation with exact binomial tests and Holm correction. Its certificate remains valid after threshold selection, its certified set contains those of the matched odds-envelope and Bonferroni rules, and it recovers every threshold that clears an explicit estimation margin. Independent linear programs match the population formulas to within on 2,220 instances. In a controlled numerical study with fully specified score–loss distributions, the matched sharp and odds rules coincide in a low-coverage regime, whereas in a high-coverage regime the sharp rule raises mean worst-case coverage from to with no observed risk violations. These results identify when normalization materially changes calibration and when the simpler odds envelope suffices.

Primary area:
learning theory
Keywords:
Selective Prediction, Distributionally Robust Risk Control, Finite-Sample Guarantees

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pXFsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies selective prediction under normalized joint density-ratio shifts. Its main contribution is an exact worst-case binary selective-risk formula determined by source coverage and accepted error rate.

Critical strengths

Its main contribution is an exact worst-case binary selective-risk formula determined by source coverage and accepted error rate.

Critical weaknesses

The manuscript is difficult to follow due to an overabundance of notation introduced without clearly motivating or defining the primary target function. The authors aim to maximize $C_{\Gamma}(a_{\tau}) $ subject to a worst-case error constraint, which is counterintuitive; in selective prediction, abstention serves primarily as a safety mechanism, making the maximization of this specific expectation questionable. Furthermore, their formulation of the worst-case error critically omits the predictor ff. On the theoretical side, the paper fails to establish convergence rates. Empirically, the numerical experiments lack comparisons with established baselines, despite existing literature in this area. The authors also fail to evaluate their method on real-world datasets, which is highly problematic as the assumed robustness conditions may not hold in practice. Finally, the paper lacks crucial guidance on how to select the parameter Γ\Gamma.

Questions for authors

None

Reviewer_yZbPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper considers settings in which we observe predictions from a model along with confidence scores estimating the model's accuracy. It then develops methods for thresholding these scores in order to maximize the number of accepted predictions while controlling the frequency at which an accepted prediction is erroneous. The paper considers settings in which the training and test data come from two different distributions whose likelihood ratio lies in the bounded interval [1/Γ,Γ][1/\Gamma, \Gamma] and develops methods that are robust to this distribution shift.

Critical strengths

The problem considered in this paper is interesting and worthy of investigation.

Critical weaknesses
  • This paper is quite poorly written and largely consists of dense definitions with minimal motivation or interpretation. In my view, even if the content of this paper was strong (which unfortunately I don't think it is), in the current research landscape it should be rejected for this reason alone.
  • The theoretical results in this paper are not particularly surprising and appear to mostly involve standard calculations. The multiple methods given for finite sample calibration are especially redundant and well known.
Questions for authors

I have no questions for the authors.

Reviewer_4p5HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the problem of selective prediction under a reweighing based distribution shift between deployment data and calibration data. The paper identifies that under unknown reweighting of the source population for given normalized two-sided density ratio bounds, naive error-odds rules and odds thightness do not hold up. Hence, the authors theoreticaly analyse the selective risk, and coverage guarantees under such scenario. The paper’s main study analyses worst-case risk for binary risk in Thm 1. In Proposition 1, it innvestigates the tightness of the naive odds rule and derive conditions for it's validity. Corollary 1 tells us how much error can we afford at a given coverage p while Proposition 2 says how much coverage is guaranteed under adversarial distribution. Thm 2 highlights the spread of the losses instead of just their means. Thm 3 provides a safe certificate by providing an upper bound on the error rate and lower bound on the coverage. Thm 4 and Prop 3 first combine a DKW coverage bound from an independent score-only sample, exact binomial tests and Holm correction. Then the paper shows it is valid after threshold selection, contains the matched odds and Bonferroni certified sets, and recovers every threshold that clears an explicit margin. The numerical evaluations suggest that the sharp and odds rules coincide in a low coverage regime while in that of high coverage, the sharp rule seems to increase mean worst-case coverage without violating risk.

Critical strengths
  • The problem is characterised well. Thm 1 and Eq 9 in Prop 1 provide an explicit and clean regime for when the error odds envelope is tight and how naive rule fails to maintain high coverage as the rejected samples cannot absorb the imbalance.
  • The authors prove that the HOLM-SHARP-RC is valid after selecting threshold based on the data and that on the same data, its certified set entails equivalent odds-envelope and Bonferroni rules. Moreover, Proposition 3 further provides an oracle-recovery guarantee.
  • The paper provides a clear regime where the sharp envelope stops giving gains and separates the envelope effect from the testing effect in a comparable 2x2 design.
  • The paper reports almost all configurations (seeds, exact population, LP checks) strengthening the reproducibility of the results.
Critical weaknesses
  • The error odds envelope is assumed to be familiar but there is no reference or comparison provided to any existing selective prediction or conformal procedure that uses naive odds rule.
  • Being a majorly theoretical work, the main contribution seems to be the limited to the closed form solution of a three-cell linear-fractional program and its use as a calibration target. This is incremental given that the event bounds are borrowed from Jin et al. (2023) and the testing tools are standard as noted by the authors.
  • The paper claims in the introduction that the procedure applies to classifiers, verifiers and generative models alike, but there is no evidence or evaluations for any of these models.
  • While the paper aims to address covariate shift, there is no explicit theorem, result or experiment evaluating it. It is only considered as a special case and has looser bounds.
  • The theoretical gain obtained seems to be narrow. It is clear from eq 9 that the improvement occurs only when p(1−q)>Γ/(1+Γ)p(1-q) > \Gamma/(1+\Gamma). Even then, the error target relaxes from α/Γ2\alpha/\Gamma^2 to α/Γ\alpha/\Gamma at full acceptance i.e. at most by a factor of Γ\Gamma. Is there a way to provide evidence that the practical operating points would actually fall in this regime?

Experiments:

  • The numerical evaluations are limited to one synthetic score loss family of uniform scores with quadratic error curve with two choices of cc for each regime.
  • It is straightforward to evaluate trained predictors on existing benchmarks like CIFAR-10 to CIFAR-10-C, ImageNet to ImageNet-v2/WILDS datasets. This would directly help improve the applicability of the procedure to real world scenarios.

Writing:

  • The abstract is hard to follow and assumes the reader to have certain context of the paper that is clarified in the later sections, possibly due to heavy AI assistance in writing it, which in itself is not a negative unless it makes the reading harder rather than easier to follow . Terms like ‘Γ2\Gamma^2 error-odds envelope’, ‘rejected examples can absorb the normalisation imbalance’, ‘explicit estimation margin’, ‘matched odds-envelop rule’, ’score-only coverage estimation’ require context that makes sense after reading details later in the paper.
  • I think a notation table would help keep up with the symbols introduced. Certain symbol representations are conflicting. E.g. cc denotes both the rejected mass (A.1) and the error-curve parameter (Eq. 24), CC denotes both the certificate and CΓC_\Gamma.

Overall, the paper is technically sound, but the insights are somewhat narrow and limited in scope. Evaluating pretrained models or at least one of the models that the authors claimed in the introduction (end of pg1) on real world shift or show that the realistic shifts also operate in the regime where sharp envelop helps would help strengthen the contributions of the paper.

Questions for authors
  • Which existing selective-prediction or conformal methods calibrate with ρΓ\rho_\Gamma?
  • Is there a way to determine Γ\Gamma? The certificate seems to rely on it but the authors do not suggest a way to identify/choose/estimate it from the source data.
  • Under pure covariate shift, can we get a tighter certificate from Theorem 2 with Hr=a(e(X)−r)H_r = a(e(X)-r) using a calibrated estimate of e(X)=P(ℓ=1∥X)e(X) = P(\ell=1 \| X)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.