acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Conformal Risk Control under Covariate Shift

Abstract

Machine learning predictors are rarely deployed on data that match the data they were calibrated on. Conformal prediction and its risk-control generalization promise distribution-free guarantees at deployment, but only under exchangeability or a covariate shift whose likelihood ratio is known exactly—and the estimated-ratio case, the one that occurs in practice, is explicitly open, out of reach of the Bayesian construction available under exchangeability. We reach it from a different starting point: pushing a Dirichlet process prior on the calibration distribution through the likelihood ratio yields the exact Bayesian posterior over the deployed risk. That posterior is exact given the weight function, which at deployment is itself estimated, so we further prove finite-sample upper and lower bounds on the risk the selected threshold actually incurs, in terms of the weight-estimation error and the conditional shift. On synthetic and real covariate shifts, in regression and classification, the realized failure rate then stays close to target where shift-blind selection fails, and what deployment receives is a posterior over the risk of every threshold rather than a single number.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal risk control, covariate shift, Bayesian bootstrap

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_L2UrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies constructing prediction sets or intervals with a guarantee that the risk is controlled at a pre-specified level under covariate shift. The goal is to control both marginally and with high probability. To achieve this, the authors propose Weighted Bayesian Conformal Prediction (WBCP), which derives the exact posterior over the deployed risk curve using Bayesian bootstrap with weighting to account for the covariate shift. This distribution is then used to extract the HDP threshold, which is claimed to satisfy the high-probability risk control requirement over this distribution. Then, WBCP chooses the deployed threshold as the larger one between the HDP threshold and a threshold obtained by weighted conformal risk control. The paper claims that the prediction sets or intervals generated by this approach satisfy both an expected risk guarantee and a high-probability risk guarantee. Experiments on real and synthetic datasets demonstrate the theoretical guarantees of the proposed method and compare its performance to several baselines.

Critical strengths
  1. The proposed extension of Bayesian bootstrapping for risk control under covariate shift is interesting and could be useful in practice. The experiments demonstrate the practicality of the proposed method.
  2. The two-sided deployment certificate of Theorem 4 is informative - it explains how the errors of the underlying components affect the risk attained by the method.
Critical weaknesses
  1. Some critical baselines do not appear in the experiments. For example, the methods of Almeida et al. (2025) and Park et al. (2022) provide high-probability risk control (or miscoverage control for Park et al. (2022)) under covariate shift, and are applicable in the framework studied in this paper. This paper must address these works and compare the proposed method to theirs. The absence of this comparison raises critical concerns about the novelty of the proposed approach and its practical advantages.
  2. I believe there is a mistake in the proof of Theorem 2. In line 1170, the authors claim that “pushing infimum inside” in equations (68)-(69) of BQ-CP’s proof “hold for any distribution on the simplex.“ This step requires that the mass law of WBCP be the same for every prior, which is not proved in this paper. The authors must settle this point: either prove why this step can be conducted, or change the method so that it holds.
Questions for authors
  1. How does WBCP perform compared to the methods of Almeida et al. (2025) and Park et al. (2022)?
  2. The relative frequency of trials whose realized risk exceeded α\alpha for WBCP with oracle weights exceeds the target 95% in some of the experiments. Can the authors verify that this frequency is bounded from above by β\beta as the number of trials increases?

References

Almeida et al., High Probability Risk Control Under Covariate Shift, 2025

Park et al., PAC Prediction Sets Under Covariate Shift, 2022

Reviewer_jimkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes weighted Bayesian conformal prediction (WBCP) for selecting risk-controlling thresholds under covariate shift with estimated density ratios . It reweights Bayesian-bootstrap draws, selects a threshold at a chosen posterior credibility level, and clamps it to the empirical weighted-risk selection. A separate finite-sample certificate relates deployed risk to weight-estimation error, the score's dependence on covariates, and conditional shift. Experiments cover synthetic regression, a multilabel count loss, PovertyMap-WILDS, and Fashion-MNIST. The reported results show substantially lower failure frequencies than the chosen baselines in several settings, at the cost of larger prediction sets or abstention.

Critical strengths
  1. The paper addresses a useful problem because deployment often differs from calibration and the density ratio is usually estimated . It also explains why a posterior derived under exchangeability cannot simply be reused under shift.
  2. The change-of-measure construction in section 3.1 is simple to sample . Normalizing weighted exponential draws gives posterior masses on the observed scores under the stated Bayesian-bootstrap model without requiring a Dirichlet approximation.
  3. The deployment certificate separates weight error, the score's dependence on covariates, and conditional shift. It's product term explains why weight error matters less when the conditional risk changes little across covariates.
  4. The experiments include useful checks beyond the main comparisons . Oracle weights, the credibility sweep and the misspecified discriminator help show where the method works and where it can fail
  5. The limitations clearly explain that the posterior takes the weight function as given and cannot account for every deployment error . The discussion of what unlabeled diagnostics cannot detect is especially useful.
Critical weaknesses

Major

  1. The posterior derivation does not yet justify the extra atom used by the algorithm . Theorem 6 gives masses on the n observed scores , whereas Eq. (6) adds a worst case atom weighted by wˉ\bar w . This affects certification and abstention, so the completed construction needs it's own justification.
  2. The distinction between posterior credibility and frequentist failure control is acknowledged but not maintained consistently. Corollary 3 is a posterior statement, while Theorem 4 bounds risk at the target plus an error budget , with confidence set by δ\delta . The oracle row in Table 1 reports 6.7% failure [6.2 , 7.2] against a 5% target.
  3. The numerical usefulness of Theorem 4 is not demonstrated at the experimental sample sizes . With n=200n=200, δ=0.05\delta=0.05 and even  B=1B=1, Lemma 9 gives ηn≈0.39\eta_n \approx 0.39, already larger than the regression risk target of 0.1. The Gaussian multilabel design also has an unbounded oracle ratio, whereas Assumption 1 requires bounded deployed weights.

Moderate

  1.  The threshold used for the claimed inherited marginal guarantee is unclear. Eq. (1) omits the test-point correction used by weighted conformal prediction, while Appendix C.1 describes the W-CRC baseline as transductive. These are not the same definition.
  2. The interpretation of σpost\sigma_{\mathrm{post}} relies on a result for a different posterior model. Theorem 16 assumes a Dirichlet surrogate, whereas Algorithm 1 samples normalized weighted exponentials. Appendix D.4 says the surrogate is conservative at small effective sample sizes, but does not establish that connection.

Minor

  1. The practical results would be easier to assess with consistent reporting of certificate availability. PovertyMap retains 747 of 2,600 re-splits after screening, and Table 2 conditions its small-sample failure rates on certified trials. Low conditional failure and frequent abstention describe different parts of the deployment trade-off.
Questions for authors
  1. Please justify the additional atom in Eq. (6) under the stated posterior model, or identify it as an additional modeling assumption. Theorem 6 and Algorithm 1 use different numbers of atoms, so the exactness claim should refer to the object actually sampled.
  2. Please state the strongest frequentist claim supported for the credibility level β\beta. Corollary 3 and Theorem 4 concern different probability statements, and the oracle result in Table 1 exceeds nominal failure. The wording should not imply that these results establish exact 5% failure control at the original risk target.
  3.  Please tabulate the weight bound BB and concentration term ηn\eta_n for the settings where Theorem 4 is applied. This would distinguish informative certificates from loose bounds or empirical tests outside Assumption 1, particularly in the Gaussian multilabel design.
  4. Please give one precise definition of the threshold λ^\hat\lambda used in Algorithm 1 and connect it to the claimed inherited guarantee. Eq. (1) lacks a test-point mass, while Appendix C.1 describes a transductive W-CRC baseline.
  5. Please establish how the uncertainty interpretation of σpost\sigma_{\mathrm{post}} carries over to the exact sampler. Theorem 16 assumes the Dirichlet surrogate, so the small-effective-sample claim needs a proof or a direct comparison between the two laws.
  6. Please report certificate availability alongside conditional failure rates. Separating overlap-screen rejection from failure to reach the requested credibility would clarify the small-sample and PovertyMap results without treating abstention as either a success or a violation.
1 comment

Thank you for the detailed responses. The authors have addressed my main concerns through substantial clarifications and revisions. They corrected the oracle calculation, withdrew unsupported claims and inconsistent statistics, narrowed the theoretical guarantees, and improved reporting of certificate availability. Therefore, I increase my score.

Reviewer_4KdKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends the Bayesian conformal risk control framework to a setting with convariate shifts. A Dirichlet prior is imposed on the calibration law and yields exact weighted Bayesian bootstrap. The cutoff λ\lambda is chosen as the smallest λ\lambda such that the probability of the posterior loss being controlled at α\alpha given the calibration set is at least the credibility parameter β\beta. The authors also provide a finite sample lower bound and upper bound of the test risk at the chosen λdep\lambda_{\mathrm{dep}}. They provide experiments over synthetic regression and real datasets.

Critical strengths
  • The novelty of the paper is clear. It extends BQ-CP to be more applicable to covariate shifts where the calibration and test points are not exchangeable.
  • The method can provide posterior distributions over the test risk at user-chosen loss level α\alpha and credibility β\beta, and provides a simple calibration method to choose λHPDβ\lambda_{HPD}^{\beta}.
  • The derivation of the posterior distribution and the finite sample two-sided bounds of the test risk are clear.
Critical weaknesses
  • The only real competitor that accounts for covariate shift is W-CRC, which controls risk in expectation and has no failure-rate parameter (like β\beta). Comparing failure frequencies favors the proposed method by design. Also, the calibrated sets of WBCP are consistently larger than W-CRC. The benefits of WBCP against W-CRC should be properly addressed under fair comparison.
  • Comparison against more baselines that account for covariate shifts (such as localized CP (Barber et al. (2023), also cited in paper) should be conducted.
  • The practical informativeness of the bounds on Rtest(λ)R_{\mathrm{test}}(\lambda) (Theorem 4) should be further investigated. Reporting the empirical bounds evaluated on real experiments and its comparison of α\alpha should be reported.
  • (Several notations and objects are hard to interpret and should be made more clear for better understanding. These include:
    • what L+(w)L^+(w) represents
    • undefined base measure G0G_0
    • how α\alpha and β\beta are chosen in practice and how they should be interpreted
Questions for authors
  • How often does λ^\hat \lambda exceed λHPDβ\lambda_{\mathrm{HPD}}^\beta across experiments? How do results of WBCP change without the max⁡\max operator on these terms?
  • Could a more informative prior help when the effective sample size is small or the covariate shift is large?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.