- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies constructing prediction sets or intervals with a guarantee that the risk is controlled at a pre-specified level under covariate shift. The goal is to control both marginally and with high probability. To achieve this, the authors propose Weighted Bayesian Conformal Prediction (WBCP), which derives the exact posterior over the deployed risk curve using Bayesian bootstrap with weighting to account for the covariate shift. This distribution is then used to extract the HDP threshold, which is claimed to satisfy the high-probability risk control requirement over this distribution. Then, WBCP chooses the deployed threshold as the larger one between the HDP threshold and a threshold obtained by weighted conformal risk control. The paper claims that the prediction sets or intervals generated by this approach satisfy both an expected risk guarantee and a high-probability risk guarantee. Experiments on real and synthetic datasets demonstrate the theoretical guarantees of the proposed method and compare its performance to several baselines.
Critical strengths
- The proposed extension of Bayesian bootstrapping for risk control under covariate shift is interesting and could be useful in practice. The experiments demonstrate the practicality of the proposed method.
- The two-sided deployment certificate of Theorem 4 is informative - it explains how the errors of the underlying components affect the risk attained by the method.
Critical weaknesses
- Some critical baselines do not appear in the experiments. For example, the methods of Almeida et al. (2025) and Park et al. (2022) provide high-probability risk control (or miscoverage control for Park et al. (2022)) under covariate shift, and are applicable in the framework studied in this paper. This paper must address these works and compare the proposed method to theirs. The absence of this comparison raises critical concerns about the novelty of the proposed approach and its practical advantages.
- I believe there is a mistake in the proof of Theorem 2. In line 1170, the authors claim that “pushing infimum inside” in equations (68)-(69) of BQ-CP’s proof “hold for any distribution on the simplex.“ This step requires that the mass law of WBCP be the same for every prior, which is not proved in this paper. The authors must settle this point: either prove why this step can be conducted, or change the method so that it holds.
Questions for authors
- How does WBCP perform compared to the methods of Almeida et al. (2025) and Park et al. (2022)?
- The relative frequency of trials whose realized risk exceeded for WBCP with oracle weights exceeds the target 95% in some of the experiments. Can the authors verify that this frequency is bounded from above by as the number of trials increases?
References
Almeida et al., High Probability Risk Control Under Covariate Shift, 2025
Park et al., PAC Prediction Sets Under Covariate Shift, 2022
Thank you for the detailed responses. The authors have addressed my main concerns through substantial clarifications and revisions. They corrected the oracle calculation, withdrew unsupported claims and inconsistent statistics, narrowed the theoretical guarantees, and improved reporting of certificate availability. Therefore, I increase my score.