acceptodds
Under review as a conference paper at ICLR 2027

Towards Robust and Efficient Conformal Prediction Under Covariate Shift

Abstract

Conformal prediction provides model-agnostic prediction sets with nominal coverage, enabling rigorous uncertainty quantification for machine-learning predictions. A major challenge in conformal prediction is covariate shift, where source and target covariate distributions differ while the conditional outcome distribution remains unchanged. Existing methods often fail to balance robustness and efficiency: weighted conformal prediction (WCP) may yield excessively wide or infinite sets under severe covariate shift, whereas doubly robust prediction (DRP) may substantially undercover because it relies on a single pair of component-model estimates whose finite-sample errors can reinforce one another. We propose doubly propagated calibration (DPC) to attain nominal target coverage while keeping prediction sets informative. DPC repeatedly regenerates the DRP component-model estimates across data splits and propagates the resulting finite-sample estimation uncertainty through downstream calibration. Within each regeneration run, DPC balances source and target samples on estimated coverage probabilities and then takes the union of candidate sets across runs, so that one reliable candidate can protect final coverage. Across severe-shift simulations and California Housing experiments, DPC attains near-nominal 90% coverage in settings where DRP substantially undercovers, with intervals about 40% narrower than WCP in simulations and widths less than 9% above the oracle benchmark on California Housing.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal prediction, Covariate shift, Doubly propagated calibration, Uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_U3RDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces doubly propagated calibration (DPC), a doubly robust conformal prediction algorithm mitigating the finite-sample errors in nuisance fitting. The DPC method appears to be built on the doubly robust prediction (DRP). The proposed method generally attains the nominal coverage (while slightly conservative but has great improvement compared to existing benchmarks) in simulation settings compared.

Critical strengths
  1. The paper's formulation is excellent: from reviewing existing approaches briefly, pointing out key limitations and gaps, to introducing the proposed approaches gradually and clearly.

  2. Presentation is helpful, e.g., Figure 1 at the beginning is convincing and helpful for understanding the question and scope. Figure 2 is intuitive and highly summarized. Algorithm 1 is helpful for my understanding.

  3. Theoretical investigations are deep. Experiments are comprehensive and informative.

Critical weaknesses

Major comment

How does the method compare to some intermediate version of DRP to DPC. For example, using cross-fitted estimator in DRP instead of plug-in. Is their any optimality guarantee (especially in terms of efficiency) by the proposed DPC.

Minor issues

  1. Please do not let any formulas, tables, figures or text out the specified margin limit. This cannot pass the final camera-ready check. Equations (2), (24), (42), (98), (118) etc. require revisions. Please check all lines in the paper. Enabling "grid view" in pdf viewer tool could be helpful.

  2. "Existing methods often fail to balance robustness and efficiency" is a relatively strong and absolute opinion. They "can" but may not be said as "often". WCP may fail under severe covariate shifts, but sufficient overlap is an underlying assumption of this method, so that it is not so fair to claim this method often fails.

  3. Could the authors highlight the novelty beyond DRP (Yang et al; 2024) in more depth? For example, Step 2 might be something their work cannot handle, if I understand correctly. These novel algorithms/steps are not naturally possible choices under Yang et al.'s framework, but require their own developments. In addition, you may highlight one advantage of adopting Yang et al. (2024)'s equation, besides doubly robust, is it attains the semiparametric efficiency bound under correctly specified nuisance models. Your algorithm further increases the flexibility of the nuisance fitting, so it may help improving efficiency towards the efficiency bound.

  4. (Optional). I know this suggestion might be a little personal preference, but I feel some simulation data generation details are not necessary for the main text readability, which could be moved to appendix, and just summarize in words what does each scenario mean.

  5. Theorem 3.1 is the key of this paper. It requires some remarks in the paper about its practical implications.

Questions for authors
  1. You mentioned in the end of Section 2, DRP removes nuisance-induced bias, but not sampling variability. Did you mean their original algorithm didn't use sample-splitting? Is this related to using plugging-in estimator or some other reasons? If nuisances are trained in a training set, why that is still a plug-in?

  2. So is your final estimator still using the influence function (or empirical moment equation) in (2), found by Yang et al. (2024)? The main change is the implementation details?

  3. Is the slightly conservative performance in simulation expected rather than by chance? Is that an asymptotic behavior or just finite sample?

Reviewer_aBoYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses an important issue of covariate shift in the conformal prediction framework through the proposed DPC (doubly propagated calibration). DPC overcomes the limitations of doubly robust prediction: reliance on a single pair of component-model estimates. The main idea is to use a collection of such estimates and then construct conservative union sets that are guaranteed to satisfy the asymptotic conditional coverage guarantee (Theorem 3.1). Simulations demonstrate the effectiveness of the proposed DPC over DRP and WCP.

Critical strengths
  1. DPC proposes to use multiple generations to capture variability linked to estimates of density-ratios and conditional score-CDF, and then a union strategy to construct final sets that satisfy asymptotic coverage guarantee (Theorem 3.1). The ideas of using multiple runs and the union set strategy are easy to implement and hence practically appealing.
Critical weaknesses
  1. The union strategy can produce overly conservative prediction sets. The paper does not clarify how their step1 differs from DRP within each generation. Since all M generations are parallel(?), they are only mixed using the max strategy in step 3. It is not evident if there is any efficiency gains in general. I don't think the issue that motivated the paper- robust and efficient prediction sets is thoroughly addressed through theoretical claims in the paper. In my opinion, the step2 tilting is what results in better threshold calibration, and that's what leads to a target coverage guarantee. I am not convinced that the role of M generations with union at the end is the main contributor to the performance of DPC, as the paper suggests.

  2. The simulations should be broad enough to isolate the different effects of the three stages and provide clarity on M=1 vd large M generations.

  3. The trade-offs between informativeness and coverage constraints are not well-explored in the experimental section. Theory does not guarantee the informativeness of the DPC prediction sets. What situations might lead to non-informative prediction sets is not discussed.

Questions for authors
  1. Within each generation, how does DPC step1 differ from earlier approaches such as DRP? Because step3 then constructs conservative sets by taking unions, how does this step lead to informative prediction sets? In simulations, why don't you have M=1 case for DPC, which will help to distinguish where is the improvemnt coming from the the whole process.

  2. Does Theorem 3.1 say anything about how to choose M? In line 323, is it suggested that M should be very large? How does this matter for the informativeness of the prediction sets?

Reviewer_v2eVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Doubly Propagated Calibration (DPC) for conformal prediction under covariate shift, addressing the overly wide prediction sets of weighted conformal prediction (WCP) and the potential undercoverage of doubly robust prediction (DRP). DPC repeatedly regenerates nuisance estimators, applies exponential tilting to balance score-CDF features, and takes the union of candidate prediction sets. The paper establishes theoretical coverage guarantees under certain conditions and demonstrates improved coverage-efficiency trade-offs through simulations and California Housing experiments.

Critical strengths
  1. The paper proposes Doubly Propagated Calibration (DPC) to overcome finite-sample sensitivity in doubly robust calibration, combining repeated nuisance regeneration with union-based aggregation to propagate estimation uncertainty, alongside coverage-oriented exponential tilting to mitigate transport discrepancies via conditional score-CDF balancing. This methodological design is backed by a comprehensive theoretical analysis that establishes probabilistic coverage guarantees, interprets the tilting step as a KL information projection, and specifies the conditions under which tilting enhances candidate calibration reliability.

  2. Across multiple simulation settings and the California Housing dataset, DPC substantially improves coverage over DRP while producing considerably narrower prediction sets than WCP. The ablation studies further demonstrate the contributions of regeneration and exponential tilting.

Critical weaknesses
  1. Theorem 3.1 relies on Condition A.5, which assumes a positive conditional probability of obtaining a successful candidate without establishing such a bound for the proposed 20% subsampling procedure. Proposition A.2 additionally requires a bounded oracle density ratio, excluding the nonzero Gaussian mean-shift settings considered in Section 4.1. Condition A.1 assumes uniformly bounded fitted weights, yet the exponential tilting procedure does not explicitly enforce this constraint. Moreover, exact balancing may be infeasible, but Algorithm 1 provides no fallback. Most importantly, taking the maximum over candidate thresholds can amplify extreme or infinite values, while the theoretical results provide no corresponding upper bound on the final interval width. Consequently, the coverage guarantees do not establish when DPC produces finite and informative prediction intervals under severe covariate shift.

  2. The abstract states that DPC achieves interval widths less than 9% above Oracle on California Housing, whereas Table 2 reports widths approximately 32% larger in the hardest settings (4.36/3.29, 4.43/3.36, and 4.86/3.66). Section 4.1 and Appendix D.1 also specify inconsistent mean-shift grids, while Figure 3 describes mean absolute threshold error despite plotting mean threshold shortfall. Furthermore, the main propagation comparison uses (M=50) for propagation-only and (M=20) for DPC, whereas Table 2 uses (M=60) without a clear selection rationale. Although some matched-(M) ablations are provided, systematic comparisons under equal computational budgets, together with coverage-width curves, interval-width variability, and runtime measurements, are still missing. These limitations make it difficult to determine whether the reported improvements arise from more effective calibration or simply more conservative threshold selection.

Questions for authors
  1. Theorem 3.1 relies on Condition A.5, which assumes a positive conditional probability (p_{\varepsilon_n}) of obtaining a successful candidate. Can the authors provide sufficient conditions or an explicit lower bound for this probability under the proposed 20% subsampling scheme and nuisance learners? In particular, how does the reduced fitting sample size affect the likelihood of obtaining a finite, rate-controlled candidate? Furthermore, Proposition A.2 assumes a bounded oracle density ratio, which does not hold for the nonzero Gaussian mean-shift settings in Section 4.1. Which theoretical guarantees apply to these experiments?

  2. Since the final threshold is defined as (\widehat{\theta}{\mathrm{DPC}}=\max{1\le m\le M}\widehat{\theta}_m^{\mathrm{tilt}}), increasing (M) can improve coverage but also increases the risk of selecting an excessively large threshold. Can the authors establish an upper bound on the final threshold or its deviation from the oracle threshold under suitable conditions? Condition A.3 additionally treats infinite thresholds as successful candidates, despite their lack of informativeness. Could the authors investigate how interval-width distributions, extreme thresholds, and the frequency of infinite intervals change with (M), rather than reporting only mean widths?

  3. The tilting procedure uses (K_m=31) score-CDF features, which may be highly correlated, particularly with limited calibration samples. Appendix A.2 acknowledges that exact balancing may be infeasible, but Algorithm 1 does not specify how such cases are handled. Does the implementation use regularization, weight clipping, approximate balancing, or a fallback to propagation-only calibration? Can the authors report optimization failure frequencies, balance residuals, maximum tilted weights, and effective sample sizes? It would also be useful to clarify whether these practical modifications preserve the assumptions underlying the theoretical guarantees.

  4. The abstract claims that DPC achieves interval widths less than 9% above Oracle on California Housing, whereas Table 2 reports gaps of approximately 32% in the hardest settings. How was the 9% figure obtained? Moreover, Section 4.1 specifies mean-shift magnitudes ({0,1,\ldots,6}), while Appendix D.1 uses ({0,2,\ldots,20}). Which grid generated the reported simulations? Figure 3 also describes mean absolute threshold error in its caption but labels the axes as mean threshold shortfall; which quantity was actually computed? Finally, Algorithm 1 calculates a baseline DRP threshold in Line 10 that is not used in the returned DPC threshold. Was this baseline included in the experimental implementation?

  5. Figure 3 compares propagation-only calibration with (M=50) against full DPC with (M=20), while Table 2 uses (M=60). Could the authors provide paired comparisons at matched (M) and comparable computational budgets, reporting coverage, interval width, variability, and runtime? How were (M), the 20% subsampling rate, and (K_m) selected without using target evaluation outcomes? Additionally, since the experiments primarily use absolute-residual scores, could the authors evaluate DPC with locally adaptive scores such as CQR and on at least one additional real-world covariate-shift benchmark to assess whether the efficiency gains generalize beyond constant-width prediction intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.