acceptodds
Under review as a conference paper at ICLR 2027

Conformal Prediction under Distribution Shift: False Coverage Proportion Guarantees

Abstract

Conformal Prediction (CP) is a distribution-free and model-agnostic framework for uncertainty quantification that constructs prediction sets with rigorous finite-sample coverage guarantees. Despite these advantages, the classical CP guarantee is marginal and therefore does not directly control the realized proportion of miscovered observations across a test sample. This has motivated the study of the False Coverage Proportion (FCP) in the literature, which provides a stronger form of coverage assessment over multiple predictions. Nevertheless, both classical CP and existing FCP theory typically rely on exchangeability between calibration and test data, an assumption that may fail under calibration-to-test distribution shift. Recent works have proposed Weighted CP to address this issue for classical CP by reweighting calibration observations to account for such distribution shifts. However, existing FCP theory does not directly extend to this weighted setting and has largely remained restricted to exchangeable data, leaving FCP control under distribution shift substantially unexplored. Therefore, in this work, we establish finite-sample FCP guarantees for Weighted CP under calibration-to-test distribution shift. We further study the inverse FCP problem of selecting a miscoverage level to achieve a prescribed target FCP. For practical implementation, we also develop data-driven procedures for estimating the quantities appearing in our bounds and establish their convergence. Empirically, we validate our theoretical guarantees across multiple classification and regression datasets under both classical Covariate Shift and more challenging one-phase shifts. We further investigate the large-sample behavior of the proposed bounds through asymptotic studies.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal Prediction, False Coverage Proportion, Distribution Shift, Covariate Shift, Weighted Conformal Prediction, Finite-Sample Guarantees, Uniform Coverage Control, Uncertainty Quantification.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fq9DOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides analysis of False Coverage Proportion (FCP) with distribution shift. Theoretical and empirical evidence are shown to validate the results.

Critical strengths

This paper is easy to read, and rather complete. The authors provide both empirical and theoretical evidence to validate their claims.

Critical weaknesses

Unfortunately, I find this paper's contribution a little incremental. The line of FCP is widely explored, as cited by this paper. And conformal prediction under covariate shift is also well-explored. The techniques of this paper directly follow those tricks, if I may say so. Therefore, I cannot find noval contributions of this paper.

Besides, the writing of this paper is somehow confusing. I do not understand how section 3 contributes to this community, since breaking exchangeability leads to the failure of the bound is somehow expected. I do not think it deserves a whole section to validate this. Some other presentations are similar. For example, does the formula in Line 258 is rather important? I think use the in-line formula is enough. Overall, I think the writing needs a polish.

Questions for authors

Could the authors show the challenges on combining the FCP and distribution shift tricks in conformal prediction, either theoretically or empirically?

6 comments
AuthorsOpenReview ↗

Reply [1/4]

W1. Unfortunately, I find ... contributions of this paper.

Q1. Could the authors show the challenges on combining the FCP and distribution shift tricks in conformal prediction, either theoretically or empirically?

We thank the reviewer for this important observation and the opportunity to clarify our theoretical contributions. We acknowledge that both False Coverage Proportion (FCP) and Weighted Conformal Prediction (CP) are well-established research directions. However, we respectfully disagree that establishing finite-sample FCP guarantees under Distribution Shift follows directly from combining existing techniques. Although these frameworks are conceptually related, their existing theoretical results do not directly resolve the key mathematical difficulties arising when calibration and test observations follow different distributions. Our contribution lies in overcoming these difficulties to establish explicit finite-sample guarantees, including uniform and inverse FCP control, under general One-Phase Distribution Shift (OPDS).

1. Why existing approaches are insufficient.

We first distinguish our contributions from three closely related lines of research.

  • Existing FCP theory [1, 2]. These works establish finite-sample, high-probability control of the realized FCP under exchangeability. Their guarantees exploit the joint structure of conformal p-values when calibration and test scores share the same distribution. Under Distribution Shift, this structure is generally lost, and the resulting guarantees need not remain valid. Indeed, our experiments in Section 3 (Figure 1) demonstrate that applying these bounds under Covariate Shift can result in empirical guarantee probabilities substantially below the nominal level, sometimes reaching zero.

  • Weighted FCP under Distribution Shift [3]. This is the closest related work, as discussed in Appendix A.2. Gazin [3] studies weighted FCP under Distribution Shift, including oracle and non-oracle weights, through functional central limit theorems characterizing the limiting FCP process. While these results provide an important asymptotic understanding of how distribution shifts affect conformal errors, weak convergence does not establish finite-sample probability guarantees. In particular, it does not provide an explicit non-asymptotic error term controlling the deviation of the realized FCP at given calibration and test sample sizes (n,m)(n,m). Obtaining such guarantees requires a separate concentration analysis, rather than directly applying the asymptotic results.

  • PAC and training-conditional coverage under Covariate Shift [4]. These results control the conditional miscoverage probability under assumptions relating the calibration and test distributions, particularly the preservation of P(Y∣X)P(Y\mid X) and suitable importance weighting. We agree that such conditional-coverage bounds can be combined with test-batch concentration to obtain FCP guarantees within their respective settings. However, our Assumptions 2.2 and 2.3 are substantially more general: they permit changes in P(Y∣X)P(Y\mid X), do not require oracle density ratios, and do not assume that the weighted calibration-score distribution matches the test-score distribution. Consequently, the existing conditional-coverage guarantees cannot be directly applied.

The last distinction illustrates the central theoretical difficulty. Let

qn(α):=Ptest(pw,(n)≤α∣Dcal)q_n(\alpha):=\mathbb P_{\mathrm{test}}\left(p^{w,(n)}\leq\alpha\mid\mathcal D_{\mathrm{cal}}\right)

denote the calibration-conditional miscoverage probability. Since test observations are independent given the calibration data, a standard Hoeffding argument yields

P(FCPmw,(n)(α)≤qn(α)+εtest∣Dcal)≥1−δtest.\mathbb P\left(\mathrm{FCP}^{w,(n)}_m(\alpha)\leq q_n(\alpha)+\varepsilon_{\mathrm{test}}\mid\mathcal D_{\mathrm{cal}}\right)\geq1-\delta_{\mathrm{test}}.

However, this only controls the realized FCP around the unknown conditional probability qn(α)q_n(\alpha), not around the nominal miscoverage level α\alpha. Under general OPDS, these quantities may differ substantially. For example, consider unit weights with calibration scores S∼Unif(0,1)S\sim\mathrm{Unif}(0,1) and test scores T∼Unif(0.5,1.5)T\sim\mathrm{Unif}(0.5,1.5). This setting satisfies Assumptions 2.2 and 2.3, yet for α=0.1\alpha=0.1, as n→∞n\to\infty,

qn(0.1)⟶P(T≥0.9)=0.6.q_n(0.1)\longrightarrow\mathbb P(T\geq0.9)=0.6.

Thus, the conditional miscoverage probability approaches 0.60.6, rather than the nominal level 0.10.1. No improvement in test-batch concentration alone can resolve this discrepancy, because the difficulty arises from the systematic difference between the underlying score distributions, not from sampling fluctuations within the test batch. This demonstrates that establishing FCP control under general OPDS requires an additional theoretical ingredient: a finite-sample characterization of qn(α)q_n(\alpha) that explicitly accounts for Distribution Shift.

AuthorsOpenReview ↗

Reply [2/4]

2. How our work overcomes these difficulties.

Our approach addresses this problem by introducing a shift-dependent population function and establishing explicit high-probability calibration-to-test inequalities.

Let an=w∞/na_n=w_\infty/n and μw=Ecal[w(X)]\mu_w=\mathbb E_{\mathrm{cal}}[w(X)]. Define the weighted empirical calibration distribution and its population counterpart as

F^w,n(t)=1n∑i=1nw(Xi)1Si≤tan+1n∑i=1nw(Xi), and Fw,n(t)=Ecal[w(X)1S≤t]μw+an.\widehat F_{w,n}(t)=\frac{\frac1n\sum_{i=1}^{n}w(X_i)\mathbf1\\{S_i\leq t\\}}{a_n+\frac1n\sum_{i=1}^{n}w(X_i)},\text{ and }F_{w,n}(t)=\frac{\mathbb E_{\mathrm{cal}}[w(X)\mathbf1\\{S\leq t\\}]}{\mu_w+a_n}.

We then introduce the population function

Gw,n(u):=Ptest(Fw,n(T)≥1−u),G_{w,n}(u):=\mathbb P_{\mathrm{test}}\left(F_{w,n}(T)\geq1-u\right),

which explicitly captures the relationship between the weighted calibration-score distribution and the test-score distribution. Unlike the nominal level α\alpha, this quantity remains meaningful even when the weighting function does not perfectly correct the shift.

The essential technical challenge is to establish a finite-sample comparison between the random weighted empirical distribution F^w,n\widehat F_{w,n} and its population counterpart Fw,nF_{w,n}. This is not a direct application of the standard conformal rank argument: the calibration distribution is self-normalized, with a random numerator and denominator constructed from the same observations, and the resulting probability must subsequently be evaluated under a potentially different test distribution.

Our analysis overcomes this difficulty in two complementary ways.

Fixed-level control (Theorem 4.1). For a prescribed α\alpha, we construct an appropriate population threshold τα\tau_\alpha and consider

Zi=w(Xi)[1Si≤τα−(1−α)].Z_i=w(X_i)\left[\mathbf1\\{S_i\leq\tau_\alpha\\}-(1-\alpha)\right].

By exploiting the bounded range of these weighted variables, we establish a concentration inequality that directly controls the relevant self-normalized comparison, obtaining the explicit error

Δfix=εwμw+anlog⁡(1/δcal)2n.\Delta_{\mathrm{fix}}=\frac{\varepsilon_w}{\mu_w+a_n}\sqrt{\frac{\log(1/\delta_{\mathrm{cal}})}{2n}}.

The resulting calibration-conditional inequality is

Pcal(qn(α)≤Gw,n(α+Δfix))≥1−δcal.\mathbb P_{\mathrm{cal}}\left(q_n(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{fix}})\right)\geq1-\delta_{\mathrm{cal}}.

This is the crucial step that is not directly provided by existing Covariate Shift PAC guarantees under our general OPDS assumptions. Combining it with standard concentration over the independent test batch yields

P(FCPmw,(n)(α)≤Gw,n(α+Δfix)+εfix)≥1−δcal−δtest.\mathbb P\left(\mathrm{FCP}^{w,(n)}_m(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{fix}})+\varepsilon_{\mathrm{fix}}\right)\geq1-\delta_{\mathrm{cal}}-\delta_{\mathrm{test}}.

Uniform control (Theorem 4.2). Extending the preceding argument uniformly over α∈(0,1)\alpha\in(0,1) presents an additional challenge. A fixed-threshold concentration inequality cannot simply be applied to an uncountable collection of thresholds through a union bound. To address this, Appendix B.2 develops a uniform concentration argument for the weighted empirical numerator through a randomized-thinning construction. We combine a one-sided DKW inequality with McDiarmid's inequality to establish

P(sup⁡t∈R1n∑i=1nw(Xi)1Si≤t−Ecal[w(X)1S≤t]≤r+)≥1−δ+.\mathbb P\left(\sup_{t\in\mathbb R}\\{\frac1n\sum_{i=1}^{n}w(X_i)\mathbf1\\{S_i\leq t\\}-\mathbb E_{\mathrm{cal}}[w(X)\mathbf1\\{S\leq t\\}]\\}\leq r_+\right)\geq1-\delta_+. We separately control the random normalization using Hoeffding's inequality. Combining these concentration events gives

P(sup⁡t∈RF^w,n(t)−Fw,n(t)≤Δunif)≥1−δ+−δ−,\mathbb P\left(\sup_{t\in\mathbb R}\\{\widehat F_{w,n}(t)-F_{w,n}(t)\\}\leq\Delta_{\mathrm{unif}}\right)\geq1-\delta_+-\delta_-,

where Δunif\Delta_{\mathrm{unif}} is explicitly characterized in Theorem 4.2. In particular, this establishes the simultaneous implication

F^w,n(T)≥1−α⟹Fw,n(T)≥1−α−Δunif,∀α∈(0,1),\widehat F_{w,n}(T)\geq1-\alpha\quad\Longrightarrow\quad F_{w,n}(T)\geq1-\alpha-\Delta_{\mathrm{unif}},\qquad\forall\alpha\in(0,1),

which yields

Pcal(∀α∈(0,1), qn(α)≤Gw,n(α+Δunif))≥1−δ+−δ−.\mathbb P_{\mathrm{cal}}\left(\forall\alpha\in(0,1),\ q_n(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{unif}})\right)\geq1-\delta_+-\delta_-.

Only after establishing this uniform calibration inequality can we invoke test-batch DKW concentration to obtain our final guarantee:

P(∀α∈(0,1),FCPmw,(n)(α)≤Gw,n(α+Δunif)+εunif)≥1−δ+−δ−−δtest.\mathbb P\left(\forall\alpha\in(0,1), \mathrm{FCP}^{w,(n)}_m(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{unif}})+\varepsilon_{\mathrm{unif}}\right)\geq1-\delta_+-\delta_--\delta_{\mathrm{test}}. Therefore, the difficulty is not merely applying concentration twice. It lies in constructing and proving the appropriate calibration-to-test inequalities under general Distribution Shift, while explicitly controlling the dependence induced by weighted normalization and obtaining simultaneous validity over all miscoverage levels. These are the central theoretical contributions of Theorems 4.1 and 4.2.

AuthorsOpenReview ↗

Reply [3/4]

3. Additional contributions.

Beyond these forward guarantees, our work also addresses two important problems that are not directly resolved by the aforementioned approaches.

Inverse FCP guarantees. Rather than fixing α\alpha and studying the resulting FCP, we consider the inverse problem of selecting a conformal level to achieve a prescribed target FCP β\beta with high probability. Under general OPDS, Gw,nG_{w,n} need not be strictly increasing or invertible, making direct inversion inappropriate. We address this through its generalized inverse

Gw,n+(u):=inf⁡τ∈[0,1]:Gw,n(τ)>u,G_{w,n}^{+}(u):=\inf\\{\tau\in[0,1]:G_{w,n}(\tau)>u\\},

and establish both fixed-target and uniform-in-target guarantees in Corollaries 5.5 and 5.6. For example, the fixed-target result guarantees

P(FCPmw,(n)([Gw,n+(β−εfix)−Δfix−η]+)≤β)≥1−δcal−δtest,\mathbb P\left(\mathrm{FCP}^{w,(n)}_m\left(\left[G_{w,n}^{+}(\beta-\varepsilon_{\mathrm{fix}})-\Delta_{\mathrm{fix}}-\eta\right]_+\right)\leq\beta\right)\geq1-\delta_{\mathrm{cal}}-\delta_{\mathrm{test}},

for any η>0\eta>0. This extends our analysis from characterizing realized FCP to selecting a miscoverage level that satisfies a desired batch-level error tolerance.

Practical estimation and consistency. A further difficulty is that Gw,nG_{w,n} and its generalized inverse depend on unknown population distributions and are generally not directly computable. Accordingly, Section 5 develops Algorithms 1 and 2 to construct empirical approximations using weighted calibration observations, with uniform consistency established in Theorems 5.3 and 5.7 under Assumption 5.2. Importantly, Assumption 5.2 is introduced for the consistency analysis of these estimation procedures, not for the main finite-sample FCP guarantees. Algorithms 1 and 2 remain computationally applicable when this assumption is violated, although their consistency is then no longer guaranteed.

We further investigate the relaxation of this assumption in Appendix C.2. By introducing independent labeled holdout observations from the test distribution, Algorithms 3 and 4 allow the unknown population quantities to be estimated without exact score transport. Theorems C.2 and C.3 establish uniform consistency under the substantially weaker Assumption C.1, which does not require the weighted calibration-score distribution to match, or even closely approximate, the test-score distribution. These results further extend the applicability of our estimation framework to settings with imperfect weighting.

4. Summary of our contributions.

While the underlying statistical tools are classical, we respectfully emphasize that our work is not simply a direct combination of previously established FCP and Weighted CP techniques. The main contributions are:

  1. Establishing explicit finite-sample, high-probability FCP guarantees under general OPDS, beyond the exchangeability and Covariate Shift conditions underlying existing finite-sample approaches.
  2. Developing new calibration-to-test concentration arguments for self-normalized weighted empirical distributions, including uniform control over a continuum of miscoverage levels.
  3. Extending these guarantees to the inverse FCP problem, with both fixed-target and uniform-in-target guarantees.
  4. Developing practical estimation procedures, proving their consistency, and providing a theoretically justified extension under a weaker score-distribution assumption.

We have added Appendix C.5 to clarify the relationship with prior approaches and the technical challenges addressed by our finite-sample FCP guarantees under general OPDS.

References

[1] Gazin, U., Blanchard, G., & Roquain, E. (2024). Transductive conformal inference with adaptive scores. International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 1504–1512. PMLR.

[2] Blain, A., Thirion, B., & Neuvial, P. (2025). False coverage proportion control for conformal prediction. Forty-second International Conference on Machine Learning (ICML).

[3] Gazin, U. (2024). Asymptotics for conformal inference. arXiv preprint arXiv:2409.12019.

[4] Pournaderi, M., & Xiang, Y. (2024). Training-conditional coverage bounds under covariate shift. arXiv preprint arXiv:2405.16594.

Reviewer_m5NDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates false coverage proportion (FCP) control for weighted conformal prediction under calibration-to-test distribution shift. It derives finite-sample upper bounds for the FCP at fixed and uniform miscoverage levels, develops corresponding inverse calibration procedures, and proposes weighted estimators of the quantities appearing in the theoretical bounds. Numerical studies are conducted under covariate shift and synthetic score-transport shift.

Critical strengths
  1. The paper studies a meaningful extension of FCP control beyond the classical exchangeable setting. Theorems 4.1 and 4.2 isolate calibration uncertainty and test-batch uncertainty and provide both fixed-α\alpha and simultaneous-in-α\alpha statements.

  2. The paper treats forward and inverse FCP questions in a unified framework. The empirical study is broad and includes asymptotic behavior, efficiency, runtime, and model/weight ablations.

Critical weaknesses

1. Limited novelty relative to existing weighted FCP theory. Gazin (2024) has already studied weighted FCP under distribution shift, including oracle and non-oracle weights. While Appendix A.2 highlights the distinction between asymptotic and finite-sample guarantees, the underlying framework is closely related, leaving the methodological novelty beyond finite-sample analysis somewhat limited. Moreover, the experiments lack a comparison with the weighted asymptotic results of Gazin (2024).

  • Gazin, U. (2024). Asymptotics for conformal inference. arXiv preprint arXiv:2409.12019.

2. Unnecessary complexity of the proposed estimators. Under Assumption 5.2, Gw,nG_{w,n} admits an explicit expression depending only on μw\mu_w (Appendix B.4, around line 1845). A simpler plug-in estimator based on μ^w\widehat{\mu}_w would avoid the sorting procedures in Algorithms 1 and 2 and achieve lower computational complexity. The advantage of the proposed estimators over this straightforward alternative remains unclear.

3. Gap between theoretical and practical finite-sample guarantees. The finite-sample FCP bounds depend on the unknown Gw,nG_{w,n}, which is replaced by a plug-in estimate in practice. The established asymptotic consistency does not ensure finite-sample validity of these estimated bounds, as estimation uncertainty is not incorporated.

4. Restrictive score-transport assumption. Practical estimation relies on the strong Assumption 5.2, requiring exact matching between the weighted calibration-score and test-score distributions. This condition may be difficult to satisfy or verify, particularly under shifts in P(Y∣X)P(Y\mid X). Robustness to violations of this assumption is not investigated.

5. Insufficient investigation of bound tightness. The empirical pass rates are frequently near one, suggesting potentially conservative bounds. However, the experiments do not adequately quantify the gap between the realized FCP and its upper bound or compare tightness against alternative procedures.

Questions for authors

Please see the issues raised in the Weaknesses section.

8 comments
AuthorsOpenReview ↗

Reply [1/6]

W1. Gazin (2024) has already studied weighted FCP under distribution shift, including oracle and non-oracle weights. While Appendix A.2 highlights the distinction between asymptotic and finite-sample guarantees, the underlying framework is closely related, leaving the methodological novelty beyond finite-sample analysis somewhat limited. Moreover, the experiments lack a comparison with the weighted asymptotic results of Gazin (2024).

We thank the reviewer for highlighting the relationship with Gazin (2024). We acknowledge that Gazin (2024) provides an important foundation for understanding weighted FCP under Distribution Shift, including both oracle and non-oracle weighting schemes. However, our contribution addresses a fundamentally different theoretical objective: establishing explicit, finite-sample, high-probability FCP guarantees, rather than characterizing the asymptotic behavior of the FCP process. As discussed in Appendix A.2, this distinction is substantial, both mathematically and in terms of the resulting guarantees.

Theoretical distinction and technical challenges. Gazin (2024, Theorems 3.3–3.4) establishes functional central limit theorems characterizing the asymptotic fluctuations of weighted FCP. In particular, writing τn,m=nm/(n+m)\tau_{n,m}=nm/(n+m), their results take the form

τn,m(FCPmw,(n)−Gw)→dZw,n,m→∞,\sqrt{\tau_{n,m}}\left(\mathrm{FCP}^{w,(n)}_m-G^w\right)\xrightarrow{d}\mathcal Z_w,\qquad n,m\to\infty,

where GwG^w characterizes the limiting behavior under Distribution Shift and Zw\mathcal Z_w is a Gaussian limiting process. While this provides a precise characterization of the limiting distribution, weak convergence does not establish finite-sample probability guarantees. In particular, it does not quantify the approximation error between the finite-sample FCP distribution and its asymptotic limit at a given (n,m)(n,m). Consequently, asymptotically valid confidence bounds need not attain their nominal coverage probability for finite calibration and test samples. Our work instead establishes guarantees of the form

P(FCPmw,(n)(α)≤Gw,n(α+Δfix)+εfix)≥1−δ,\mathbb P \left(\mathrm{FCP}^{w,(n)}_m(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{fix}})+\varepsilon_{\mathrm{fix}}\right)\geq1-\delta, and, more strongly,

P(∀α∈(0,1), FCPmw,(n)(α)≤Gw,n(α+Δunif)+εunif)≥1−δ.\mathbb P\left(\forall\alpha\in(0,1),\ \mathrm{FCP}^{w,(n)}_m(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{unif}})+\varepsilon_{\mathrm{unif}}\right)\geq1-\delta.

These guarantees hold for every finite n,mn,m, with explicit error terms depending on the sample sizes, weight bound, and prescribed failure probability. Moreover, our results require only Assumptions 2.2 and 2.3, without the differentiability and additional regularity conditions imposed in the corresponding asymptotic analysis of Gazin (2024).

Importantly, obtaining these finite-sample guarantees is not a direct consequence of the asymptotic results. The mathematical difficulty lies in controlling the self-normalized weighted empirical calibration distribution, whose numerator and denominator are dependent random quantities, while simultaneously accounting for uncertainty from the independent test batch. For Theorem 4.1, we construct a population threshold and derive a concentration inequality for centered weighted calibration variables, yielding the explicit error

Δfix=εwμw+w∞/nlog⁡(1/δcal)2n.\Delta_{\mathrm{fix}}=\frac{\varepsilon_w}{\mu_w+w_\infty/n}\sqrt{\frac{\log(1/\delta_{\mathrm{cal}})}{2n}}.

Theorem 4.2 is more technically demanding because the guarantee must hold simultaneously over all α∈(0,1)\alpha\in(0,1). We establish a uniform concentration inequality for the weighted empirical numerator using a randomized-thinning construction, one-sided DKW, and McDiarmid's inequality, while independently controlling the random normalization through Hoeffding's inequality (Appendix B.2). These arguments yield an explicit uniform deviation bound that remains valid at finite sample sizes.

Thus, our results require a separate non-asymptotic concentration analysis that is not supplied by the functional central limit theorems of Gazin (2024). This provides a rigorous finite-sample guarantee rather than a large-sample approximation.

AuthorsOpenReview ↗

Reply [2/6]

Further contributions beyond finite-sample analysis. Our work also extends the scope of inference in two important directions. First, Corollaries 5.5 and 5.6 address the inverse FCP problem, selecting a conformal miscoverage level to satisfy a prescribed target FCP with high probability, both at fixed targets and uniformly over targets. This problem is not addressed by Gazin (2024). Second, Section 5 develops practical estimation procedures for the population quantities appearing in our bounds, with consistency established in Theorems 5.3 and 5.7 under the stated additional assumptions. These developments extend beyond merely characterizing the asymptotic FCP distribution.

Regarding the experimental comparison. We appreciate the reviewer's suggestion. However, the primary objectives of the two approaches are different. Gazin (2024) provides asymptotic approximations whose validity is established as the sample sizes diverge, whereas our experiments primarily evaluate whether the proposed bounds satisfy their prescribed finite-sample coverage probabilities. Consequently, a direct comparison of nominal guarantee satisfaction would evaluate the two approaches against a criterion for which only our population-level bounds are designed to provide finite-sample validity. While comparing the empirical tightness and approximation accuracy of asymptotic and non-asymptotic bounds could provide complementary insights, it would not replace the central finite-sample validity evaluation conducted in our paper.

We have also added Appendix C.5 to explain the calibration-to-test inequalities underlying our finite-sample FCP guarantees under general OPDS.

AuthorsOpenReview ↗

Reply [3/6]

W2. Under Assumption 5.2, Gw,nG_{w,n} admits an explicit expression depending only on μw\mu_w (Appendix B.4, around line 1845). A simpler plug-in estimator based on μ^w\widehat{\mu}_w would avoid the sorting procedures in Algorithms 1 and 2 and achieve lower computational complexity. The advantage of the proposed estimators over this straightforward alternative remains unclear.

We thank the reviewer for this observation. We agree that, under Assumption 5.2, Gw,nG_{w,n} admits an explicit expression depending only on μw\mu_w, and that substituting its empirical estimate μ^w\widehat{\mu}_w provides a computationally simpler alternative. However, we would like to emphasize that Assumption 5.2 is introduced to establish rigorous consistency guarantees for our estimation algorithms, while the algorithms themselves are designed as part of a more general estimation framework that remains applicable even when this assumption is not exactly satisfied.

Motivation for Algorithms 1 and 2. Assumption 5.2 requires the weighted calibration-score distribution to exactly reproduce the test-score distribution:

Ftest(t)=μw−1Ecal[w(X)1S≤t],∀t∈R.F_{\mathrm{test}}(t)=\mu_w^{-1}\mathbb E_{\mathrm{cal}}[w(X)\mathbf1\\{S\leq t\\}],\qquad \forall t\in\mathbb R.

Although this assumption provides a convenient theoretical characterization and enables us to establish rigorous consistency guarantees, its exact satisfaction is difficult to ensure in practical applications, particularly when the weighting function is estimated or imperfectly specified. In such settings, the closed-form expression identified by the reviewer is no longer necessarily equal to the true Gw,nG_{w,n}.

Our motivation for Algorithms 1 and 2 is therefore to develop a more general, distribution-based estimation framework that constructs empirical approximations of Gw,nG_{w,n} and its generalized inverse directly from weighted calibration scores, rather than relying exclusively on the closed-form characterization. While Assumption 5.2 provides the theoretical foundation for establishing consistency, our objective is to develop methods that remain practically applicable even when this assumption is not exactly satisfied. In particular, Algorithms 1 and 2 remain well-defined under violations of Assumption 5.2 and can still provide useful surrogate estimates in practice, although their theoretical consistency is no longer guaranteed. We acknowledge that such violations may introduce estimation bias that cannot generally be eliminated using calibration data alone. Nevertheless, we believe it is important to distinguish between the assumptions required to establish rigorous theoretical guarantees and the broader practical applicability of the proposed estimation procedures.

Importantly, our paper already addresses this limitation in Appendix C.2.We extend the estimation framework to Algorithms 3 and 4, which incorporate independent labeled holdout observations from the test distribution. Theorems C.2 and C.3 establish their uniform consistency under the substantially weaker Assumption C.1, without requiring exact score transport. This extension illustrates the broader motivation for developing empirical distribution-based estimators beyond the special case admitting an explicit formula.

We therefore agree that the proposed closed-form plug-in estimator offers a simpler implementation when Assumption 5.2 is justified. Nevertheless, the distribution-based formulation provides a general computational framework that naturally extends to settings with imperfect weighting and additional test-distribution information.

We have added a remark at the end of Appendix D.2 clarifying the simpler closed-form plug-in alternative under Assumption 5.2 and the motivation for our distribution-based estimation framework.

Reviewer_ihQtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the false coverage proportion (FCP) of weighted conformal prediction when calibration and test observations follow different distributions. It gives finite-sample upper bounds for FCP at a fixed miscoverage level and uniformly over miscoverage levels, then inverts these bounds to select a level for a target FCP. It also proposes empirical estimators of the quantities appearing in the bounds and evaluates the procedures on regression and classification datasets under covariate shift and a synthetic score-transport shift.

Critical strengths
  • The paper considers a meaningful batch-level coverage criterion under distribution shift, and its oracle FCP bounds are stated explicitly for both fixed and uniform miscoverage levels.
  • The soundness is good.
Critical weaknesses
  • The novelty is limited, as the paper combines established FCP questions with weighted conformal prediction. Under covariate shift, existing PAC and training-conditional coverage results already bound the test miscoverage probability conditional on the calibration data [1,2]. Given the independent test observations assumed here, standard Hoeffding or DKW inequalities then yield finite-batch FCP bounds, including uniform versions when the calibration bound is uniform. The inverse results amount to inverting these bounds, while the more general distribution-shift bounds still involve an unknown population function. I do not see a sufficiently distinct theoretical contribution beyond these combinations.
  • Under Assumption 5.2, the function targeted by Algorithm 1 has the explicit form Gw,n(u)=[u−(1−u)w∞/(nμw)]G_{w,n}(u)=[u-(1-u)w_\infty/(n\mu_w)], and its generalized inverse is also explicit. This makes it unclear why Algorithms 1 and 2 are needed instead of simply estimating the mean weight and substituting it into these formulas. In addition, the finite-sample FCP guarantees use the population function Gw,nG_{w,n} , whereas the implemented bounds replace it with an estimate for which only consistency is established. The paper does not give a finite-sample correction that would guarantee the same coverage probability for these plug-in bounds.
  1. Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. PAC Prediction Sets Under Covariate Shift.
  2. Mehrdad Pournaderi and Yu Xiang. Training-Conditional Coverage Bounds under Covariate Shift.
Questions for authors
  • Can the authors identify which part of Theorems 4.1 and 4.2 cannot be obtained from existing training-conditional coverage results together with concentration over the test batch?

  • Given the closed form of $G_{w,n} $ under Assumption 5.2, what is the benefit of Algorithms 1 and 2, and is there a finite-sample guarantee for the resulting plug-in FCP bounds?

5 comments
AuthorsOpenReview ↗

Reply [1/3]

W1. The novelty is limited, as the paper combines established FCP questions with weighted conformal prediction. Under ... I do not see a sufficiently distinct theoretical contribution beyond these combinations.

Q1. Can the authors identify which part of Theorems 4.1 and 4.2 cannot be obtained from existing training-conditional coverage results together with concentration over the test batch?

We thank the reviewer for this important observation and the opportunity to clarify our theoretical contributions. We agree that, once a suitable high-probability bound on the calibration-conditional miscoverage probability is available, concentration over an independent test batch can be handled using standard Hoeffding or DKW inequalities. However, this observation does not directly establish our results. The principal theoretical difficulty lies in deriving the required calibration-conditional bound under general One-Phase Distribution Shift (OPDS), rather than in the subsequent test-batch concentration step.

Limitations of existing approaches. Existing PAC and training-conditional coverage results under Covariate Shift rely on a specific relationship between the calibration and test distributions, notably the preservation of P(Y∣X)P(Y\mid X) and the use of an appropriate importance-weighting scheme. These assumptions enable the conditional miscoverage probability to be controlled relative to the nominal level α\alpha. In contrast, our Assumptions 2.2 and 2.3 do not require Covariate Shift, an oracle density ratio, or even that the weighted calibration-score distribution matches the test-score distribution. In particular, changes in P(Y∣X)P(Y\mid X) are permitted. Consequently, a bound of the form qn(α)≤α+εq_n(\alpha)\leq\alpha+\varepsilon need not hold, even asymptotically. To see why this prevents the proposed approach from directly establishing our results, define

qn(α):=Ptest(pw,(n)≤α∣Dcal).q_n(\alpha):=\mathbb P_{\mathrm{test}}\left(p^{w,(n)}\leq\alpha\mid\mathcal D_{\mathrm{cal}}\right).

Conditionally on the calibration data, the test miscoverage indicators are i.i.d. Bernoulli variables with mean qn(α)q_n(\alpha). Hence, Hoeffding's inequality yields

P(FCPmw,(n)(α)≤qn(α)+εtest∣Dcal)≥1−δtest.\mathbb P \left(\mathrm{FCP}^{w,(n)}_m(\alpha)\leq q_n(\alpha)+\varepsilon_{\mathrm{test}}\mid\mathcal D_{\mathrm{cal}}\right)\geq1-\delta_{\mathrm{test}}.

This is precisely what test-batch concentration establishes: concentration around the unknown qn(α)q_n(\alpha), not around the nominal level α\alpha. Under general OPDS, the discrepancy can be large. For example, consider unit weights with calibration scores S∼Unif(0,1)S\sim\mathrm{Unif}(0,1) and test scores T∼Unif(0.5,1.5)T\sim\mathrm{Unif}(0.5,1.5), satisfying Assumptions 2.2–2.3. As n→∞n\to\infty, at α=0.1\alpha=0.1,

qn(0.1)⟶P(T≥0.9)=0.6,q_n(0.1)\longrightarrow\mathbb P(T\geq0.9)=0.6,

rather than 0.10.1. Thus, directly applying a nominal-level training-conditional coverage bound followed by test-batch concentration would fail in this setting. A separate finite-sample characterization of qn(α)q_n(\alpha) that explicitly accounts for distribution shift is necessary.

Our theoretical contribution. Rather than assuming that qn(α)q_n(\alpha) is close to α\alpha, we identify the appropriate shift-dependent population function Gw,nG_{w,n} and establish explicit high-probability comparisons with it. Specifically, writing

F^w,n(t)=1n∑i=1nw(Xi)1Si≤tw∞n+1n∑i=1nw(Xi), and Fw,n(t)=Ecal[w(X)1S≤t]μw+w∞/n,\widehat F_{w,n}(t)=\frac{\frac1n\sum_{i=1}^n w(X_i)\mathbf1\\{S_i\leq t\\}}{\frac{w_\infty}{n}+\frac1n\sum_{i=1}^n w(X_i)}, \text{ and }F_{w,n}(t)=\frac{\mathbb E_{\mathrm{cal}}[w(X)\mathbf1\\{S\leq t\\}]}{\mu_w+w_\infty/n},

we define Gw,n(u)=Ptest(Fw,n(T)≥1−u)G_{w,n}(u)=\mathbb P_{\mathrm{test}}(F_{w,n}(T)\geq1-u). The central calibration inequalities established in Appendices B.1 and B.2 are

Pcal(qn(α)≤Gw,n(α+Δfix))≥1−δcal,\mathbb P_{\mathrm{cal}}\left(q_n(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{fix}})\right)\geq1-\delta_{\mathrm{cal}},

and, more strongly,

Pcal(∀α∈(0,1), qn(α)≤Gw,n(α+Δunif))≥1−δ+−δ−.\mathbb P_{\mathrm{cal}}\left(\forall\alpha\in(0,1),\ q_n(\alpha)\leq G_{w,n}(\alpha+\Delta_{\mathrm{unif}})\right)\geq1-\delta_+-\delta_-.

These inequalities constitute the missing step in the reviewer's proposed approach. They hold under Assumptions 2.2 and 2.3, without requiring the Covariate Shift assumptions used by existing training-conditional coverage results. In particular, Gw,nG_{w,n} explicitly characterizes the discrepancy between the weighted calibration-score distribution and the test-score distribution, allowing our guarantees to remain valid even when the weighting scheme does not perfectly correct the distribution shift.

AuthorsOpenReview ↗

Reply [2/3]

Establishing these inequalities requires controlling a self-normalized weighted empirical distribution whose numerator and denominator depend on the same calibration observations. For the fixed-α\alpha result (Appendix B.1), we construct a population threshold τα\tau_\alpha and control the weighted variables

Zi=w(Xi)[1Si≤τα−(1−α)]Z_i=w(X_i)\left[\mathbf1\\{S_i\leq\tau_\alpha\\}-(1-\alpha)\right]

using Hoeffding's inequality. This yields the explicit calibration error

Δfix=εwμw+w∞/nlog⁡(1/δcal)2n.\Delta_{\mathrm{fix}}=\frac{\varepsilon_w}{\mu_w+w_\infty/n}\sqrt{\frac{\log(1/\delta_{\mathrm{cal}})}{2n}}.

For the uniform result (Appendix B.2), pointwise concentration is insufficient because the guarantee must hold simultaneously over a continuum of miscoverage levels. We instead establish uniform concentration of the weighted empirical numerator through a randomized-thinning construction, combining a one-sided DKW inequality with McDiarmid's inequality. We separately control the random normalization using Hoeffding's inequality and combine these events to derive the explicit Δunif\Delta_{\mathrm{unif}} in Theorem 4.2. The essential consequence of this analysis is the high-probability implication

F^w,n(T)≥1−α⟹Fw,n(T)≥1−α−Δ,\widehat F_{w,n}(T)\geq1-\alpha\quad\Longrightarrow\quad F_{w,n}(T)\geq1-\alpha-\Delta,

which establishes qn(α)≤Gw,n(α+Δ)q_n(\alpha)\leq G_{w,n}(\alpha+\Delta) with explicit finite-sample errors, including uniformly over α\alpha.

To directly answer Q1, we agree that existing training-conditional coverage guarantees under Covariate Shift can be combined with test-batch concentration to yield FCP bounds within their respective assumptions. However, they do not directly provide the shift-dependent calibration-conditional inequalities required for Theorems 4.1 and 4.2 under general OPDS. Deriving these inequalities, with explicit finite-sample constants and uniform validity, is the central theoretical contribution of our results. The subsequent Hoeffding/DKW concentration over the test batch is indeed standard, but it becomes sufficient only after this missing step has been established.

In summary, our contribution is not the use of Hoeffding or DKW concentration itself, but the derivation of explicit, finite-sample calibration-to-test inequalities for weighted conformal prediction under general OPDS, together with their fixed-level and uniform FCP consequences. This extends beyond the settings covered directly by existing training-conditional coverage theory. Our inverse FCP results and practical estimation procedures then build on this theoretical framework. We will revise the manuscript to clarify this distinction and more explicitly acknowledge the connection to prior PAC coverage results.

We have added Appendix C.5 to clarify the connection to existing training-conditional coverage results and the calibration-to-test inequalities underlying Theorems 4.1 and 4.2 under general OPDS.

AuthorsOpenReview ↗

Reply [3/3]

W2. Under Assumption 5.2, the function targeted by Algorithm 1 has the explicit form Gw,n(u)=[u−(1−u)w∞/(nμw)]G_{w,n}(u) = [u - (1-u)w_\infty/(n\mu_w)], and its generalized inverse is also explicit ... The paper does not give a finite-sample correction that would guarantee the same coverage probability for these plug-in bounds.

Q2. Given the closed form of Gw,nG_{w,n} under Assumption 5.2, what is the benefit of Algorithms 1 and 2, and is there a finite-sample guarantee for the resulting plug-in FCP bounds?

We thank the reviewer for this insightful observation. We agree that the stated closed-form expressions for Gw,nG_{w,n} and its generalized inverse hold under Assumption 5.2. However, the important distinction is that Assumption 5.2 is an additional condition imposed for the consistency analysis of Algorithms 1 and 2, rather than a requirement of our main theoretical framework. Its exact satisfaction may also be difficult to ensure in practical distribution-shift settings.

Role of Assumption 5.2 and the estimation algorithms. Our finite-sample FCP guarantees in Theorems 4.1 and 4.2 require only Assumptions 2.2 and 2.3, allowing general OPDS without requiring the weighting scheme to exactly recover the test-score distribution. Under these assumptions, the population quantity Gw,nG_{w,n} is generally intractable. Assumption 5.2 additionally requires

Ftest(t)=Ecal[w(X)1S≤t]μw,∀t∈R.F_{\mathrm{test}}(t)=\frac{\mathbb E_{\mathrm{cal}}[w(X)\mathbf1\\{S\leq t\\}]}{\mu_w},\qquad\forall t\in\mathbb R.

This exact score-transport relation yields the closed form identified by the reviewer:

Gw,n(u)=[u−(1−u)w∞nμw]+.G_{w,n}(u)=\left[u-\frac{(1-u)w_\infty}{n\mu_w}\right]_+.

We acknowledge that, when Assumption 5.2 holds exactly, estimating μw\mu_w and substituting it into this expression provides a simpler alternative to Algorithms 1 and 2.

However, real-world weighting schemes are generally estimated from finite data and may not perfectly reproduce the test-score distribution. In such settings, the above closed form is no longer guaranteed to characterize the true Gw,nG_{w,n}. This motivates retaining a general distribution-based estimation framework rather than relying exclusively on the exact score-transport identity.

Specifically, Algorithms 1 and 2 construct empirical approximations using the weighted calibration-score distribution as a surrogate for the unavailable test-score distribution. They remain computationally applicable when Assumption 5.2 is only approximately satisfied, although their theoretical consistency is established under its exact satisfaction. We do not claim that these calibration-only estimators remain universally consistent under arbitrary violations of Assumption 5.2; such violations can introduce systematic estimation bias.

Importantly, our framework also explicitly addresses this limitation in Appendix C.2. When an independent labeled holdout sample from the test distribution is available, Algorithms 3 and 4 estimate Gw,nG_{w,n} and its generalized inverse directly using test-distribution information. Theorems C.2 and C.3 establish their uniform consistency under the substantially weaker Assumption C.1, which does not require the weighted calibration-score distribution to match the test-score distribution. Therefore, the proposed framework is not restricted to exact score transport: it provides different estimation procedures depending on the available information and the assumptions that can be justified.

Finite-sample validity of the plug-in bounds. We thank the reviewer for highlighting this important distinction. While Theorems 4.1 and 4.2 establish rigorous finite-sample guarantees for the population-based FCP bounds under Assumptions 2.2 and 2.3, Theorems 5.3 and 5.7 establish the uniform consistency of their plug-in estimators, rather than finite-sample guarantees for the estimated bounds themselves. We acknowledge that our current work does not establish that the uncorrected plug-in bounds retain the same finite-sample coverage probability. Nevertheless, our consistency results provide theoretical justification for their asymptotic accuracy, while the empirical evaluations in Section 6 and Appendix D demonstrate their effectiveness across the considered distribution-shift settings. Extending these results to rigorous finite-sample plug-in guarantees is an important direction for future research. Potential approaches include deriving non-asymptotic uniform estimation-error bounds for Gw,nG_{w,n} and its generalized inverse, developing concentration inequalities for self-normalized weighted empirical processes, and incorporating explicit estimation-uncertainty corrections into the FCP bounds while accounting for the overall confidence budget.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful evaluations, constructive comments, and valuable suggestions, which have helped us further improve and clarify our work. We appreciate the reviewers' recognition of the importance of False Coverage Proportion (FCP) control under Distribution Shift, the soundness of our theoretical results, and the breadth of our empirical evaluations.

Several common concerns emerged regarding (1) the novelty of our contributions relative to existing FCP and Weighted Conformal Prediction literature, (2) the assumptions and practical applicability of our estimation procedures, particularly the distinction between population-level and plug-in guarantees, and (3) the tightness of our bounds and the motivation and clarity of our experimental presentation.

To address these concerns, we have provided detailed theoretical explanations distinguishing our work from existing approaches, particularly the asymptotic results of Gazin (2024) and training-conditional coverage theory. Importantly, we emphasize that our main contribution lies in establishing explicit finite-sample FCP guarantees under general One-Phase Distribution Shift, including fixed-level, uniform, and inverse FCP control. We have further clarified the technical challenges involved in controlling self-normalized weighted empirical distributions and obtaining high-probability calibration-to-test guarantees beyond standard Covariate Shift assumptions.

We have also clarified the role of Assumption 5.2 in establishing estimator consistency, highlighted its relaxation under the weaker Assumption C.1 in Appendix C.2, and explicitly acknowledged the distinction between finite-sample population guarantees and asymptotically consistent plug-in estimators. Furthermore, we have conducted additional experiments quantifying bound tightness against existing approaches, demonstrating that our bounds achieve reasonable gaps while avoiding the systematic underestimation observed in prior methods. We have also clarified the practical motivation for Section 3 and will further polish the manuscript to improve its clarity and presentation.

We believe these clarifications and additional experimental results strengthen the theoretical positioning, practical relevance, and empirical evidence of our work. We sincerely thank the reviewers again for their time and consideration, and we welcome any further feedback.

AuthorsOpenReview ↗

Summary of Revisions

Incorporating the reviewers' valuable comments and suggestions, we have conducted additional empirical studies and planned several revisions to strengthen the theoretical positioning, practical relevance, and clarity of our manuscript. The main revisions are summarized below:

  1. Expanded comparison with prior approaches (new Appendix C.5). We provide a detailed comparison with existing FCP theory, the asymptotic weighted FCP results of Gazin (2024), and PAC/training-conditional coverage methods under Covariate Shift. We clarify why these approaches do not directly establish our finite-sample guarantees under general One-Phase Distribution Shift (OPDS), emphasizing the technical challenges of controlling self-normalized weighted empirical distributions and establishing uniform calibration-to-test inequalities. We further highlight our distinct contributions in fixed, uniform, and inverse FCP control, with the main distinctions emphasized in the Introduction.

  2. Clarified estimation assumptions and theoretical limitations (Appendix C.5 and Section 5). We clarify that Assumption 5.2 is required for the consistency guarantees of Algorithms 1 and 2, rather than for our main finite-sample FCP theory. We also emphasize the weaker Assumption C.1 and the holdout-based estimation procedures developed in Appendix C.2. Additionally, we explicitly distinguish the finite-sample guarantees for population-based bounds from the asymptotic consistency of their plug-in estimators, acknowledging finite-sample plug-in validity as an important direction for future research.

  3. Additional evaluation of bound tightness (new Appendix D.9). We have conducted additional experiments on Diamonds and Fashion-MNIST under Covariate Shift and Score-Transport Shift, directly quantifying the mean gap between realized FCP and estimated upper bounds. We compare our method against DKW and CoJER, demonstrating that our bounds maintain reasonably small positive gaps, whereas existing methods systematically underestimate the realized FCP. These results complement our previous guarantee-probability evaluations by assessing both empirical validity and bound tightness.

  4. Clarified the motivation for Section 3 (new remark in Appendix D.2). We emphasize that Section 3 is intended to establish the practical limitations of existing FCP bounds beyond the theoretical loss of guarantees under violated exchangeability. In particular, our experiments demonstrate that these bounds can fail substantially in practice, providing empirical justification for developing Distribution-Shift-aware FCP guarantees.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.