acceptodds
Under review as a conference paper at ICLR 2027

Distributional Balancing Weights for Conformal Calibration under Covariate Shift

Abstract

Conformal prediction provides distribution-free marginal coverage under exchangeability. Under covariate shift, covariate distributions change while the conditional response distribution remains unchanged. Weighted conformal prediction corrects this mismatch using known density ratios, which can be difficult to estimate. We propose conformal calibration with distributional balancing weights, learned to align source calibration and unlabeled target covariate distributions without using calibration or target responses. Characteristic function distance provides a common framework for kernel-based and energy-based balancing. We establish a finite-sample error bound for estimating the target score distribution, yielding approximate marginal coverage guarantees. The bound separates covariate imbalance, approximation of conditional score probabilities, target-sample uncertainty, and weight concentration. We connect this bound to weight optimization and develop a validation rule for selecting weights without responses. A spectral analysis shows that nearly uniform fitted weights can miss increasingly fine-scale shifts and yield persistent undercoverage despite bounded density ratios. Controlled synthetic and semi-synthetic experiments reproduce this failure and show that tuning bandwidth and regularization can restore near-nominal coverage. Together, these results highlight the importance of kernel choice and regularization beyond controlling weight concentration alone.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, covariate shift, uncertainty quantification, distributional balancing, kernel mean matching, spectral analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pcRUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conformal calibration under covariate shift, where the conditional response law is unchanged but source and target covariate distributions differ. The authors propose learning distributional balancing weights to align source calibration covariates with unlabeled target covariates, then calibrating from the weighted score distribution, with characteristic-function distances unifying several balancing criteria. On the theory side, they discuss error decompositions for the target score distribution, approximate coverage, and links between kernel geometry/regularization and coverage; experiments include controlled synthetic settings and some semi-synthetic/real comparisons.

Critical strengths
  1. Emphasizing that bounded ratios or controlled weight concentration need not imply that the shift has been corrected is useful and interesting for the community. Besides, it explain the importance of kernel geometry/regularization via theory and experiments.
  2. Attempting to separate imbalance, approximation error, sampling, and weight concentration makes different balancers easier to compare and strengthens credibility.
Critical weaknesses
  1. The method positioning is unclear for now. The paper seems to present DB-CC largely as a common interface over existing response-free weighted rules, while KMM-style weighted calibration already exists.
  2. Besides, the main experiments do not appear to include a direct comparison with KMM-CP. At the same time, relative to the stronger baselines in the paper, the main results often look close but not stably better.
  3. On fine-scale and high-frequency shifts, learned methods appear unable to approach the oracle, thus the theory diagnoses a problem that the method does not seem to fix.
Questions for authors
  1. Under the same protocol and candidate library, what are the coverage / set length / ESS numbers versus KMM-CP (or its closest implementation)? If the gap is small, how should the algorithmic contribution be separated from the spectral/diagnostic analysis?
  2. Relative to Selected flexible WCP / quadratic WCP, are the main-table differences stable and better?
1 comment

We thank the reviewer pcRU for recognizing the importance of separating spectral resolution from weight concentration and for requesting matched comparisons with paired uncertainty.

Q1. Matched Gaussian comparison and the contribution beyond it. The implementation linked in KMM-CP was inaccessible when preparing this comparison. We therefore implemented its nonselective Gaussian KMM calibration component under matched settings. Matching the kernel, normalization, regularization, constraints, and quantile rule yields the same objective, weights, and thresholds as Gaussian DB-CC. The revision makes this correspondence explicit in Section 5.6 and Appendix G.7.

The synthetic comparison fixes preprocessing, bandwidth/ridge grids, cap, ESS screen, uniform fallback, common validation geometry, and quantile convention, varying only the candidate kernel library. At n=1,000n=1{,}000, Table 9 reports:

ShiftGaussian-only: coverage / length / ESS fractionSelected DB-CC: coverage / length / ESS fraction
Interaction0.8917 / 5.54 / 0.6830.8909 / 5.53 / 0.715
Checkerboard k=1k=10.8770 / 4.56 / 0.9090.8777 / 4.57 / 0.902
Checkerboard k=2k=20.8702 / 4.45 / 0.9520.8710 / 4.46 / 0.941

The incremental effects are small in these settings. The added severe-shift Parkinson's comparison standardizes data splits, tuning grids, and the response-free criterion form. Each family minimizes candidate-specific validation CFD plus 0.01/ESS0.01/\sqrt{\mathrm{ESS}}, then refits on the full unlabeled target sample. Appendix G.7 distinguishes this protocol from Algorithm 1's common-geometry, no-refit selection.

Table 24 reports the following shared-grid differences, alternative minus matched Gaussian:

KernelAbsolute coverage-error difference (percentage points)Mean length differenceESS differenceSuccessful pairs
Product-Laplace-1.76 [-2.83, -0.70]0.18 [-0.19, 0.55]10.4 [6.9, 13.8]49
Matérn-3/2-0.77 [-1.43, -0.10]-0.06 [-0.35, 0.23]3.1 [1.8, 4.4]50

There were 50 attempted repetitions. Intervals are pointwise 95% Monte Carlo t-intervals. Absolute coverage errors and lengths are averaged across five nominal levels within each repetition; ESS is computed once per fit. This supports reduced repeated-sample calibration error in this experiment, with uncertainty about average length changes.

The analytical contribution is distinct from a new Gaussian weighting objective: Theorem 1's four-part score-CDF bound, Corollary 2's objective-value connection, Theorem 3's population-balance oracle inequality, and Theorem 2's explicit coverage separation for empirical constrained optimizers.

Q2. Stability relative to flexible and quadratic WCP. The benefits are regime-dependent. For DB-CC minus selected flexible WCP at n=1,000n=1{,}000, Table 10 gives these paired absolute-coverage-gap differences in percentage points:

ShiftDifference [pointwise 95% Monte Carlo interval]
Interaction-0.192 [-0.295, -0.088]
Checkerboard k=1k=1-0.343 [-0.428, -0.257]
Checkerboard k=2k=2-0.018 [-0.067, 0.032]

At n=300n=300, DB-CC has larger mean absolute gaps for both checkerboards. Quadratic WCP is competitive for the interaction shift: at n=1,000n=1{,}000, its mean absolute gap is 0.0123, versus 0.0130 for DB-CC (Table 9). We therefore report the favorable and unfavorable regimes rather than claim consistent superiority.

Fine-scale failure and recovery. The higher-frequency checkerboard remains unresolved: at k=2k=2, n=1,000n=1{,}000, learned procedures cover approximately 87%, while oracle WCP remains near 90%. Section 6 states this limitation explicitly. The circular experiment separately demonstrates recovery: tuning raises Gaussian coverage from 0.8608 to 0.8983, matching tuned Matérn at 0.8980 (Table 13). Together, these results establish a failure mechanism and a controlled remedy, while identifying a setting where the tested libraries still have insufficient effective resolution.

Reviewer_5kbFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes distributionally balanced conformal calibration (DB-CC). DB-CC replaces density-ratio estimation in weighted conformal prediction with response-free balancing weights, which make the weighted calibration covariates match the target covariates under a characteristic function distance (covering kernel MMD and energy distance). The authors prove a finite-sample bound on the error of the estimated target score CDF, decomposed into covariate imbalance, score-approximation error, target-sampling error, and an effective-sample-size term. Experiments on synthetic, semi-synthetic, and Parkinson’s data show coverage broadly comparable to flexible estimated-ratio WCP.

Critical strengths
  • The paper gives a clean way to think about when reweighting works under covariate shift: the coverage error of any weighting method depends mainly on how well the reweighted calibration data match the target data and how uneven the weights are.
  • The spectral analysis shows that the kernel choice determines which shifts can be corrected at all, and that a poorly suited kernel can miss a fine-grained shift while the weights still look normal.
Critical weaknesses
  • The main weakness is the novelty of the method compared to WCP and KMM-CP. Methodologically, DB-CC is very close to KMM-CP, which also uses kernel balance in CP. The differences listed in Appendix C.2 are mostly about the analysis rather than the method, and KMM-CP isn’t included as a baseline. Also, DB-CC keeps the weighted-quantile structure of WCP and only changes how the weights are obtained.
  • It is hard to see when I should use DB-CC over estimated-ratio WCP: in Table 1 the two have nearly identical coverage in every setting, and on the diabetes data DB-CC does slightly worse than linear WCP, with only Parkinson’s showing a modest gain. I would also have liked comparisons with KMM-CP directly.
Questions for authors
  • How can we detect failure (miscoverage) without oracle quantities? As a user, we may get miscoverage but there is no way of checking it exactly. For split CP and WCP with a known density ratio, the guarantee is exact so the failure detection is not necessary. However, the coverage guarantee for DB-CC is approximate. How do we quantify potential miscoverage here?
  • Some conditional CP targeting methods, such as Gibbs et al. (2025) "Conformal Prediction With Conditional Guarantees", guarantee exact coverage over all covariate shifts in a chosen function class. With a kernel class matching DB-CC’s balancing geometry, how do the two compare?
3 comments

We thank the reviewer 5kbF for asking what DB-CC adds to existing calibration methods and what practitioners can conclude from its guarantee.

Contribution and relationship to KMM-CP. Gaussian DB-CC coincides with regularized Gaussian KMM calibration when the kernel, normalization, penalty, constraints, and quantile rule are matched. Our contribution is the calibration-level analysis: a score-CDF error decomposition for general balance–transfer pairs, its optimization-objective consequence, a response-free population-balance selection guarantee, and an empirical-optimizer-to-coverage spectral result.

In particular, Theorem 2 proves limiting coverage of 0.86 versus 0.90 for fixed Gaussian and Matérn geometries along an increasing-frequency sequence with an unchanged bounded density-ratio distribution. Nearly uniform fitted weights can therefore leave persistent undercoverage. Joint bandwidth–regularization tuning restores Gaussian coverage to 0.8983 (Table 13), demonstrating how the resolution–regularization mechanism can guide recovery.

Practical comparisons. The implementation linked in KMM-CP was inaccessible when preparing the comparison, so we implemented its nonselective Gaussian component under matched settings. Section 5.6 and Appendix G.7 standardize data splits, tuning grids, criterion form, and quantile convention. Validation uses each candidate's geometry, followed by refitting; this procedure is distinguished from Theorem 3's common-geometry, no-refit rule.

Among 50 attempted shared-grid repetitions, the successful paired comparisons contain 49 Product-Laplace pairs and 50 Matérn pairs. Their five-level mean within-repetition absolute coverage errors are 11.67 and 12.49 percentage points, versus 13.43 and 13.26 for Gaussian on the respective paired sets. The reductions are 1.76 [0.70, 2.83] and 0.77 [0.10, 1.43] percentage points, with pointwise 95% Monte Carlo t-intervals (Table 24). Both increase ESS; the length-difference intervals include zero. These results identify setting-dependent benefits. Linear WCP remains effective under the linear diabetes shift, and we do not claim uniform superiority over estimated-ratio WCP.

Section 5.5 separately evaluates interval informativeness. Oracle WCP covers 0.901 with 65.1% infinite intervals, while DB-CC covers 0.914–0.920 with finite intervals among successful fits. Table 21 reports uncertainty and success counts of 98/100, 99/100, and 100/100 for the three kernels. The revised text identifies this as an informativeness–guarantee trade-off: oracle WCP retains its exact marginal guarantee, whereas DB-CC's coverage remains assumption-dependent. Finiteness is also shared by matched nonselective KMM calibration.

Q1. Quantifying potential miscoverage. Under Theorem 1's conditions, with probability at least 1−δ1-\delta over calibration and unlabeled target samples, Corollary 1 gives

\leq \alpha+\Delta_{\delta,D}(\pi).$$ Empirical discrepancy and ESS are observable, and the stated concentration conditions supply a target-sampling radius. A numerical coverage certificate additionally requires valid bounds on the approximation envelope for the conditional score probabilities. Small observed imbalance and high ESS alone cannot certify coverage. Appendix D.6 and Section 6 now make this interpretation explicit. When a valid bound on the complete error term is available, Corollary 1 gives the corresponding quantile-level adjustment on the same high-probability event. **Q2. Comparison with Gibbs et al. (2025).** Their randomized finite-dimensional construction gives exact finite-sample coverage over its class of covariate shifts. Their regularized infinite-dimensional/RKHS construction instead characterizes coverage through a regularization-dependent term; using a full Gaussian RKHS does not inherit the finite-dimensional exact guarantee. The function spaces also represent different objects: covariate tilts in Gibbs et al., versus conditional score probabilities $h_s(x)$ in our balance–transfer condition. DB-CC uses unlabeled covariates from one specified target population to fit weights and estimate its score CDF. Matching the kernel therefore does not equate the assumptions, calibration constructions, or guarantees. The approaches address complementary calibration targets.

Dear Reviewer 5kbF,

Thank you again for your time and thoughtful review. We have provided a detailed response to your comments and concerns in our rebuttal and revised the paper (in blue).

We would greatly appreciate it if you could take a moment to review our responses and let us know whether they adequately address your concerns. If any questions or concerns remain, we would be happy to provide further clarification.

Thank you again for your time and consideration.

I thank the authors for the added comparison and the candid clarifications. However, the response confirms that DB-CC coincides with KMM calibration and the empirical gains are modest and setting-dependent, so the practical case for the method over KMM-CP and WCP remains limited. I appreciate the analytical and empirical contributions, but I will keep my score.

Reviewer_5KcZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies conformal calibration under covariate shift using distributional balancing weights. Building on KMM-CP, it develops a general coverage-error bound and a label-free procedure for selecting weighting configurations. Its spectral analysis shows how kernel choice and regularization can cause undercoverage despite high effective sample size. Synthetic and semi-synthetic experiments examine these effects and recovery through tuning.

Critical strengths
  1. The paper connects distributional balance to calibration error. Building on the prior work, the paper brings a generalized perspective in conncting distribution mismatch error to calibration error.
  2. The spectral analysis identifies a useful failure mode. The paper shows that nearly uniform weights and high ESS can coexist with persistent undercoverage when the kernel fails to resolve score-relevant shifts.
  3. The controlled experiments support the proposed mechanism. The circular example reproduces the predicted Gaussian-Matérn separation, and bandwidth–regularization tuning removes the gap.
Critical weaknesses
  1. The experiments do not clearly establish the incremental benefit over KMM-CP. For integrable spectra, the proposed discrepancy is MMD with the corresponding kernel, and the weighting objective closely follows regularized KMM. A comparison with KMM-CP under matched bandwidth selection, regularization, weight constraints, and calibration conventions is needed to isolate the benefit of the proposed approach
  2. The recovery experiment primarily demonstrates the importance of tuning. Tuning Gaussian bandwidth and regularization raises coverage from 0.8608 to 0.8983, essentially matching tuned Matérn. This supports the spectral explanation, but does not establish an advantage over conventional KMM with comparable tuning.
  3. The experimental settings are relatively simple and narrow. The main studies use a one-dimensional circular construction, five-dimensional synthetic covariates, and two tabular datasets with artificially induced shifts. Performance with high-dimensional covariates, learned representations, or naturally occurring shifts remains untested.
  4. The broader empirical improvements are limited. Selected DB-CC performs comparably to existing nonlinear corrections on the interaction shift, while all learned methods fail on the harder checkerboard shift. The diabetes experiment does not establish an advantage over Gaussian balancing or flexible WCP. These results leave the practical significance of the additional methodology unclear.
Questions for authors
  1. Can the authors compare against KMM-CP with matched bandwidth selection, regularization, weight constraints, and calibration conventions? What improvement remains after controlling for these choices?
  2. Since tuning Gaussian bandwidth and regularization largely removes the spectral coverage gap, what additional practical benefit does the proposed method provide over similarly tuned Gaussian KMM?
  3. Can the evaluation include higher-dimensional covariates or learned representations and naturally occurring distribution shifts? The current experiments provide limited evidence about performance beyond controlled, relatively simple settings.
  4. How sensitive is the proposed selection procedure to the common validation kernel and candidate library? Can it reliably select configurations with better coverage when smaller validation MMD does not imply smaller calibration error?
2 comments

We thank the reviewer 5KcZ for recognizing the spectral failure mode and requesting comparisons that distinguish balancing geometry from tuning and implementation.

Q1. Matched comparison with KMM-CP. The implementation linked in KMM-CP was inaccessible when preparing this comparison. We therefore implemented its nonselective Gaussian KMM calibration component under a matched protocol. Matching the Gaussian kernel, normalization, regularization, constraints, and quantile convention gives the same objective, weights, and thresholds as Gaussian DB-CC. Section 5.6 and Appendix G.7 make this correspondence explicit.

The added Parkinson's comparison uses common data splits, tuning grids, and the same response-free selection criterion form across methods. Within each family, selection minimizes validation CFD plus 0.01/ESS0.01/\sqrt{\mathrm{ESS}}, evaluated using each candidate's kernel and bandwidth, followed by refitting on the full unlabeled target sample. This is a standardized within-family comparison, distinct from Algorithm 1's common-geometry, no-refit selection.

For 50 attempted shared-grid repetitions, Table 24 reports:

Alternative minus matched GaussianAbsolute coverage-error difference (percentage points)Mean length differenceESS differenceSuccessful pairs
Product-Laplace-1.76 [-2.83, -0.70]0.18 [-0.19, 0.55]10.4 [6.9, 13.8]49
Matérn-3/2-0.77 [-1.43, -0.10]-0.06 [-0.35, 0.23]3.1 [1.8, 4.4]50

Intervals are pointwise 95% Monte Carlo t-intervals. Absolute errors are averaged over nominal levels 0.50–0.90 within each repetition before forming paired contrasts. On the respective paired sets, Gaussian's mean errors are 13.43 and 13.26 percentage points, compared with 11.67 and 12.49. Thus, alternative geometries reduce repeated-sample calibration error in this setting; the length intervals include zero.

Q2. What remains after comparable tuning? Tuned Gaussian and Matérn achieve essentially identical circular-model coverage: 0.8983 and 0.8980 (Table 13). This supports the mechanism in Theorem 2. The theorem connects empirical constrained optimization to limiting coverage of 0.86 versus 0.90 while the oracle-ratio distribution and overlap remain unchanged. Equation (11) identifies spectral mode strength relative to regularization as the governing quantity; adapting bandwidth changes that ratio and restores Gaussian calibration.

This is an increasing-frequency sequence in which both the shift and conditional response law vary; both kernels can be consistent at fixed frequency. Together, the theorem and recovery experiment establish a failure mechanism and a controlled remedy, rather than an intrinsic ranking of kernel families. The matched Parkinson's results provide complementary evidence that geometry can still matter under standardized tuning.

Q3. Empirical scope. The added experiments extend shift severity within the Parkinson's cohort. They do not constitute high-dimensional, learned-representation, or natural-shift benchmarks. Our evidence concerns controlled synthetic shifts and induced shifts in fixed tabular cohorts. The controlled synthetic designs isolate shift structure while preserving the conditional response law; broader deployment performance remains to be evaluated.

Q4. Validation kernel and candidate library. Theorem 3 guarantees population balance in the fixed validation geometry. Its coverage consequence additionally requires approximation of the conditional score probabilities and weight-concentration control. Appendix E.1 explains that changing the kernel mixture can change the approximation envelope, even without a candidate-wise union-bound penalty.

Table 11 isolates library sensitivity while fixing the validation kernel. At n=1,000n=1{,}000, expanding Gaussian-only to the three-family library changes mean absolute coverage error by -0.002, -0.082, and -0.077 percentage points for the interaction and two checkerboard settings; the respective pointwise 95% intervals are [-0.048, 0.044], [-0.145, -0.020], and [-0.111, -0.043]. These small effects address candidate-library sensitivity, not stability across validation kernels. We interpret the selector as a response-free balance criterion with an explicit transfer condition, rather than a guarantee of coverage-optimal selection.

Dear Reviewer 5KcZ,

Thank you again for your time and thoughtful review. We have provided a detailed response to your comments and concerns in our rebuttal and revised the paper (in blue).

We would greatly appreciate it if you could take a moment to review our responses and let us know whether they adequately address your concerns. If any questions or concerns remain, we would be happy to provide further clarification.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

general response

We thank the reviewers for recognizing the value of linking distributional balance to calibration and identifying undercoverage despite high effective sample size (ESS). The revision foregrounds our central contribution: an analysis connecting empirical balancing, response-free selection, and spectral resolution to actual target coverage. All (major) changes are highlighted in blue.

Theorem 1 separates achieved imbalance, score approximation, target-sample uncertainty, and weight concentration. Corollary 2 connects these quantities to the optimization objective. Theorem 3 establishes a population-balance oracle inequality in a fixed validation geometry. Theorem 2 then proves a coverage separation for empirical constrained optimizers under bounded shifts with an unchanged oracle-ratio distribution; the adaptive experiment demonstrates recovery, including for Gaussian balancing.

Section 5.6 and Appendix G.7 add a matched implementation of KMM-CP's nonselective Gaussian calibration component. Under shared-grid tuning, Product-Laplace and Matérn-3/2 reduce the five-level mean within-repetition absolute coverage error by 1.76 and 0.77 percentage points. Table 24 reports paired uncertainty, successful-pair counts, length differences, and ESS differences. The appendix now specifies the candidate-specific validation criterion and post-selection refitting, distinguishing this comparison from the common-geometry procedure analyzed in Theorem 3.

Section 5.5 and Appendix G.6 separately examine finite versus infinite intervals under severe induced shift, with Monte Carlo standard errors and fitting-success counts. The interpretation explicitly distinguishes empirical interval informativeness from the different coverage guarantees.

Appendices D.6 and E.1 clarify what observed balance and ESS can establish and the additional approximation conditions required for coverage. All section and table references in our responses refer to the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.