- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies conformal calibration under covariate shift, where the conditional response law is unchanged but source and target covariate distributions differ. The authors propose learning distributional balancing weights to align source calibration covariates with unlabeled target covariates, then calibrating from the weighted score distribution, with characteristic-function distances unifying several balancing criteria. On the theory side, they discuss error decompositions for the target score distribution, approximate coverage, and links between kernel geometry/regularization and coverage; experiments include controlled synthetic settings and some semi-synthetic/real comparisons.
Critical strengths
- Emphasizing that bounded ratios or controlled weight concentration need not imply that the shift has been corrected is useful and interesting for the community. Besides, it explain the importance of kernel geometry/regularization via theory and experiments.
- Attempting to separate imbalance, approximation error, sampling, and weight concentration makes different balancers easier to compare and strengthens credibility.
Critical weaknesses
- The method positioning is unclear for now. The paper seems to present DB-CC largely as a common interface over existing response-free weighted rules, while KMM-style weighted calibration already exists.
- Besides, the main experiments do not appear to include a direct comparison with KMM-CP. At the same time, relative to the stronger baselines in the paper, the main results often look close but not stably better.
- On fine-scale and high-frequency shifts, learned methods appear unable to approach the oracle, thus the theory diagnoses a problem that the method does not seem to fix.
Questions for authors
- Under the same protocol and candidate library, what are the coverage / set length / ESS numbers versus KMM-CP (or its closest implementation)? If the gap is small, how should the algorithmic contribution be separated from the spectral/diagnostic analysis?
- Relative to Selected flexible WCP / quadratic WCP, are the main-table differences stable and better?
We thank the reviewer pcRU for recognizing the importance of separating spectral resolution from weight concentration and for requesting matched comparisons with paired uncertainty.
Q1. Matched Gaussian comparison and the contribution beyond it. The implementation linked in KMM-CP was inaccessible when preparing this comparison. We therefore implemented its nonselective Gaussian KMM calibration component under matched settings. Matching the kernel, normalization, regularization, constraints, and quantile rule yields the same objective, weights, and thresholds as Gaussian DB-CC. The revision makes this correspondence explicit in Section 5.6 and Appendix G.7.
The synthetic comparison fixes preprocessing, bandwidth/ridge grids, cap, ESS screen, uniform fallback, common validation geometry, and quantile convention, varying only the candidate kernel library. At n=1,000, Table 9 reports:
The incremental effects are small in these settings. The added severe-shift Parkinson's comparison standardizes data splits, tuning grids, and the response-free criterion form. Each family minimizes candidate-specific validation CFD plus 0.01/ESS, then refits on the full unlabeled target sample. Appendix G.7 distinguishes this protocol from Algorithm 1's common-geometry, no-refit selection.
Table 24 reports the following shared-grid differences, alternative minus matched Gaussian:
There were 50 attempted repetitions. Intervals are pointwise 95% Monte Carlo t-intervals. Absolute coverage errors and lengths are averaged across five nominal levels within each repetition; ESS is computed once per fit. This supports reduced repeated-sample calibration error in this experiment, with uncertainty about average length changes.
The analytical contribution is distinct from a new Gaussian weighting objective: Theorem 1's four-part score-CDF bound, Corollary 2's objective-value connection, Theorem 3's population-balance oracle inequality, and Theorem 2's explicit coverage separation for empirical constrained optimizers.
Q2. Stability relative to flexible and quadratic WCP. The benefits are regime-dependent. For DB-CC minus selected flexible WCP at n=1,000, Table 10 gives these paired absolute-coverage-gap differences in percentage points:
At n=300, DB-CC has larger mean absolute gaps for both checkerboards. Quadratic WCP is competitive for the interaction shift: at n=1,000, its mean absolute gap is 0.0123, versus 0.0130 for DB-CC (Table 9). We therefore report the favorable and unfavorable regimes rather than claim consistent superiority.
Fine-scale failure and recovery. The higher-frequency checkerboard remains unresolved: at k=2, n=1,000, learned procedures cover approximately 87%, while oracle WCP remains near 90%. Section 6 states this limitation explicitly. The circular experiment separately demonstrates recovery: tuning raises Gaussian coverage from 0.8608 to 0.8983, matching tuned Matérn at 0.8980 (Table 13). Together, these results establish a failure mechanism and a controlled remedy, while identifying a setting where the tested libraries still have insufficient effective resolution.