acceptodds
Withdrawn from ICLR 2027

Injective Recalibration for Canonical Calibration Under Covariate Shift Without Labels

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Estimating canonical calibration error (CE) on a target domain without target labels is a challenge in deployed machine learning systems facing covariate shift. We study this problem on the probability simplex, where a Dirichlet-kernel density estimator yields a consistent surrogate for the source-domain conditional that can be evaluated on unlabeled target predictions. Our central observation is that the resulting target CE admits a triangle-inequality upper bound into two terms: a tractable term that depends on the recalibration map and is estimable from labeled source data, and a residual term that captures the source–target drift of the prediction-space label conditional. We show that injectivity of is sufficient to make the residual mismatch term independent of the choice of recalibration map within the injective family, providing a principled justification for restricting label-free recalibration to maps such as temperature scaling. We further demonstrate empirically that the injectivity of the base classifier's representation is associated with the source-to-target estimation gap of the Dirichlet-KDE estimator under rotation shift on MNIST. Finally, we introduce a label-free overlap diagnostic that detects regions of the target prediction distribution lying outside the source support, where absolute continuity fails and the surrogate becomes unreliable. Across MNIST and SVHN under strict reweighting-induced covariate shift, CIFAR-10-C, CIFAR-10CINIC-10, Camelyon17-WILDS, and OfficeHome, the proposed method matches or improves upon SOTA methods on most settings; we additionally characterize the regime—severe shift with collapsed support overlap—where all source-only methods, including ours, fail, and argue that the overlap diagnostic should be reported alongside any such CE estimate.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Calibration, Calibration under Distribution Shift, Covariate Shift, Post-hoc Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7NsAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces label-free calibration error estimation under covariate shift. It uses Dirichlet-KDE to estimate label conditionals in prediction space. Theoretical analysis motivates injective recalibration maps such as temperature scaling. An overlap diagnostic identifies unreliable estimates caused by insufficient source support. Experiments verify competitive performance while revealing failures under severe support mismatch.

Critical strengths
  1. Post-hoc calibration under domain shift scenarios is a very important problem.

  2. Using the probability estimated by the classifier is a commonly used method, and I agree with this technique.

  3. The proposed method is effective compared with some traditional methods.

Critical weaknesses
  1. The novelty of the proposed method appears limited. Although the authors discuss its differences from importance-weighting methods, these distinctions do not sufficiently establish a substantive methodological contribution.

  2. The experimental evaluation is limited to CPCS and TransCal, both published in 2020. Comparisons with more recent and competitive baselines are needed to substantiate the effectiveness of the proposed method.

  3. The experimental setup requires further clarification. The authors should specify which underlying domain adaptation method, if any, is used and which model architectures are evaluated for calibration.

  4. The motivation for using KDE is insufficiently developed. Further analysis is needed to explain its suitability for this task and its advantages over alternative conditional probability estimators.

Questions for authors

The paper’s formatting requires improvement, particularly the sizing and layout of figures and tables, to ensure readability and visual consistency. I recommend polishing the overall paper carefully.

Reviewer_7WmSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies classifier calibration under domain shift without target labels. It uses labeled source data and unlabeled target predictions to construct a calibration objective and select a temperature. Experiments cover synthetic and real domain shifts.

Critical strengths
  • The method does not require target labels or retraining the classifier.

  • The experiments cover several domain-shift settings and also report failure cases under severe shifts.

Critical weaknesses
  • The baseline comparison is too limited to show that the method is competitive. The main experiments compare only CPCS and TransCal, both from 2020, and Oracle TS, which uses target labels. The related-work section discusses more recent label-free methods, including Tomani et al. (2023), Zou et al. (2023), Hu et al. (2024), and Penso and Goldberger (2025), but none are evaluated and no reason is given for excluding them.

  • The benefit of coverage weighting is not demonstrated. The paper does not directly compare the final method with its unweighted version, nor does it systematically report no-recalibration or standard source-validation temperature scaling baselines across datasets. It is therefore unclear whether the gains come from coverage weighting or simply from selecting a different temperature.

Questions for authors

Why are only 8 of the 15 CIFAR-10-C corruption types evaluated, and would the conclusions hold across all 15?

Reviewer_AAgBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies recalibration using labeled source data and unlabeled target examples. For injective recalibration maps, an upper bound on target canonical calibration error separates a source-conditional matching term from a conditional-drift residual that is independent of the map. The implementation estimates the source label conditional with Dirichlet-kernel regression and fits a temperature using a coverage-weighted loss on target predictions. It also reports uncovered target prediction mass as a warning signal. Experiments include controlled reweighting, corruption shifts and cross-domain image classification, with CPCS, TransCal and a target-label oracle as the main comparisons.

Critical strengths
  1. The distinction between injectivity of the recalibrator and information lost by the classifier is useful. The derivation in Appendix A makes clear why the conditional-drift term remains fixed within the injective family. This gives a readable account of what source labels can support and what remains unidentified.

  2. The paper includes meaningful negative results. The contrast-severity-5 example and Camelyon17 Center 2 show that the proposed objective can fail alongside existing methods. Reporting these cases, rather than only the settings with good overlap, helps establish the practical scope of the approach.

Critical weaknesses
  1. The experiments do not show whether coverage weighting improves on the objective motivated by the bound. Equation (4) calls for minimizing A, but all experiments use A_cov. Appendix B explains that weighting changes the quantity being minimized and gives the covariance term relating the two. A comparison of Equations (6) and (7) is needed to see whether this change helps. The main comparisons also lack source-fitted temperature scaling and the uncalibrated model. Without these baselines, it is hard to tell how much is gained by using target predictions and coverage weights, beyond fitting a temperature with source labels.

  2. I could not find enough detail to reproduce the overlap results in Table 1. Section 4.4 and Appendix I do not say how the source-coverage score is normalized or which cutoff is used for individual predictions. The reported κ values apply to the fraction of uncovered predictions, so they do not supply this missing cutoff. It is also unclear how the thresholds were chosen and checked. Changing κ from 0.20 to 0.25 reduces detection from 13/14 to 5/14 bad conditions. Would the same thresholds work under a different shift? The KDE subsample size, bandwidth grid and temperature bounds are also missing. Equation (8) includes a regularizer, but Algorithm 1 leaves it out.

Questions for authors
  1. Can you compare Equation (6), the coverage-weighted loss, source-only temperature scaling and T=1 on the same held-out target samples? This would help show whether weighting helps across settings, including when source and target overlap is poor.

  2. Which coverage score and cutoff were used for Table 1, and how were the thresholds selected? Please also provide the missing optimization settings and explain the regularizer difference between Equation (8) and Algorithm 1. Testing the diagnostic on a separate shift family with the thresholds fixed would help show whether the results carry over.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.