- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces label-free calibration error estimation under covariate shift. It uses Dirichlet-KDE to estimate label conditionals in prediction space. Theoretical analysis motivates injective recalibration maps such as temperature scaling. An overlap diagnostic identifies unreliable estimates caused by insufficient source support. Experiments verify competitive performance while revealing failures under severe support mismatch.
Critical strengths
-
Post-hoc calibration under domain shift scenarios is a very important problem.
-
Using the probability estimated by the classifier is a commonly used method, and I agree with this technique.
-
The proposed method is effective compared with some traditional methods.
Critical weaknesses
-
The novelty of the proposed method appears limited. Although the authors discuss its differences from importance-weighting methods, these distinctions do not sufficiently establish a substantive methodological contribution.
-
The experimental evaluation is limited to CPCS and TransCal, both published in 2020. Comparisons with more recent and competitive baselines are needed to substantiate the effectiveness of the proposed method.
-
The experimental setup requires further clarification. The authors should specify which underlying domain adaptation method, if any, is used and which model architectures are evaluated for calibration.
-
The motivation for using KDE is insufficiently developed. Further analysis is needed to explain its suitability for this task and its advantages over alternative conditional probability estimators.
Questions for authors
The paper’s formatting requires improvement, particularly the sizing and layout of figures and tables, to ensure readability and visual consistency. I recommend polishing the overall paper carefully.