- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates a relevant problem in calibration literature: most models and calibration approaches leverage a cross-entropy objective but a cross-entropy vanishing gradient does not necessarily lead to calibrated probabilities. In the paper, authors resort to a notion of calibration know as projected smooth calibration which evaluates calibration residuals against a class of Lipschitz-1 function of linear projections of the output probabilities. These are termed witnesses.
First, authors characterize the finite sample variability induced in the calibration error by evaluation with acces to limited data. Thus, even perfectly claibrated models can appear miscalibrated.
Second, authors formally isolate when loss stationarity implies calibration. The condition they derive is that the indicator of each group of instances sharing the same probabilistic output can be identified via a linear combination of the features available to the linear head.
Third, authors extend this analysis to finte training and derive a bound decomposable in three interrpetable terms: gradient control, , the error induced by not reaching loss-stationarity, witness approximation, , the error induced by not recovering witnsses perfectly via linear projections of the feature space, finite sample variability, , the error induced by evaluating calibration with a limited number of samples.
Finally, authors validate their theoretical analysis with a CNN on Cifar10.
Critical strengths
I consider the analysis carried in the paper relevant and insightful. First, better understanding the relationship between features and calibration erros is an open research direction in the field. Second, it is commonly assumed that minizing the corss-entropy loss, it being a stricly proper scoring rule, would lead, at least in the limit, to perfect calibration. I therefore consider of value the considerations of the paper that relate the properties of gradient based optimization of the aforementioned objective with the resulting absence of a calibration guarantee. Moreover the theoretical results truly offer a better understanding of the phenomenon and the interpretale decomposition of the bound in theorem 3 could be of value for practitioners or researchers working on calibration.
Critical weaknesses
ALthough the paper idea deserves praising, I have several concerns.
My first concern is clarity. It is especially difficult to obtain a deep understanding of the results presented in the paper. Authors could really improve the presentation by introducing the scope and the relevant take-aways of each theoretical contribution before presenting it. This is particularly important because the theoretical results carry a non-negligible depth of information but the reader is often left to infer it from a rather dry text and very dense notation instead of being introduced to it. This is particularly problematic in section 6 which is too proof oriented for its own readability. The notation is also very heavy where it could be substantially slimmed in various parts of the text. For instance, line 188, is introduced but to me it seems to be used only once. It could just be absorbed by since the ones cancel leaving . Many of such changes would really help reducing the congitive load on the reader, which is currently substantial.
I also have concerns with regard to evaluation in the paper. I understand that the analysis is only verified with a and data. ALthough this provides a starting point, the message of the paper naturally raises the question of whether different representations would yield differet results. I therefore consider the evaluation heavily limited. I consider crucial to perform the same evaluation with at least strong type of ( or are two suitable options) and with a . The latter being extremely relevant as it induces a different inductive bias and therefore allowing to conclude whether the result apply also to representations with a foudnamentally different geometric structure. Moreover the paper works in the multiclass setting, so an important experiment would be to test if the results scale with the number of classes leveraging .
My last concern regards the paper positioning and motivation. In its current state the paper is not properly motivating why accessing representations would be a relevant iea to explore. Indeed, one could even dismiss the paper claiming that classical post-hoc calibration techniques do not have access to representations and the analysis is therefore of no practical use as it applies to components practicioners cannot act on. I strongly recommend authors to motivate their work through the fields of local calibration [2-7] and multicalibration [9-12] (authors themselves acknowledge the connection with multicalibration but do not cite a substantial portion of relevant literature). These fields are deeply connected to the paper because, motivated by the failures of global calibration notions [1], they learn very expressive calibration maps acting on both features (multicalibration) or representation spaces of neural networks (local calibration) rather than logits. Moreover, the analysis of the paper is extremely relevant because the developped methodologies often rely on NLL-based objectives. I think authors should significantly strenghten the paper motivation explicitely mentioning that calibration from representation space/features is a very recent and promising development, possibly acknowledging the correspoinding literature.
REFERNCES
[1] Xiong, Miao, et al. "Proximity-informed calibration for deep neural networks." NEURIPS23
[2] Luo, Rachel, et al. "Local calibration: metrics and recalibration." Uncertainty In UAI22.
[3] Barbera, Cesare, et al. "Multiclass local calibration with the Jensen-Shannon distance." In AISTATS26.
[4] Ding, Zhipeng, et al. "Local temperature scaling for probability calibration." In ICCV21.
[5] Valk, Kaspar, and Meelis Kull. "Assuming locally equal calibration errors for non-parametric multiclass calibration." In TMLR23
[6] Lin, Zhen, Shubhendu Trivedi, and Jimeng Sun. "Taking a step back with kcal: Multi-class kernel-based calibration for deep neural networks." In ICLR23.
[7] Barbera, Cesare, et al. "Divide et Calibra: Multiclass Local Calibration via Vector Quantization." arXiv preprint (2026).
[8] Jung, Christopher, et al. "Moment multicalibration for uncertainty estimation." In Conference on Learning Theory. PMLR, 2021.
[9] Baldeschi, Riccardo Colini, et al. "Multicalibration yields better matchings." In ICML26
[10] Jin, Hongyi Henry, et al. "Discretization-free multicalibration through loss minimization over tree ensembles." In NEURIPS25
[11] Tax, Niek, et al. "Mcgrad: Multicalibration at web scale." In SIGKDD26
[12] Hébert-Johnson, Ursula, et al. "Calibration for the (computationally-identifiable) masses." arXiv preprint (2017).
Questions for authors
I would kindly ask authors to address my concerns in the weaknesses section.
2 comments
I appreciate the authors rebuttal. I tried to download the new pdf, but the changes are very hard to track. Before I evaluate their response, I would kindly ask authors to highlight the changes (text of a different color, also for captions of new tables and images, is a quite standard pratice)
Thank you for recognizing the value of our representation analysis and the decomposition in Theorem 3, and for the suggestions on presentation and related work. We have revised the manuscript to make the contribution more explicit: we characterize what optimizing an existing multiclass linear head can certify about calibration, and identify the representation properties that make this inference valid.
Why access to representations is central to the problem: Our setting directly models linear probing, where practitioners freeze an encoder and train a classification head on its features. Those features determine which residual patterns the cross-entropy gradient controls. Access to them therefore lets us evaluate the representation component of a calibration certificate after training. This motivation applies to the model being optimized, independently of the choice of a subsequent recalibration method. The local and feature-aware methods you highlighted provide an additional practical motivation, which we now develop in the Introduction and Related Work.
What our analysis adds to the suggested literature: The cited works establish important calibration results for specific estimators, objectives, or corrective function classes. For example, KCal constructs a kernel-based predictor on learned embeddings and derives calibration guarantees under density-estimation assumptions. Barbera et al.'s Jensen–Shannon approach combines alignment to estimated local class frequencies with a cross-entropy term on those estimates. Jin et al. obtain multicalibration through squared-loss minimization over tree ensembles under a loss-saturation condition, while MCGrad iteratively fits corrections using features and current predictions. These results concern the corresponding recalibration procedures and their assumptions; they do not make population stationarity in an arbitrary fixed linear head sufficient for calibration.
Our contribution identifies exactly the feature condition needed for that implication. Theorem 2 gives a necessary and sufficient criterion on finite support, with the input distribution, features, and predictor fixed and the implication quantified over conditional label laws. Theorem 3 then quantifies nonzero gradients, imperfect witness representation, and sampling uncertainty, while Proposition D.1 makes a full-class approximation upper bound computable. Thus, the use of a cross-entropy objective alone is not the guarantee: the available representation determines what its stationary gradient can establish. We clarify this distinction in Related Work and Sections 5–6. We also distinguish the targets: our witnesses depend on predicted probabilities, whereas multicalibration additionally tests input-defined subgroups. These connections strengthen the motivation while keeping the proved guarantees precise.
Clarity and notation: We have made the analytical message easier to follow before introducing its notation. The Introduction and Sections 4–6 now state each result's scope and main implication before the formal statement. In Section 6, we explain how the gradient controls represented residuals, how witness approximation controls the remainder, and how sampling uncertainty enters population transfer. The detailed proof appears in Appendix C. Following your specific notation suggestion, we define the witness-complexity factor directly in Section 4, organize the notation in Appendix H, and use Figure 1 to connect stationarity, representation, and the anchor construction.
Representation families and the number of classes: The analysis permits nonlinear frozen representations; the linearity assumption concerns the head. To examine how the numerical bounds depend on representation geometry, we added ImageNet-pretrained ResNet-50 and ViT-B/16 on CIFAR-100, using the same 2,000 training and 1,000 test images (Section 7.5 and Appendix G.7). Tables 13–14 report accuracy, model sizes, empirical certificates, their component terms, and runtimes. Tables 2 and 17 extend the projected-witness comparison through m=100, with conditional-null diagnostics to interpret the observed scores. Appendix G.6 separately isolates feature dimension through nested representations. These studies address the proposed CNN/transformer and class-count comparisons and show how representation quality affects the quantities appearing in the theory.