acceptodds
Under review as a conference paper at ICLR 2027

Multiclass Projected Calibration: Estimation Limits and Cross-Entropy Stationarity

Abstract

When training linear classification heads on frozen feature representations, a vanishing cross-entropy gradient signals optimization progress but does not, by itself, guarantee calibrated probabilities. We identify exactly when a zero population cross-entropy gradient guarantees multiclass calibration for finite-support input distributions. We show that this guarantee holds for every conditional label distribution if and only if linear combinations of the features can isolate each group of inputs assigned the same probability vector. If they cannot, even a global population minimizer can remain miscalibrated. We then move from exact stationarity to finite training and derive a high-probability certificate for projected smooth calibration, a multiclass calibration metric that tests differences between labels and predicted probabilities using bounded Lipschitz functions of one-dimensional probability projections. The certificate separates optimization error, representation error from calibration tests that the frozen features cannot accurately represent, and finite-sample uncertainty. We also derive a computable upper bound on this representation error using only the model predictions and frozen features. Finally, we characterize the finite-sample uncertainty incurred when calibration is evaluated on held-out data and construct an exactly calibrated predictor whose empirical score matches the scaling of our upper bound up to logarithmic factors. Synthetic controls and frozen CNN features on CIFAR-10 illustrate how feature representation affects calibration, the finite-training certificate, and the statistical limits of empirical calibration auditing.

Primary area:
learning theory
Keywords:
multiclass calibration, projected smooth calibration, probability calibration, cross-entropy, gradient descent, learning theory, uncertainty quantification, representation learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_9wAiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates a relevant problem in calibration literature: most models and calibration approaches leverage a cross-entropy objective but a cross-entropy vanishing gradient does not necessarily lead to calibrated probabilities. In the paper, authors resort to a notion of calibration know as projected smooth calibration which evaluates calibration residuals against a class of Lipschitz-1 function of linear projections of the output probabilities. These are termed witnesses.

First, authors characterize the finite sample variability induced in the calibration error by evaluation with acces to limited data. Thus, even perfectly claibrated models can appear miscalibrated.

Second, authors formally isolate when loss stationarity implies calibration. The condition they derive is that the indicator of each group of instances sharing the same probabilistic output can be identified via a linear combination of the features available to the linear head.

Third, authors extend this analysis to finte training and derive a bound decomposable in three interrpetable terms: (i)(i) gradient control, i.e.i.e., the error induced by not reaching loss-stationarity, (ii)(ii) witness approximation, i.e.i.e., the error induced by not recovering witnsses perfectly via linear projections of the feature space, (iii)(iii) finite sample variability, i.e.i.e., the error induced by evaluating calibration with a limited number of samples.

Finally, authors validate their theoretical analysis with a CNN on Cifar10.

Critical strengths

I consider the analysis carried in the paper relevant and insightful. First, better understanding the relationship between features and calibration erros is an open research direction in the field. Second, it is commonly assumed that minizing the corss-entropy loss, it being a stricly proper scoring rule, would lead, at least in the limit, to perfect calibration. I therefore consider of value the considerations of the paper that relate the properties of gradient based optimization of the aforementioned objective with the resulting absence of a calibration guarantee. Moreover the theoretical results truly offer a better understanding of the phenomenon and the interpretale decomposition of the bound in theorem 3 could be of value for practitioners or researchers working on calibration.

Critical weaknesses

ALthough the paper idea deserves praising, I have several concerns.

My first concern is clarity. It is especially difficult to obtain a deep understanding of the results presented in the paper. Authors could really improve the presentation by introducing the scope and the relevant take-aways of each theoretical contribution before presenting it. This is particularly important because the theoretical results carry a non-negligible depth of information but the reader is often left to infer it from a rather dry text and very dense notation instead of being introduced to it. This is particularly problematic in section 6 which is too proof oriented for its own readability. The notation is also very heavy where it could be substantially slimmed in various parts of the text. For instance, line 188, Aμ(n)A_\mu(n) is introduced but to me it seems to be used only once. It could just be absorbed by Cμ(n)C_\mu(n) since the ones cancel leaving Cμ(n)=1+12(rlog⁡(en/r))C_\mu(n)=1+ 12(\sqrt{r}\log(en/r)). Many of such changes would really help reducing the congitive load on the reader, which is currently substantial.

I also have concerns with regard to evaluation in the paper. I understand that the analysis is only verified with a CNN\texttt{CNN} and CIFAR-10\texttt{CIFAR-10} data. ALthough this provides a starting point, the message of the paper naturally raises the question of whether different representations would yield differet results. I therefore consider the evaluation heavily limited. I consider crucial to perform the same evaluation with at least strong type of CNN\texttt{CNN} (Res-Net\texttt{Res-Net} or ConvNeXt\texttt{ConvNeXt} are two suitable options) and with a ViT\texttt{ViT}. The latter being extremely relevant as it induces a different inductive bias and therefore allowing to conclude whether the result apply also to representations with a foudnamentally different geometric structure. Moreover the paper works in the multiclass setting, so an important experiment would be to test if the results scale with the number of classes leveraging CIFAR-100\texttt{CIFAR-100}.

My last concern regards the paper positioning and motivation. In its current state the paper is not properly motivating why accessing representations would be a relevant iea to explore. Indeed, one could even dismiss the paper claiming that classical post-hoc calibration techniques do not have access to representations and the analysis is therefore of no practical use as it applies to components practicioners cannot act on. I strongly recommend authors to motivate their work through the fields of local calibration [2-7] and multicalibration [9-12] (authors themselves acknowledge the connection with multicalibration but do not cite a substantial portion of relevant literature). These fields are deeply connected to the paper because, motivated by the failures of global calibration notions [1], they learn very expressive calibration maps acting on both features (multicalibration) or representation spaces of neural networks (local calibration) rather than logits. Moreover, the analysis of the paper is extremely relevant because the developped methodologies often rely on NLL-based objectives. I think authors should significantly strenghten the paper motivation explicitely mentioning that calibration from representation space/features is a very recent and promising development, possibly acknowledging the correspoinding literature.

REFERNCES

[1] Xiong, Miao, et al. "Proximity-informed calibration for deep neural networks." NEURIPS23

[2] Luo, Rachel, et al. "Local calibration: metrics and recalibration." Uncertainty In UAI22.

[3] Barbera, Cesare, et al. "Multiclass local calibration with the Jensen-Shannon distance." In AISTATS26.

[4] Ding, Zhipeng, et al. "Local temperature scaling for probability calibration." In ICCV21.

[5] Valk, Kaspar, and Meelis Kull. "Assuming locally equal calibration errors for non-parametric multiclass calibration." In TMLR23

[6] Lin, Zhen, Shubhendu Trivedi, and Jimeng Sun. "Taking a step back with kcal: Multi-class kernel-based calibration for deep neural networks." In ICLR23.

[7] Barbera, Cesare, et al. "Divide et Calibra: Multiclass Local Calibration via Vector Quantization." arXiv preprint (2026).

[8] Jung, Christopher, et al. "Moment multicalibration for uncertainty estimation." In Conference on Learning Theory. PMLR, 2021.

[9] Baldeschi, Riccardo Colini, et al. "Multicalibration yields better matchings." In ICML26

[10] Jin, Hongyi Henry, et al. "Discretization-free multicalibration through loss minimization over tree ensembles." In NEURIPS25

[11] Tax, Niek, et al. "Mcgrad: Multicalibration at web scale." In SIGKDD26

[12] Hébert-Johnson, Ursula, et al. "Calibration for the (computationally-identifiable) masses." arXiv preprint (2017).

Questions for authors

I would kindly ask authors to address my concerns in the weaknesses section.

2 comments

Thank you for recognizing the value of our representation analysis and the decomposition in Theorem 3, and for the suggestions on presentation and related work. We have revised the manuscript to make the contribution more explicit: we characterize what optimizing an existing multiclass linear head can certify about calibration, and identify the representation properties that make this inference valid.

  • Why access to representations is central to the problem: Our setting directly models linear probing, where practitioners freeze an encoder and train a classification head on its features. Those features determine which residual patterns the cross-entropy gradient controls. Access to them therefore lets us evaluate the representation component of a calibration certificate after training. This motivation applies to the model being optimized, independently of the choice of a subsequent recalibration method. The local and feature-aware methods you highlighted provide an additional practical motivation, which we now develop in the Introduction and Related Work.

  • What our analysis adds to the suggested literature: The cited works establish important calibration results for specific estimators, objectives, or corrective function classes. For example, KCal constructs a kernel-based predictor on learned embeddings and derives calibration guarantees under density-estimation assumptions. Barbera et al.'s Jensen–Shannon approach combines alignment to estimated local class frequencies with a cross-entropy term on those estimates. Jin et al. obtain multicalibration through squared-loss minimization over tree ensembles under a loss-saturation condition, while MCGrad iteratively fits corrections using features and current predictions. These results concern the corresponding recalibration procedures and their assumptions; they do not make population stationarity in an arbitrary fixed linear head sufficient for calibration.

    Our contribution identifies exactly the feature condition needed for that implication. Theorem 2 gives a necessary and sufficient criterion on finite support, with the input distribution, features, and predictor fixed and the implication quantified over conditional label laws. Theorem 3 then quantifies nonzero gradients, imperfect witness representation, and sampling uncertainty, while Proposition D.1 makes a full-class approximation upper bound computable. Thus, the use of a cross-entropy objective alone is not the guarantee: the available representation determines what its stationary gradient can establish. We clarify this distinction in Related Work and Sections 5–6. We also distinguish the targets: our witnesses depend on predicted probabilities, whereas multicalibration additionally tests input-defined subgroups. These connections strengthen the motivation while keeping the proved guarantees precise.

  • Clarity and notation: We have made the analytical message easier to follow before introducing its notation. The Introduction and Sections 4–6 now state each result's scope and main implication before the formal statement. In Section 6, we explain how the gradient controls represented residuals, how witness approximation controls the remainder, and how sampling uncertainty enters population transfer. The detailed proof appears in Appendix C. Following your specific notation suggestion, we define the witness-complexity factor directly in Section 4, organize the notation in Appendix H, and use Figure 1 to connect stationarity, representation, and the anchor construction.

  • Representation families and the number of classes: The analysis permits nonlinear frozen representations; the linearity assumption concerns the head. To examine how the numerical bounds depend on representation geometry, we added ImageNet-pretrained ResNet-50 and ViT-B/16 on CIFAR-100, using the same 2,000 training and 1,000 test images (Section 7.5 and Appendix G.7). Tables 13–14 report accuracy, model sizes, empirical certificates, their component terms, and runtimes. Tables 2 and 17 extend the projected-witness comparison through m=100m=100, with conditional-null diagnostics to interpret the observed scores. Appendix G.6 separately isolates feature dimension through nested representations. These studies address the proposed CNN/transformer and class-count comparisons and show how representation quality affects the quantities appearing in the theory.

I appreciate the authors rebuttal. I tried to download the new pdf, but the changes are very hard to track. Before I evaluate their response, I would kindly ask authors to highlight the changes (text of a different color, also for captions of new tables and images, is a quite standard pratice)

Reviewer_h4KYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether a vanishing cross-entropy gradient implies multiclass calibration when training a linear classifier on frozen features. The authors give a necessary and sufficient condition based on whether the feature span can represent the indicators of groups with identical predictions. The paper also derives finite-training bounds for projected smooth calibration, separating optimization, representation, and sampling errors. These results are examined through synthetic experiments and frozen CNN features on CIFAR-10.

Critical strengths
  1. Theorem 2 gives a precise condition under which cross-entropy stationarity implies calibration. The counterexample is useful because it shows that even a population optimum can remain miscalibrated when the feature representation is insufficient.
  2. Theorem 3 separates optimization error from feature approximation and finite-sample uncertainty. Theorem 1 also explains why empirical calibration scores can be positive even for a calibrated predictor. Together, these results help clarify what a small training gradient can and cannot tell us about calibration.
Critical weaknesses
  1. The theoretical bounds are interesting, but the results on real features are less convincing. In Table 1, the empirical certificates are smaller than the mean individual label-residual bound. However, in Table 5, the population calibration bound using 64 neural anchors is 1.266, compared with 0.761 for the constant baseline. This suggests that the more elaborate approximation does not necessarily produce a tighter population certificate. The authors discuss the additional uncertainty involved, but I would like to see clearer evidence of when the anchor-based approach is actually beneficial.
  2. The exact condition in Theorem 2 can be quite restrictive. When most inputs receive distinct predictions, the feature matrix needs full row rank, which seems difficult to achieve with typical frozen representations. The finite-training analysis addresses this issue through approximation error, but the experiments mainly use synthetic settings and one frozen CNN on CIFAR-10. It is difficult to judge from these results how well the bounds would behave with larger pretrained representations.
Questions for authors
  1. Why do the neural-anchor population bounds in Table 5 remain larger than the constant baseline? Under what conditions would the anchor construction give a clear advantage? A prospective evaluation using independent audit data would also be helpful.

  2. How well does the feature approximation term behave with more realistic pretrained representations? In particular, does increasing the feature dimension make the resulting calibration certificate meaningfully tighter?

1 comment

Thank you for your thoughtful questions regarding when the anchor construction yields a concrete advantage and how our bounds behave with larger pretrained representations and varying feature dimensions. Below, we address each of your questions and note where the corresponding analyses appear in the revised manuscript.

  • Why the 64-anchor bound is larger on CIFAR-10: An anchor operator tightens the uncapped population certificate when its approximation gain outweighs the extra gradient and transfer costs associated with its coefficient budget BB. In our retrospective CIFAR-10 audit, the 64 anchors reduce the mean audit approximation upper bound from 0.9160.916 (constant baseline) to 0.8460.846, but increase the mean coefficient budget from 4.4724.472 to 16.15116.151. This increase in BB amplifies the gradient and concentration costs, resulting in a mean population bound across five heads of 1.2661.266 for 64 anchors compared with 0.7610.761 for the constant baseline. We now explain this cost-benefit decomposition explicitly in Section 7.1 and Appendix G.4, revised Table 6 (originally Table 5).

  • Conditions and a concrete example where anchors win: Proposition D.1 identifies the favorable regime: features accurately represent anchor assignments, prediction-to-anchor displacement is small, and the coefficient budget BB remains controlled. At the same coefficient budget, a smaller approximation term directly tightens the empirical certificate. For the same predictor and 120-observation sample at step 1024, both the constant operator and the half-slope anchor operator have B=3B=\sqrt{3}. The anchor reduces the approximation upper bound from 0.35000.3500 to 0.20620.2062 and tightens the empirical trajectory certificate from 0.27380.2738 to 0.16130.1613. Supplement S6.1 describes this controlled operator family and reports the half-slope result; both operators' results appear in trajectory_theory/verification.json in the original code archive.

  • Evaluation with operators fixed before auditing: Following your suggestion to fix operators before inspecting audit data, we fixed the CIFAR-100 protocol before fitting the heads, constructed the candidate operators solely on training data, and evaluated their empirical certificates on a separate held-out subset of 1,0001{,}000 test images. We present this evaluation in Section 7.5 and Appendix G.7. Because the audit samples images without replacement from a fixed test pool, population guarantees additionally require a justified sampling model or a concentration bound for that finite-pool design.

  • Effect of increasing feature dimension: When the exact feature-span condition of Theorem 2 is too restrictive, Theorem 3 evaluates representations through approximate witness fitting. Holding predictions and the coefficient budget fixed, adding nested feature coordinates preserves or improves the best achievable witness fit. When we refit the head, the empirical certificate reflects the combined changes in predictions, gradients, and residual weighting. In Appendix G.6 (Tables 11–12), we isolate both effects in a retrospective study with disjoint fitting, selection, and audit subsets, expanding from q=8q=8 to q=64q=64 PCA components (feature dimension d=q+1d=q+1 includes the intercept). With predictions fixed, the three-witness approximation error drops from 0.54200.5420 to 0.42470.4247; this finite bank gives a lower bound on the full approximation profile. After refitting, the complete empirical certificate on the held-out audit subset tightens from 0.37880.3788 (q=8q=8) to 0.31180.3118 (q=64q=64), even though the full-class approximation upper bound increases. The generic population bounds remain at the cap of 2.

  • Evaluation on larger pretrained representations (ResNet-50 and ViT-B/16): To extend the representation comparison, we evaluated ImageNet-pretrained ResNet-50 and ViT-B/16 on CIFAR-100. Across both backbones, the constant operator produces the tightest empirical certificate, while training-selected anchors incur a larger approximation contribution that accounts for most of the difference. We report these pretrained backbone comparisons and term-by-term breakdowns in Section 7.5 and Appendix G.7 (Tables 13–14).

Reviewer_82YfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when a zero population cross-entropy gradient implies calibration for a linear head on frozen features. For finite input support, it gives an exact condition for this implication to hold for every conditional label distribution: the features must represent each equal-prediction group indicator. Using projected smooth calibration (Gopalan et al., 2024), it bounds sampling fluctuations and separates finite-training error into gradient, feature-approximation and sampling terms. An anchor operator makes the approximation bound computable. The experiments include synthetic examples, frozen CIFAR-10 features and supplementary independent audits.

Critical strengths
  1. Theorem 2 is stated carefully. The reverse direction constructs a valid label distribution with a miscalibrated global optimum when the feature condition fails. The authors distinguish representing prediction groups from recovering the full input-conditional label distribution, and clearly state what is fixed and what varies.

  2. Theorem 1 gives matching dependence on the projection budget and number of classes, up to logarithmic factors. This quantitative result goes beyond observing that a finite-sample score can be positive. The lower bound is correctly restricted to this empirical score. Proposition D.1 controls the full witness class with one finite operator. The supplementary independent audits are also useful.

Critical weaknesses
  1. My main concern is the significance of the characterization. The qualitative gap between loss optimization and calibration is already known, as discussed in the related work on proper losses, including Błasiok et al., When Does Optimizing a Proper Loss Yield Calibration? (2023). Theorem 2 is more precise, but fixes finite support, features and the predictor, and asks for a guarantee over every label distribution. When all predictions differ, it requires full row rank of the feature matrix. Failure only shows that some label distribution can be miscalibrated; it does not diagnose the fitted model on the actual task. The preset indicator-feature examples explain the theorem, but do not show how it guides feature choice or improvement on real data.

  2. The practical results do not yet resolve this concern. For CIFAR-10, the first term of Equation (11) already exceeds 2, the metric's maximum. The smaller synthetic population bounds rely on a known input distribution. Table 5 reports a population bound of 1.266 for 64 anchors, versus 0.761 for the constant operator. Improving on the mean individual label residual is also a weak test, since that residual can remain large under perfect calibration. The independent audits are helpful, but do not establish an advantage for the training bound based on gradients and feature approximation.

Questions for authors
  1. Can the condition in Theorem 2 guide feature selection or feature enrichment that improves calibration on independent test data? Please compare with a simple feature expansion or an existing recalibration method and report model size, accuracy and data use. A new algorithm is not required, but a concrete example of what the condition helps us decide would strengthen the paper. If the exact condition is too strong, can the approximation term in Theorem 3 provide useful guidance?

  2. For learned frozen features, when does the population training bound improve on the constant operator? Please report the sample size and runtime for the same predictor and confidence level. If no such case is demonstrated, please make that limit clear.

  3. Which files reproduce the supplementary real-data and exact-arithmetic results? The main paper mentions a code archive, while Supplement S3.6 excludes original experiment records and replay scripts from the source bundle. Please clarify the archive contents and commands needed for Tables S1 and S5.

1 comment

Thank you for your rigorous evaluation of the theoretical significance of our characterization, the practical utility of our certificates, and our reproducibility documentation. We have revised the manuscript and added targeted analyses to address each of your points directly.

On the significance of the exact characterization (Weakness 1): While the general gap between proper loss minimization and calibration is established, our work pins down the exact representational mechanism governing when cross-entropy stationarity alone—within the trained head parameterization—suffices for calibration. Specifically, our condition requires the feature span to contain the indicator for each equal-prediction group, complementing Błasiok et al. (2023), who analyze local optimality under external Lipschitz post-processing. Theorem 2 requires stationarity to imply calibration across all conditional label distributions for a fixed finite-support input distribution, representation, and predictor. This worst-case quantification over label laws necessitates full row rank when predictions are distinct, and its converse proves that whenever the span condition fails, there exists a valid label distribution under which the stationary predictor is miscalibrated. For a specific fitted task, Theorem 3 combines the observed gradient, witness-approximation error, and sampling uncertainty into a quantitative bound. We clarify this distinction between worst-case structural sufficiency and task-specific evaluation in the Related Work and Sections 5–6, and add a fixed-bin feature example connecting our span condition to histogram calibration.

On guiding feature selection and enrichment (Weakness 1; Question 1): To demonstrate how our framework supports concrete modeling decisions when the exact condition is too strong, we use the empirical certificate to balance the observed gradient against the residual-weighted approximation term across candidate feature maps. In Appendix G.6 (Tables 11–12), we added a retrospective nested-feature comparison using disjoint splits for fitting, selection, and auditing, alongside model sizes, accuracies, and sample allocations. The complete empirical certificate selects the 64-component head (matching validation NLL and the largest-dimension rule), whereas the unweighted constant-operator approximation upper bound selects eight components. On the held-out audit split, the 64-component head has a smaller certificate (0.3118 vs. 0.3788) and empirical class-wise calibration error (0.0512 vs. 0.0610). This illustrates why we evaluate approximation jointly with the remaining certificate terms. Temperature scaling further lowers the 64-component head's class-wise error to 0.0377.

On population certificates, anchor tradeoffs, and costs (Weakness 2; Question 2): We revised Section 7.1 and Appendix G.4 (Table 6) to state the scope and limitations of our population bounds explicitly. In the learned-feature setting, we have not demonstrated a case where the anchor-based population certificate improves upon the constant operator. The generic training bound reaches the metric's cap of 2. Our retrospective fixed-operator audit, using the same predictors, N=10,000N=10{,}000 audit observations, and total failure probability δ=0.05\delta=0.05 across 15 candidates, gives mean bounds across five heads of 0.761 for the constant operator and 1.266 for 64 anchors. Although anchors reduce the empirical approximation upper bound before population transfer, their larger coefficients and transfer penalties outweigh that gain. For this comparison, an anchor tightens the uncapped population bound when its reduction in the transferred approximation contribution exceeds its added coefficient-weighted gradient cost. We also clarify that the smaller controlled population bounds rely on a known input marginal, that comparison with the individual-label residual measures upper-bound tightening, since this residual can remain positive under perfect calibration, and that independent audits are distinct from the gradient-and-approximation training bound. Regarding runtimes, we note in the text that the retained CIFAR-10 records do not provide comparable operator-construction timings.

On reproducibility and exact commands (Question 3): We expanded Supplement S3.6 to map Tables S1 and S5 and the exact-arithmetic verifications to their source files and execution commands. We distinguish exporting tables from retained records, replaying exact-arithmetic verification scripts, and recomputing experiments. From the companion code archive's root, the following commands export the supplementary tables and check the exported files, respectively:

python revision_code/export_supplement_tables.py
python revision_code/export_supplement_tables.py --check
Reviewer_Vc5oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper considers multiclass calibration for linear classification heads on frozen feature representations. First, in the finite-support setting, the paper characterizes exactly when cross-entropy stationarity implies calibration (Theorem 2). Second, this is extended to finite training by accounting for nonzero gradient and imperfect witness representation, which does not require finite support (Theorem 3). An algorithm that computes an upper bound on the approximation error is also provided. The resulting certificates are evaluated on CIFAR-10.

Critical strengths
  • The paper provides a clear necessary and sufficient condition for when cross-entropy stationarity of a linear head implies calibration: a linear combination of features can isolate each group’s average residual. This condition is simple in a linear-algebraic sense and interpretable.
  • The paper connects theoretical concepts of multiclass calibration to certificates that are based on quantities available after training.
Critical weaknesses
  • The exact characterization is derived under a rather restrictive setting (frozen features, a linear head, finitely supported inputs).
  • Empirical experiments are restricted to m=1m=1.
Questions for authors
  • All experiments use m=1m = 1. I do not think the current experiments clearly demonstrate the benefit of projected smooth calibration (subset calibration) over class-wise smooth calibration. Can the authors report results for larger mm, compare them against class-wise calibration error, include a setting where subset miscalibration can be observed, and report the computational cost as mm grows?
2 comments

Thank you for your constructive feedback on both the theoretical scope of our analysis and our empirical evaluation. We have revised the manuscript and expanded our experiments to address each of your points directly.

On the scope of our characterization: Our primary goal is to isolate the exact properties of a representation that allow cross-entropy stationarity to guarantee calibration. We study a fixed (though potentially nonlinear) representation with a linear head on finitely supported inputs because this setting allows an exact geometric characterization via the feature span. In contrast, jointly training the representation dynamically alters the feature span and introduces sample dependencies that require separate analysis for population guarantees. To clarify how our theory extends beyond the exact setting, we have revised Sections 5–6 to better highlight the progression to Theorem 3. Theorem 3 relaxes the finite-support and exact-stationarity assumptions, providing a quantitative bound for general input distributions that accounts for nonzero gradients, approximate witness representations, and finite-sample uncertainty, under its stated training assumptions and with bounded features fixed independently of the head-training sample. Thus, while our exact characterization isolates the fundamental role of the representation, Theorem 3 quantifies the practical impact of imperfect representations and optimization.

On projection budgets, class-wise comparisons, and runtime cost: As requested, we expanded our evaluation across projection budgets m∈{0,1,2,K}m \in \{0, 1, 2, K\} on five predictors spanning 7, 10, and 100 classes. These experiments now include ImageNet-pretrained ResNet-50 and ViT-B/16 representations on CIFAR-100 (Section 7.5 and Appendix G.7). Table 2 directly compares class-wise smooth calibration against our projected scores (reported as tractable lower bounds on the full empirical metric), while Tables 7 and 17 detail the performance and computational cost across budgets. For our specified finite witness banks, CIFAR-100 evaluations take 1.94–2.42 seconds on 256 observations across the tested positive budgets.

On illustrating detectable subset miscalibration: Finally, to demonstrate what projected tests capture beyond class-wise calibration, we added an exact four-class construction in Appendix G.5. In this example, every individual class is perfectly calibrated conditional on its own predicted probability, yet the class subset {1,2}\{1, 2\} is miscalibrated. While class-wise calibration is blind to this failure mode, our projected smooth calibration detects the error already at m=1m=1.

I thank the authors for clarifications and conducting an expanded empirical experiments. The revised manuscript strengthens the paper. Thus, I keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revised Manuscript

We thank the reviewers for their careful reading and constructive suggestions. Your comments helped us make the theoretical contribution clearer and explain more directly what the certificates tell us about fitted models.

We characterize when cross-entropy stationarity guarantees calibration for a linear head on frozen features. Theorem 2 gives the exact feature-span condition on finite support, while Theorem 3 quantifies the gradient, witness-approximation, and sampling contributions for finite training. Theorem 1 characterizes finite-sample fluctuations. We keep these theorem statements unchanged and now introduce their assumptions and implications before the formal results. We also connect the motivation more closely to local calibration and multicalibration in the Introduction and Related Work.

Following the requests for more concrete comparisons, we added an exact separation between class-wise and projected calibration, larger projection budgets, and count-matched null diagnostics. We also evaluated pretrained CNN and ViT features on CIFAR-100 and examined feature dimension and longer head training (Section 7 and Appendices G.5–G.7). These studies help explain why a smaller gradient or a better approximation fit may still leave other certificate terms large. We clarify the distinction between empirical results and population guarantees throughout, and give the reproduction instructions in Supplement S3.6. All references below use the revised manuscript numbering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.