acceptodds
Under review as a conference paper at ICLR 2027

Beyond Overconfidence: Revisiting Calibration in Modern Vision Models

Abstract

Modern training recipes for large-scale vision models — combining aggressive augmentation, strong regularization, and web-scale pretraining — have driven substantial gains in image classification accuracy across diverse architectures and benchmarks. Their joint effect on calibration, however, remains poorly characterized. We present a large-scale calibration benchmark comprising 1218 ImageNet classifiers, a controlled ablation across 2015 ViT configurations, and a transfer evaluation on four medical imaging tasks, spanning training paradigms from classical from-scratch baselines to web-scale pretrained models with aggressive augmentation and regularization. Four findings emerge: (i) in the high-accuracy regime, modern classifiers are systematically underconfident: 98.4% of the checkpoints in the top 5% of accuracy have negative calibration bias; (ii) the training recipe, not the architecture, is the primary driver of the calibration regime, evidenced by a recipe-shaped Pareto front, consistent shifts across two architecture families, and a controlled ViT ablation; (iii) under severe distribution shift, in-distribution underconfidence reverses to overconfidence; while post-hoc calibration mitigates underconfidence in the in-distribution setting, this exacerbates overconfidence under severe distribution shifts; and (iv) the recipe-driven regime holds for medical imaging: modern-recipe models remain underconfident across four imaging modalities, while supervised ResNet baselines are overconfident on the same tasks. Together, these findings suggest that modern training recipes induce a structural calibration regime that can reverse under distribution shift.

Primary area:
datasets and benchmarks
Keywords:
Confidence calibration, Uncertainty quantification, Benchmark, Distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qCe1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a comprehensive large-scale empirical study of how modern training recipes (aggressive augmentation, strong regularization, and web-scale pre-training) affect confidence calibration in vision models. Three model populations are analyzed:

(1) 1,218 timm ImageNet-1k classifiers (87 architecture families): the population is predominantly underconfident (bias < 0 <0 for 75.5% of checkpoints), and underconfidence is concentrated at the high-accuracy end - 98.4% of the top-5% accuracy models have negative calibration bias (median bias − 0.053 −0.053). (2) 2,015 ViT configurations (Steiner et al., 2022; architecture and optimizer fixed): pre-training corpus and augmentation/regularization strength each shift the bias monotonically from overconfident toward underconfident, with pre-training corpus as the dominant factor (IN-1k → → IN-21k moves the bias by 0.116). (3) Four medical imaging tasks: fine-tuned foundation models (DINOv2 / DINOv3 / CLIP-WIT / CLIP-LAION, all ViT-B/16) are underconfident, whereas from-scratch ResNet baselines are overconfident.

The paper further reports that in-distribution underconfidence reverses to overconfidence under natural distribution shifts and across ImageNet-C corruptions, so that temperature scaling (TS) fitted on the in-distribution validation set increases ECE under severe shift. The central claim is that calibration is a "structural property set by the training pipeline," and the authors promise to release cached logits and evaluation code.

Critical strengths

S1. Substantial scale and solid engineering. Evaluating 1,218 checkpoints on IN-1k / IN-C (19×5) / four natural shifts (roughly 122k model-split evaluations) is a real investment, and the combination of within-family recipe trajectories plus a 2,015-configuration controlled ablation constitutes a better-organized evidence layout than most comparable work.

S2. A concrete, actionable and practically useful finding. The chain "ID underconfidence → reversal to overconfidence under shift → ID-fitted 𝑇<1 worsens ECE under severe shift" has a clear mechanistic explanation (the direction of 𝑇<1 is mismatched with shift-induced overconfidence) and is a direct caution for the post-hoc calibration literature. Appendix B.3 (BEiT/EVA) shows this is not specific to a single architecture.

Critical weaknesses

W1. The central causal claim - that the training recipe, not the architecture, determines the calibration regime - is not supported by the controlled evidence and is partly contradicted by the paper's own tables.

The only experiment with a controlled variable (the 2,015-configuration ViT ablation) never pushes the calibration bias below -0.011 (p. 6), whereas the effect the paper is built around is -0.05 to -0.229 (Section 4.1). Every strong underconfidence result comes from web-scale pre-training, for which no ablation exists: corpus size, objective (contrastive / self-supervised vs supervised), training length and fine-tuning procedure all co-vary, and the authors themselves state that the strong effect is "driven by recipes outside this ablation." The controlled experiment therefore cannot support the magnitude of the claim it is used to support. More directly, the paper's own tables show that architecture/scale variation is not smaller than recipe variation at matched recipes: within Sup. IN-1k + AugReg, ViT spans +0.007 (Ti/16) to +0.126 (L/16), a spread of about 0.119 against a corpus effect of 0.116 in the ablation (Table 16), and ConvNeXt spans -0.060 to -0.008 with the two ends of opposite sign (Table 15). This directly contradicts the abstract and conclusion claim that recipe effects are "consistently larger" than architecture effects. No variance decomposition, effect size, or accuracy-matched comparison is provided, and recipe, accuracy and calibration are fully entangled, so the recipe effect may be mediated by accuracy alone.

Why this is consequential: the recipe-over-architecture attribution is the paper's central scientific claim and the basis of its framing ("calibration is a structural property set by the training pipeline"). If that attribution is not established, what remains is a large-scale catalog plus the genuinely interesting observation that web-scale models are underconfident and that this regime reverses under shift - useful, but a substantially narrower contribution than the paper claims.

W2. The "revision of Minderer et al." framing is unsupported by the paper's own numbers, and the robustness check that would substantiate the comparison is missing.

The paper positions itself against the conclusion that modern models are well calibrated, but the recipes Minderer studied - Sup. IN-21k -> IN-1k with and without AugReg - are evaluated here at biases of -0.008 to +0.017, i.e. consistent with Minderer. Underconfidence appears only for web-scale / contrastive / self-supervised checkpoints, which were outside Minderer's scope. Furthermore, Minderer uses equal-mass binning while the primary metric here is equal-width M=15; the robustness check in Appendix C compares only equal-width M=15 vs M=100, which share the same failure mode and therefore say nothing about the choice of binning strategy. Most seriously, Appendix C explicitly promises equal-mass ECE results ("We re-compute ECE under equal-mass binning on the architecture-family case-study checkpoints") and those results appear nowhere in the paper, so the claim of comparability to Minderer cannot be verified by a reviewer.

Why this is consequential: the contribution is framed as correcting a widely cited result. If the paper's own measurements agree with that result for exactly the recipes it covered, the framing overstates the contribution; and the one analysis that would let a reviewer check the cross-paper comparison is absent.

Questions for authors

Q1. Does "recipe > architecture" still hold under accuracy matching (or with accuracy stratification)? Can you provide a formal effect-size / variance decomposition (e.g., a two-way architecture × × recipe analysis) to support the "consistently larger" statement?

Q2. The controlled ViT ablation reaches only bias ≈ − 0.011 ≈−0.011, an order of magnitude below the headline − 0.05 ∼ − 0.23 −0.05∼−0.23. Can the web-scale part provide a comparably strong controlled experiment (fixing at least two of corpus / objective / training length)? Otherwise, how do you argue that underconfidence is a "structural property of the recipe" rather than a by-product of web-scale pre-training?

Q3. Where are the equal-mass ECE values promised in Appendix C? If they differ from the equal-width results, how does that affect comparability with Minderer et al.?

Q4. What are the results of the medical 2 × 2 2×2 control (fine-tuning recipe × × architecture)? Under an identical fine-tuning recipe, does the under/overconfidence difference between foundation models and ResNets persist?

Q5. Does the learned logit scale (temperature parameter) of CLIP / SigLIP-type models contribute to the observed underconfidence? Are the conclusions stable after re-fitting the logit scale or using a linear probe?

Q6. Is the EVA-Giant bias of − 0.229 −0.229 an artifact of the evaluation pipeline (preprocessing / resolution / normalization)? Please provide its reliability diagram.

Q7. On which concrete downstream task does underconfidence cause a measurable loss? Could you provide a quantified example using risk-coverage or thresholded decisions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qCe1 (1/5)

We thank the reviewer for finding the evaluation extensive and well organized and its implications for post-hoc recalibration actionable. Regarding the two main concerns, we now support the attribution to the training recipe with several new analyses, including variance decompositions with accuracy control and a controlled ablation on four medical datasets (W1, Q1, Q2, Q4), and we clarify that our study extends rather than revises Minderer et al. (W2, Q3).

W1 Recipe vs. architecture.

We agree that the central claim of the paper, that the training recipe affects calibration more strongly than the architecture, needed stronger support. We therefore performed three additional analyses:

  1. Variance decomposition. A Type-II ANOVA on the controlled ViT ablation attributes 63% of the variance in calibration bias to the recipe and 22% to the architecture (see Q1).

  2. Accuracy matching. Since recipes differ in the accuracy they reach, we repeated the decomposition with accuracy as a covariate and within accuracy quintiles. The recipe remains the dominant factor (55% vs. 29%; 77% vs. 19% among the most accurate runs; see Q1).

  3. New ablation on medical datasets. We fine-tuned ViT-B/16 and ConvNeXt-B from different pre-trained checkpoints (ImageNet-1k, ImageNet-21k, and DINOv3) with identical training recipes in a full factorial grid on four medical datasets (see Q4). Again, the recipe dominates: it explains 79–97% of the variance in the bias, the architecture at most 5%.

Regarding the calibration bias values for the Vision Transformers mentioned by the reviewer: the large differences between ViTs are confined to ImageNet-1k pre-training, where they are driven mainly by ViT-L/16. In this setting, ViT-L/16 overfits and even reaches a lower accuracy than ViT-B/16, as already reported in the original study ([1], Figure 1). With ImageNet-21k pre-training, the differences between ViT sizes are small.

[1] Steiner et al., How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers, TMLR 2022.

 

W2 Relation to Minderer et al.

We agree that, for the recipes Minderer et al. studied, our results reproduce theirs, and we do not intend to position our manuscript as a revision of their work. For ViT-B/16 (IN-21k → IN-1k, AugReg), for example, the measurements released by Minderer et al. give ECE = 0.015 (equal-mass, M = 100) at an accuracy of 0.840; under the same estimator, we obtain 0.014 at an accuracy of 0.843 (see Q3). We regard this agreement as a validation of our pipeline.

Our contribution lies beyond that regime. Checkpoints fine-tuned from web-scale pre-training with strongly regularized recipes, which were outside the scope of Minderer et al., now dominate the set of models with very high accuracy that are favoured by practitioners (Fig. 2). For these, calibration moves into clear underconfidence. Sec. 4.1 already frames our study as an extension (“our broader population extends this picture”, ll. 242–243), but we agree that parts of the introduction could be read as a revision of Minderer et al. We have rephrased them accordingly (changes in blue in the uploaded revised manuscript).

The one point where our results refine their conclusions is the cause of calibration differences between model families. Minderer et al. hypothesized that the architecture plays a major role. Our controlled ablation holds the architecture fixed and varies the recipe, which tests this hypothesis directly within the ViT family; our new medical ablation additionally compares ViT-B/16 and ConvNeXt-B under identical recipes (Q4). As described in W1, Q1 and Q4, we now report this as a quantified effect, including variance decompositions and accuracy-controlled comparisons.

AuthorsOpenReview ↗

Response to Reviewer qCe1 (2/5)

Q1 Accuracy matching and variance decomposition.

Variance decomposition. We fit a Type-II ANOVA on our controlled ViT ablation grid. It attributes 63% of the variance in calibration bias to the recipe (ω² = 0.60) and 22% to the architecture (ω² = 0.21); architecture × recipe interactions explain a further 8% (R² 0.86 → 0.94). Pretraining data alone (49%) has more than twice the effect of the architecture. The same ordering holds for ECE (recipe 45% vs. architecture 16%). At a fixed architecture, the bias varies across recipes by a median of 0.22; at a fixed recipe, it varies across architectures by only 0.05. Our new medical ablation (Q4) confirms this ordering in a different domain, with an even smaller architecture share (at most 5%).

Accuracy control and stratification. We agree that accuracy is a potential confounder. Therefore, we tested on the ViT grid whether the recipe effect persists at fixed accuracy in two ways. (i) With top-1 accuracy as a covariate, the recipe still accounts for 55% of the variance in calibration bias and the architecture for 29%. Compared to the analysis without this control, the recipe share drops (63% → 55%) while the architecture share rises (22% → 29%), because part of the recipe effect operates through accuracy. The accuracy-controlled estimates are therefore conservative for the recipe. (ii) We split the runs of the ViT ablation grid into accuracy quintiles and repeated the decomposition within each stratum (table below; shares of the variance remaining after accuracy). Among the most accurate runs (top quintile, top-1 accuracy 0.78–0.86), the recipe explains 77% of the variance in bias and the architecture only 19%; in the fourth quintile, the split is 72% vs. 24%. Across all strata, the recipe dominates in four of five; the two are on par only in the middle quintile.

Accuracy quintileTop-1 acc.RecipeArchitecture
10.54–0.6664%35%
20.66–0.7055%42%
30.70–0.7448%51%
40.74–0.7872%24%
50.78–0.8677%19%
AuthorsOpenReview ↗

Response to Reviewer qCe1 (3/5)

Q2 Web-scale pre-training.

In our analyses, web-scale pre-training is not an alternative explanation to the recipe but one of its components. Throughout the paper, the training recipe denotes all choices that determine how a model’s weights are inferred for a given architecture: the pre-training corpus, objective and schedule (e.g. supervised ImageNet-1k/21k, self-supervised DINOv3, contrastive CLIP) and the fine-tuning procedure (data augmentation, Mixup/CutMix, label smoothing). The architecture, in contrast, comprises the network design and size. When we call underconfidence a structural property of the recipe, we therefore mean that it is determined by the training setup as a whole rather than by the architecture. Several components of this setup contribute: strongly regularized fine-tuning makes models underconfident, and web-scale pre-training shifts the bias further in this direction. The strongest underconfidence arises where both are present, as in our headline models.

A controlled pre-training experiment at web scale, varying corpus, objective and training length, is beyond our compute budget and the rebuttal period. Instead, we ran a new controlled ablation on four medical datasets (Q4, new App. B.6). It fixes the architecture (ViT-B/16, ConvNeXt-B) and the pre-trained checkpoint (ImageNet-1k, ImageNet-21k, DINOv3) and varies only the fine-tuning recipe in a full factorial design. Unlike the ViT grid, this ablation reaches the magnitudes of our headline models: with the strongest recipe, backbones are clearly underconfident (bias from −0.084 to −0.164), whereas the same checkpoints are overconfident with the weakest recipe. At a fixed recipe, the web-scale DINOv3 checkpoint is the most underconfident backbone on every dataset (down to −0.164). Taken together, this further supports our conclusions from the controlled ViT ablation with lower absolute biases.

Q3 Equal-mass binning.

We thank the reviewer for pointing out that the equal-mass results were missing. App. C now reports ECE under equal-width (EW) and equal-mass (EM) binning, each with M = 15 and M = 100, for the case-study checkpoints (new Table 20). For all checkpoints with |b| ≥ 0.01, the four estimates differ by at most 0.005, and at a fixed number of bins, EW and EM differ by at most 0.003. The high-accuracy underconfident checkpoints are no exception: for the 20 checkpoints with accuracy ≥ 0.85 and b < 0, EW and EM differ by at most 0.001 (M = 15) and 0.003 (M = 100), because their miscalibration is one-sided and the ECE then equals |b| under any binning. These results are in line with Minderer et al.: under their estimator (EM, M = 100), we reproduce their ECE for ViT-B/16 (0.014 vs. 0.015, see W2).

Reviewer_9Wc8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a large evaluation of vision models to revisit the effect of aggressive data-augmentation, data scale, and regularization on calibration. They find out that the best classifiers are under confident, generally, classifiers gain confidence when meeting OOD settings, and that the training recipe is the main driver of the calibration regime. Some additional experiments on medical imaging tasks confirm this behaviour. They finally conclude that the training recipes induce this calibration regime tat can reverse under OOD testing.

Critical strengths
  • Understanding calibration for classifiers is an important research topic, especially for high-stakes domains such as medical imaging where having a calibrated classifier is a fundamental requirement.

  • The paper is presented quite well, is easy to follow, and has nice plots.

  • The experimental benchsuite is quite extensive, with thousand+ pre-trained classifiers evaluated.

Critical weaknesses
  • The most important weakness of this paper, is that the main findings, which constitute the main contributions, are essentially known/expected: (1) If you enlarge the pretraining dataset massively (web-scale training)/use stronger data augmentation, you are essentially making the fitting task to the model much more difficult and hence will result less confident ID; (2) When moving from ID to OOD settings, it is known that classifiers make overconfident predictions (see all related literature of OOD detection: methods such as ReACT, ViM, etc.).

  • The paper uses the word " modern " a lot, but essentially all the training modalities and models evaluated stop around 2022 (up to ConNeXt and ViT generations). The analysis completely overlooks modern VLMs, which are currently considered the dominant paradigm even for classification.

  • The related work section is quite poor in terms of total references, considering the popularity of the calibration topic within the research community. Furthermore, the latest paper cited in this section dates back to 2024. The claim that modern architectures have not been systematically evaluated for calibration at scale is superficial (see [1]).

  • Table 1 should bold the best values and be reformatted.

[1] ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy, ICML 2024

Questions for authors

I do not have any specific questions for the authors. I recommend checking the weaknesses for potential rebuttal.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9Wc8 (1/2)

We thank the reviewer for recognizing the importance of calibration, particularly in high-stakes domains, and for finding the paper clearly presented with an extensive benchmark. Below, we clarify what are our contributions (W1), show that our study covers recent architectures (W2), and describe the revised related work section (W3) and Table 1 (W4).

W1 Novelty. We agree that it is well known that larger pretraining data and stronger augmentation and regularization can lower confidence. Our introduction references the corresponding work on the calibration effects of individual recipe components, which found that label smoothing (Müller et al., 2019) and Mixup (Thulasidasan et al., 2019) improve calibration by reducing overconfidence. These studies establish the direction of each effect, namely lower confidence, which made the overconfident models of that time better calibrated. They do not establish how these components interact with large-scale pretraining. Furthermore, larger pretraining data does not by itself produce underconfidence: Minderer et al. (2021) already evaluated models pretrained on large datasets (e.g., BiT and ViT on JFT-300M and ImageNet-21k; ResNeXt-WSL on Instagram images), found them mostly slightly overconfident, and concluded that the amount of pretraining does not fully explain differences in calibration. The models in our study were trained with more recent recipes, which combine large-scale pretraining with stronger augmentation and regularization during fine-tuning, such as label smoothing, Mixup/CutMix and RandAugment (e.g. see Wightman et al.). As noted in our introduction, the joint effect of these components has not yet been quantified at scale, and this is what we investigate in our benchmark study. We start from the most accurate models, ask how well calibrated they are, and analyze how the components of their training recipes jointly determine their calibration. We find that 98.4% of the top-5% most accurate models are underconfident. This shows that in the combination that results in highest accuracy, these components no longer correct overconfidence but overshoot calibration, which has not been previously reported. We also agree that it is well established that classifiers become more overconfident under distribution shift, and we do not claim this as new (we explicitly discuss this as related work by Ovadia et al., 2019 and Hendrycks et al., 2021). As part of our benchmark we quantify the interplay of post-hoc re-calibration and the underconfidence of today’s most capable models and illustrate how post-hoc re-calibration can amplify this overconfidence under distribution shift

W2 Model coverage and vision-language models.

We respectfully disagree that the models we evaluate stop around 2022. The models we benchmark include multiple models published after 2022: architecture families such as MobileNetV4 (2024), MambaOut (2025), FastViT (2023), RepViT (2024), Hiera (2023) and FasterNet (2023) (see Appendix Table 3); newer variants of earlier families such as ConvNeXt V2 (2023) and EVA-02 (2024); and SigLIP-SO400M (2023). Our medical experiments additionally use DINOv2 (2024) and DINOv3 (2026).

This impression may arise because our in-depth analyses focus on ConvNeXt and ViT. We chose these two architecture families because they lie on the Pareto front of accuracy and calibration (see Figure 3). We therefore included several models of each family, covering different sizes and training recipes, from supervised ImageNet training to vision-language pretraining (CLIP trained on OpenAI’s data and on LAION-2B), so that recipes can be compared within a fixed architecture family. These CLIP and SigLIP (2023) models are the image encoders of vision-language models, fine-tuned for classification, and are among the most accurate models in our study.

AuthorsOpenReview ↗

Response to Reviewer 9Wc8 (2/2)

W3 Related work

We thank the reviewer for pointing us to Vishniakov et al. (2024) and agree that the related work section was too brief. We have revised it (changes in blue) and added recent large-scale studies on the calibration of modern and vision–language models (e.g., Galil et al., 2023; Tu et al., 2023; 2024; Vishniakov et al., 2024). Please note that further literature on training recipes and their individual calibration effects is discussed in our introduction.

Vishniakov et al. (2024) compare supervised ConvNeXt and ViT models (pretrained on ImageNet-21k and fine-tuned on ImageNet-1k) with zero-shot CLIP models. They find the supervised models better calibrated but slightly underconfident and the zero-shot CLIP models overconfident. Our study, however, focuses on the high-accuracy regime. Today’s most accurate classifiers are obtained by fine-tuning large-scale pretrained encoders end-to-end, such as CLIP, SigLIP, EVA, and DINOv2/v3 models. For these models, we observe markedly stronger underconfidence, with calibration biases as low as −0.229, than the slight underconfidence of their supervised models. Beyond this, we quantify how much of the calibration bias is explained by the training recipe and how much by the architecture, and we study how in-distribution underconfidence affects post-hoc calibration under distribution shift. Vishniakov et al. address neither question.

W4 Table 1

We have reformatted Table 1 in the revised manuscript.

We hope that our responses clarify the contribution of our work and that our benchmark covers recent architectures and web-scale pretrained models. We would be glad to answer any further questions.

I would like to thank the authors for their answers and the effort put into the rebuttal.

The answers to W2 and W4 have covered two of my four concerns. However, W1 and W3, which address the main contribution and novelty of the work, remain unaddressed (I appreciate the expansion of the Related Work section). I see this paper more as an incremental, for sure more extensive, analysis, rather than truly novel work compared to previous publications such as [1].

To this end, I am keeping my score.

AuthorsOpenReview ↗

Response to follow-up: contributions and novelty

We thank the reviewer for the follow-up. We agree that the technical novelty of our submission is limited. We propose no new method but substantially extend earlier empirical analyses, in particular Minderer et al. (2021). However, novelty in science does not only stem from technical aspects, but can also come from empirical findings that reveal overlooked effects and challenge common assumptions.

This is where we see the contribution of our work. We (i) show that today’s most accurate classifiers are systematically and strongly underconfident (98.4% of the top-5% of 1,218 ImageNet checkpoints, calibration biases down to −0.229), contrary to the common assumption that modern classifiers are overconfident, well calibrated, or slightly underconfident; (ii) show that this underconfidence transfers to medical imaging, with calibration biases of similar magnitude (down to −0.164 on four medical datasets); (iii) quantify where the underconfidence comes from: the training recipe explains 63% of the variance in calibration bias and the architecture 22% in the ViT grid, and 79–97% vs. at most 5% on the medical datasets.

To the best of our knowledge, no previous calibration benchmark has reported underconfidence of this strength and consistency, or traced it to the training recipe, across such a broad range of models.

Reviewer_aWMdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines calibration in modern vision models through a large-scale evaluation of ImageNet classifiers, comparisons across training configurations, and fine-tuning on four medical imaging tasks. It reports widespread underconfidence among highly accurate models and argues that training recipes play a larger role than architecture in shaping calibration. It also finds that temperature scaling fitted on in-distribution data can worsen calibration under distribution shift.

Critical strengths

The paper addresses an important and practically relevant problem. It is clearly written and well organized, with a substantial experimental evaluation supporting its main observations. The findings provide useful insights into the reliability of modern vision models and offer a valuable reference for future research on calibration and uncertainty estimation.

Critical weaknesses

1.The analysis does not sufficiently distinguish overconfidence and underconfidence across confidence ranges. Although selected reliability diagrams are provided, the main conclusions largely characterize models through aggregate calibration bias. Appendix B.4 does not fully justify treating cancellation between overconfidence and underconfidence as negligible. For example, Table 15 reports ECE = 0.036 and bias = −0.002 for ConvNeXt-L trained on IN-1k with AugReg, demonstrating considerable cancellation despite the small aggregate bias 2. The study considers a narrow range of calibration approaches, emphasizing augmentation and regularization and using only temperature scaling for recalibration. Excessive confidence suppression during training can itself produce underconfidence, as previously observed with focal loss. Without comparisons covering calibration-oriented losses, nonparametric methods, and recent sample-adaptive approaches, it remains unclear how much the findings reflect these known training effects and whether they extend beyond the selected recipes and correction method. This limits support for the broader conclusions about calibration.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer aWMd (1/2)

We thank the reviewer for finding the manuscript clearly written and well organized, the experimental evaluation substantial, and the findings a valuable reference for future research on calibration. Following the reviewer's suggestions, we now further investigate the effect of cancellation (W1) and extend the post-hoc calibration section with seven further post-hoc calibrators (W2).

W1 Over- vs. underconfidence across confidence ranges.

We agree that bias as aggregate measure has inherent limitations and now further investigate the effect of cancellation. We used calibration bias b as our main metric because our central question is the direction of miscalibration, and comparing models at this scale requires a single signed quantity. We acknowledge, however, that a scalar bias can hide cancellation, and App. B.4 did not sufficiently rule this out. Because the bias is the bin-weighted mean of the per-bin gaps, the ECE splits into an overconfident part O and an underconfident part U (O=∑mM∣Bm∣nmax⁡(confm−accm,0)O = \sum_m^M \frac{|B_m|}{n} \max(\mathrm{conf}_m - \mathrm{acc}_m, 0) over the MM bins BmB_m, and UU analogously with accm−confm\mathrm{acc}_m - \mathrm{conf}_m), with ECE = O + U and b = O − U. We now report O and U in Tables 17 and 18 and summarize them across all checkpoints in Table 19; per-checkpoint values are provided in the supplementary material of the revised submission. Our new analysis revealed that across the population, strong cancellation as in the ConvNeXt-L example is rare and confined to small |b|:

  • Across all 1,218 ImageNet checkpoints, the median cancellation (the share of the ECE that cancels in the bias, 1 − |b|/ECE) is 0.03%, and the median O is 1.810-5, i.e. for more than half of the checkpoints virtually no confidence bin is overconfident.

  • For the 1,004 checkpoints (82%) with |b| ≥ 0.03, the median cancellation is 0% and only 1.3% exceed 25%.

  • Cancellation is substantial only for small |b|. For the 73 checkpoints with |b| < 0.01, the median is 79% (e.g. ConvNeXt-L), but their ECE is small anyway (median 0.022).

  • Among the 61 top-5% checkpoints (by ImageNet top-1 accuracy), 60 are underconfident (median b = −0.053) and the median cancellation is 0.4%.

Restricting the analysis to the 1,004 checkpoints with an unambiguous sign (|b| ≥ 0.03) raises the share of underconfident checkpoints from 75.7% to 82.1%, and the top-5% models show negligible cancellation. Therefore, our main conclusions are not weakened by cancellation.

AuthorsOpenReview ↗

Response to Reviewer aWMd (2/2)

W2 Narrow range of calibration approaches.

Calibration-oriented losses. Several training objectives have been proposed specifically to improve calibration, and we agree that they are relevant to the calibration of image classifiers. Our primary goal, however, is not to determine how individual training methods affect calibration. Instead, we take a practical perspective and ask how the most accurate modern models are calibrated as they are trained and released in practice (Sec. 4.1), and then analyze how the components of their training recipes contribute to the observed calibration (Sec. 4.2). Most of these models are trained with cross-entropy on soft targets from label smoothing or Mixup/CutMix, some with binary cross-entropy instead, combined with augmentation and regularization such as RandAugment and stochastic depth. Calibration-oriented losses such as focal loss and its adaptive variants [1-3], margin-based label smoothing [4] or auxiliary calibration losses such as MDCA [5] are not part of these recipes; we now mention them in the introduction and state why they are outside the scope of our study.

Post-hoc calibration methods. To test the robustness of our conclusions to the choice of post-hoc calibration method, we complement TS with alternative post-hoc calibrators, using an identical protocol (fit on the same held-out split, evaluation on the same ID and shifted test sets):

  • parametric: vector scaling (VS), ensemble temperature scaling (ETS);
  • nonparametric: histogram binning (HB), isotonic regression (IR), BBQ, spline calibration (SPL);
  • sample-adaptive: parameterized temperature scaling (PTS)

The following table shows the calibration bias bb for three web-scale ViT models (ImageNet top-1 accuracy in parentheses):

ModelTest setUncal.TSETSVSHBIRBBQSPLPTS
ViT-B/32 (0.853)ImageNet (ID)−0.066+0.033+0.033+0.034+0.000+0.000+0.000+0.002+0.001
IN-V2−0.025+0.082+0.082+0.088+0.042+0.039+0.044+0.045+0.040
IN-Sketch+0.052+0.171+0.171+0.199+0.139+0.105+0.150+0.142+0.120
ViT-L/14 (0.882)ImageNet (ID)−0.100+0.031+0.031+0.028−0.004−0.003−0.004−0.003+0.000
IN-V2−0.067+0.076+0.076+0.073+0.032+0.029+0.033+0.030+0.038
IN-Sketch−0.037+0.118+0.118+0.114+0.088+0.056+0.096+0.072+0.088
ViT-H/14 (0.886)ImageNet (ID)−0.120+0.031+0.031+0.028+0.001+0.001+0.001+0.001+0.002
IN-V2−0.088+0.074+0.074+0.069+0.037+0.033+0.037+0.034+0.034
IN-Sketch−0.066+0.115+0.115+0.111+0.077+0.052+0.080+0.064+0.079

All three models are underconfident in distribution (bias −0.066 to −0.120), and every post-hoc calibrator compensates the miscalibration. Under distribution shift, however, the result is the same for every method. After each of the eight calibrators, all three models are overconfident on both ImageNet-V2 and ImageNet-Sketch (48 of 48 cases), although without correction they are underconfident on ImageNet-V2 and, except for ViT-B/32, on ImageNet-Sketch. On ImageNet-Sketch the correction also increases the magnitude of the miscalibration in 22 of 24 cases. The reason is that each calibrator learns a correction that matches the in-distribution underconfidence. When the shift itself moves the model towards overconfidence, this correction overshoots, regardless of whether it is parametric, nonparametric or sample-adaptive.

We thank the reviewer for these suggestions, which substantially strengthens the paper. We hope these additions address the reviewer’s concerns and would be glad to answer further questions.

References

[1] Mukhoti et al., Calibrating Deep Neural Networks using Focal Loss, NeurIPS 2020.

[2] Ghosh et al., AdaFocal: Calibration-aware Adaptive Focal Loss, NeurIPS 2022.

[3] Tao et al., Dual Focal Loss for Calibration, ICML 2023.

[4] Liu et al., The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration, CVPR 2022.

[5] Hebbalaguppe et al., A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration, CVPR 2022.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for thorough and constructive reviews. We are grateful that the reviewers considered our central finding, that today’s most accurate classifiers are systematically underconfident, important and practically relevant (aWMd, 9Wc8), the evaluation extensive and well organized (all), and its implications for post-hoc recalibration actionable (qCe1). The reviews converge on four main points, which we address with new analyses and a revised PDF (changes in blue).

1. New controlled ablation on medical datasets (qCe1 Q2, Q4). We fine-tuned ViT-B/16 and ConvNeXt-B from three pre-trained checkpoints (ImageNet-1k, ImageNet-21k, DINOv3) with identical recipes in a full factorial grid (augmentation × Mixup/CutMix × label smoothing) on four medical datasets. With the strongest recipe (web-scale pretraining combined with strong regularization and data augmentations), all backbones become clearly underconfident (calibration bias down to −0.164), while the same checkpoints are overconfident with the weakest recipe.

2. Quantification of recipe and architecture effects (qCe1 W1, Q1). We added Type-II variance decompositions, which attribute 63% of the variance in calibration bias to the recipe and 22% to the architecture in the controlled ViT grid and 79–97% vs. at most 5% in the medical ablation.

3. Cancellation effects in calibration bias (aWMd W1, qCe1 Q3). We now decompose the ECE into over- and underconfident parts and show that cancellation in the bias is negligible except for checkpoints with |b| < 0.01. Equal-width and equal-mass binning agree to within 0.005.

4. Clarified contribution and model coverage (9Wc8 W1 and W2). We have clarified that our study covers current models and what our contribution is beyond the known effects of individual recipe components and the known overconfidence under distribution shift.

Detailed answers follow under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.