- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a comprehensive large-scale empirical study of how modern training recipes (aggressive augmentation, strong regularization, and web-scale pre-training) affect confidence calibration in vision models. Three model populations are analyzed:
(1) 1,218 timm ImageNet-1k classifiers (87 architecture families): the population is predominantly underconfident (bias < 0 <0 for 75.5% of checkpoints), and underconfidence is concentrated at the high-accuracy end - 98.4% of the top-5% accuracy models have negative calibration bias (median bias − 0.053 −0.053). (2) 2,015 ViT configurations (Steiner et al., 2022; architecture and optimizer fixed): pre-training corpus and augmentation/regularization strength each shift the bias monotonically from overconfident toward underconfident, with pre-training corpus as the dominant factor (IN-1k → → IN-21k moves the bias by 0.116). (3) Four medical imaging tasks: fine-tuned foundation models (DINOv2 / DINOv3 / CLIP-WIT / CLIP-LAION, all ViT-B/16) are underconfident, whereas from-scratch ResNet baselines are overconfident.
The paper further reports that in-distribution underconfidence reverses to overconfidence under natural distribution shifts and across ImageNet-C corruptions, so that temperature scaling (TS) fitted on the in-distribution validation set increases ECE under severe shift. The central claim is that calibration is a "structural property set by the training pipeline," and the authors promise to release cached logits and evaluation code.
Critical strengths
S1. Substantial scale and solid engineering. Evaluating 1,218 checkpoints on IN-1k / IN-C (19×5) / four natural shifts (roughly 122k model-split evaluations) is a real investment, and the combination of within-family recipe trajectories plus a 2,015-configuration controlled ablation constitutes a better-organized evidence layout than most comparable work.
S2. A concrete, actionable and practically useful finding. The chain "ID underconfidence → reversal to overconfidence under shift → ID-fitted 𝑇<1 worsens ECE under severe shift" has a clear mechanistic explanation (the direction of 𝑇<1 is mismatched with shift-induced overconfidence) and is a direct caution for the post-hoc calibration literature. Appendix B.3 (BEiT/EVA) shows this is not specific to a single architecture.
Critical weaknesses
W1. The central causal claim - that the training recipe, not the architecture, determines the calibration regime - is not supported by the controlled evidence and is partly contradicted by the paper's own tables.
The only experiment with a controlled variable (the 2,015-configuration ViT ablation) never pushes the calibration bias below -0.011 (p. 6), whereas the effect the paper is built around is -0.05 to -0.229 (Section 4.1). Every strong underconfidence result comes from web-scale pre-training, for which no ablation exists: corpus size, objective (contrastive / self-supervised vs supervised), training length and fine-tuning procedure all co-vary, and the authors themselves state that the strong effect is "driven by recipes outside this ablation." The controlled experiment therefore cannot support the magnitude of the claim it is used to support. More directly, the paper's own tables show that architecture/scale variation is not smaller than recipe variation at matched recipes: within Sup. IN-1k + AugReg, ViT spans +0.007 (Ti/16) to +0.126 (L/16), a spread of about 0.119 against a corpus effect of 0.116 in the ablation (Table 16), and ConvNeXt spans -0.060 to -0.008 with the two ends of opposite sign (Table 15). This directly contradicts the abstract and conclusion claim that recipe effects are "consistently larger" than architecture effects. No variance decomposition, effect size, or accuracy-matched comparison is provided, and recipe, accuracy and calibration are fully entangled, so the recipe effect may be mediated by accuracy alone.
Why this is consequential: the recipe-over-architecture attribution is the paper's central scientific claim and the basis of its framing ("calibration is a structural property set by the training pipeline"). If that attribution is not established, what remains is a large-scale catalog plus the genuinely interesting observation that web-scale models are underconfident and that this regime reverses under shift - useful, but a substantially narrower contribution than the paper claims.
W2. The "revision of Minderer et al." framing is unsupported by the paper's own numbers, and the robustness check that would substantiate the comparison is missing.
The paper positions itself against the conclusion that modern models are well calibrated, but the recipes Minderer studied - Sup. IN-21k -> IN-1k with and without AugReg - are evaluated here at biases of -0.008 to +0.017, i.e. consistent with Minderer. Underconfidence appears only for web-scale / contrastive / self-supervised checkpoints, which were outside Minderer's scope. Furthermore, Minderer uses equal-mass binning while the primary metric here is equal-width M=15; the robustness check in Appendix C compares only equal-width M=15 vs M=100, which share the same failure mode and therefore say nothing about the choice of binning strategy. Most seriously, Appendix C explicitly promises equal-mass ECE results ("We re-compute ECE under equal-mass binning on the architecture-family case-study checkpoints") and those results appear nowhere in the paper, so the claim of comparability to Minderer cannot be verified by a reviewer.
Why this is consequential: the contribution is framed as correcting a widely cited result. If the paper's own measurements agree with that result for exactly the recipes it covered, the framing overstates the contribution; and the one analysis that would let a reviewer check the cross-paper comparison is absent.
Questions for authors
Q1. Does "recipe > architecture" still hold under accuracy matching (or with accuracy stratification)? Can you provide a formal effect-size / variance decomposition (e.g., a two-way architecture × × recipe analysis) to support the "consistently larger" statement?
Q2. The controlled ViT ablation reaches only bias ≈ − 0.011 ≈−0.011, an order of magnitude below the headline − 0.05 ∼ − 0.23 −0.05∼−0.23. Can the web-scale part provide a comparably strong controlled experiment (fixing at least two of corpus / objective / training length)? Otherwise, how do you argue that underconfidence is a "structural property of the recipe" rather than a by-product of web-scale pre-training?
Q3. Where are the equal-mass ECE values promised in Appendix C? If they differ from the equal-width results, how does that affect comparability with Minderer et al.?
Q4. What are the results of the medical 2 × 2 2×2 control (fine-tuning recipe × × architecture)? Under an identical fine-tuning recipe, does the under/overconfidence difference between foundation models and ResNets persist?
Q5. Does the learned logit scale (temperature parameter) of CLIP / SigLIP-type models contribute to the observed underconfidence? Are the conclusions stable after re-fitting the logit scale or using a linear probe?
Q6. Is the EVA-Giant bias of − 0.229 −0.229 an artifact of the evaluation pipeline (preprocessing / resolution / normalization)? Please provide its reliability diagram.
Q7. On which concrete downstream task does underconfidence cause a measurable loss? Could you provide a quantified example using risk-coverage or thresholded decisions?
5 comments
Response to Reviewer qCe1 (2/5)
Q1 Accuracy matching and variance decomposition.
Variance decomposition. We fit a Type-II ANOVA on our controlled ViT ablation grid. It attributes 63% of the variance in calibration bias to the recipe (ω² = 0.60) and 22% to the architecture (ω² = 0.21); architecture × recipe interactions explain a further 8% (R² 0.86 → 0.94). Pretraining data alone (49%) has more than twice the effect of the architecture. The same ordering holds for ECE (recipe 45% vs. architecture 16%). At a fixed architecture, the bias varies across recipes by a median of 0.22; at a fixed recipe, it varies across architectures by only 0.05. Our new medical ablation (Q4) confirms this ordering in a different domain, with an even smaller architecture share (at most 5%).
Accuracy control and stratification. We agree that accuracy is a potential confounder. Therefore, we tested on the ViT grid whether the recipe effect persists at fixed accuracy in two ways. (i) With top-1 accuracy as a covariate, the recipe still accounts for 55% of the variance in calibration bias and the architecture for 29%. Compared to the analysis without this control, the recipe share drops (63% → 55%) while the architecture share rises (22% → 29%), because part of the recipe effect operates through accuracy. The accuracy-controlled estimates are therefore conservative for the recipe. (ii) We split the runs of the ViT ablation grid into accuracy quintiles and repeated the decomposition within each stratum (table below; shares of the variance remaining after accuracy). Among the most accurate runs (top quintile, top-1 accuracy 0.78–0.86), the recipe explains 77% of the variance in bias and the architecture only 19%; in the fourth quintile, the split is 72% vs. 24%. Across all strata, the recipe dominates in four of five; the two are on par only in the middle quintile.
| Accuracy quintile | Top-1 acc. | Recipe | Architecture |
|---|---|---|---|
| 1 | 0.54–0.66 | 64% | 35% |
| 2 | 0.66–0.70 | 55% | 42% |
| 3 | 0.70–0.74 | 48% | 51% |
| 4 | 0.74–0.78 | 72% | 24% |
| 5 | 0.78–0.86 | 77% | 19% |
Response to Reviewer qCe1 (3/5)
Q2 Web-scale pre-training.
In our analyses, web-scale pre-training is not an alternative explanation to the recipe but one of its components. Throughout the paper, the training recipe denotes all choices that determine how a model’s weights are inferred for a given architecture: the pre-training corpus, objective and schedule (e.g. supervised ImageNet-1k/21k, self-supervised DINOv3, contrastive CLIP) and the fine-tuning procedure (data augmentation, Mixup/CutMix, label smoothing). The architecture, in contrast, comprises the network design and size. When we call underconfidence a structural property of the recipe, we therefore mean that it is determined by the training setup as a whole rather than by the architecture. Several components of this setup contribute: strongly regularized fine-tuning makes models underconfident, and web-scale pre-training shifts the bias further in this direction. The strongest underconfidence arises where both are present, as in our headline models.
A controlled pre-training experiment at web scale, varying corpus, objective and training length, is beyond our compute budget and the rebuttal period. Instead, we ran a new controlled ablation on four medical datasets (Q4, new App. B.6). It fixes the architecture (ViT-B/16, ConvNeXt-B) and the pre-trained checkpoint (ImageNet-1k, ImageNet-21k, DINOv3) and varies only the fine-tuning recipe in a full factorial design. Unlike the ViT grid, this ablation reaches the magnitudes of our headline models: with the strongest recipe, backbones are clearly underconfident (bias from −0.084 to −0.164), whereas the same checkpoints are overconfident with the weakest recipe. At a fixed recipe, the web-scale DINOv3 checkpoint is the most underconfident backbone on every dataset (down to −0.164). Taken together, this further supports our conclusions from the controlled ViT ablation with lower absolute biases.
Q3 Equal-mass binning.
We thank the reviewer for pointing out that the equal-mass results were missing. App. C now reports ECE under equal-width (EW) and equal-mass (EM) binning, each with M = 15 and M = 100, for the case-study checkpoints (new Table 20). For all checkpoints with |b| ≥ 0.01, the four estimates differ by at most 0.005, and at a fixed number of bins, EW and EM differ by at most 0.003. The high-accuracy underconfident checkpoints are no exception: for the 20 checkpoints with accuracy ≥ 0.85 and b < 0, EW and EM differ by at most 0.001 (M = 15) and 0.003 (M = 100), because their miscalibration is one-sided and the ECE then equals |b| under any binning. These results are in line with Minderer et al.: under their estimator (EM, M = 100), we reproduce their ECE for ViT-B/16 (0.014 vs. 0.015, see W2).
Response to Reviewer qCe1 (1/5)
We thank the reviewer for finding the evaluation extensive and well organized and its implications for post-hoc recalibration actionable. Regarding the two main concerns, we now support the attribution to the training recipe with several new analyses, including variance decompositions with accuracy control and a controlled ablation on four medical datasets (W1, Q1, Q2, Q4), and we clarify that our study extends rather than revises Minderer et al. (W2, Q3).
W1 Recipe vs. architecture.
We agree that the central claim of the paper, that the training recipe affects calibration more strongly than the architecture, needed stronger support. We therefore performed three additional analyses:
Variance decomposition. A Type-II ANOVA on the controlled ViT ablation attributes 63% of the variance in calibration bias to the recipe and 22% to the architecture (see Q1).
Accuracy matching. Since recipes differ in the accuracy they reach, we repeated the decomposition with accuracy as a covariate and within accuracy quintiles. The recipe remains the dominant factor (55% vs. 29%; 77% vs. 19% among the most accurate runs; see Q1).
New ablation on medical datasets. We fine-tuned ViT-B/16 and ConvNeXt-B from different pre-trained checkpoints (ImageNet-1k, ImageNet-21k, and DINOv3) with identical training recipes in a full factorial grid on four medical datasets (see Q4). Again, the recipe dominates: it explains 79–97% of the variance in the bias, the architecture at most 5%.
Regarding the calibration bias values for the Vision Transformers mentioned by the reviewer: the large differences between ViTs are confined to ImageNet-1k pre-training, where they are driven mainly by ViT-L/16. In this setting, ViT-L/16 overfits and even reaches a lower accuracy than ViT-B/16, as already reported in the original study ([1], Figure 1). With ImageNet-21k pre-training, the differences between ViT sizes are small.
[1] Steiner et al., How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers, TMLR 2022.
W2 Relation to Minderer et al.
We agree that, for the recipes Minderer et al. studied, our results reproduce theirs, and we do not intend to position our manuscript as a revision of their work. For ViT-B/16 (IN-21k → IN-1k, AugReg), for example, the measurements released by Minderer et al. give ECE = 0.015 (equal-mass, M = 100) at an accuracy of 0.840; under the same estimator, we obtain 0.014 at an accuracy of 0.843 (see Q3). We regard this agreement as a validation of our pipeline.
Our contribution lies beyond that regime. Checkpoints fine-tuned from web-scale pre-training with strongly regularized recipes, which were outside the scope of Minderer et al., now dominate the set of models with very high accuracy that are favoured by practitioners (Fig. 2). For these, calibration moves into clear underconfidence. Sec. 4.1 already frames our study as an extension (“our broader population extends this picture”, ll. 242–243), but we agree that parts of the introduction could be read as a revision of Minderer et al. We have rephrased them accordingly (changes in blue in the uploaded revised manuscript).
The one point where our results refine their conclusions is the cause of calibration differences between model families. Minderer et al. hypothesized that the architecture plays a major role. Our controlled ablation holds the architecture fixed and varies the recipe, which tests this hypothesis directly within the ViT family; our new medical ablation additionally compares ViT-B/16 and ConvNeXt-B under identical recipes (Q4). As described in W1, Q1 and Q4, we now report this as a quantified effect, including variance decompositions and accuracy-controlled comparisons.