Beyond Overconfidence: Revisiting Calibration in Modern Vision Models

Modern training recipes for large-scale vision models — combining aggressive augmentation, strong regularization, and web-scale pretraining — have driven substantial gains in image classification accuracy across diverse architectures and benchmarks. Their joint effect on calibration, however, remains poorly characterized. We present a large-scale calibration benchmark comprising 1218 ImageNet classifiers, a controlled ablation across 2015 ViT configurations, and a transfer evaluation on four medical imaging tasks, spanning training paradigms from classical from-scratch baselines to web-scale pretrained models with aggressive augmentation and regularization. Four findings emerge: (i) in the high-accuracy regime, modern classifiers are systematically underconfident: 98.4% of the checkpoints in the top 5% of accuracy have negative calibration bias; (ii) the training recipe, not the architecture, is the primary driver of the calibration regime, evidenced by a recipe-shaped Pareto front, consistent shifts across two architecture families, and a controlled ViT ablation; (iii) under severe distribution shift, in-distribution underconfidence reverses to overconfidence; while post-hoc calibration mitigates underconfidence in the in-distribution setting, this exacerbates overconfidence under severe distribution shifts; and (iv) the recipe-driven regime holds for medical imaging: modern-recipe models remain underconfident across four imaging modalities, while supervised ResNet baselines are overconfident on the same tasks. Together, these findings suggest that modern training recipes induce a structural calibration regime that can reverse under distribution shift.