- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes Consistency Calibration (CC), a post-hoc method that replaces softmax confidence with the empirical frequency with which each class wins the argmax under i.i.d. perturbations applied at the data, feature, or logit level. Noise family and strength are selected on a validation set by minimizing ECE. A theory section shows that under Gumbel$(0,\tau)$ logit noise the consistency vector equals and that the Bregman Information of the perturbed logits is sandwiched by and . Experiments cover CIFAR-10/100, ImageNet-1K and ImageNet-LT across six architectures, comparing against eight post-hoc and eight training-time calibration methods
Critical strengths
- Broad and well-organized empirical study. Four datasets and six backbones (ResNet-50/110, Wide-ResNet, DenseNet-121, Swin-B, ViT-B/16, ViT-B/32), compared against 8 post-hoc methods (TS/ETS/PTS/CTS/GC/FC) and 8 training-time methods (Brier/MMCE/LS/FLSD/FL/CLAS/ACLS), with supplementary AdaECE / CECE / accuracy results. CC attains the lowest ECE in 14 of 16 dataset-model combinations in Table 1, and improves 63 of 64 train-time-plus-CC pairs in Table 2. This is a substantial amount of work.
- The method is simple and the motivation is clear. Argmax-flip frequency under perturbation requires one forward pass plus argmax operations, no retraining or ensembling.
Critical weaknesses
- The theory covers exactly the case that provably coincides with the baseline, so the theory and the headline empirical claim contradict each other. Theorem 1 and Eq. (10) state that under i.i.d. Gumbel$(0,\tau)$ perturbations the expected consistency vector is exactly . This is a direct consequence of the Gumbel-Max trick, i.e. logit-level CC under Gumbel noise is mathematically the same single-parameter family as Temperature Scaling with temperature . Since the paper also states that TS's temperature is selected by minimizing validation ECE (Appendix B), both methods optimize the same scalar family on the same criterion, so their optimal validation ECE must coincide up to Monte-Carlo error (order ). Yet Table 4 (ImageNet / ResNet-50) reports TS = 2.09 vs. Gumbel-CC = 0.87 (a 58% relative gap), while Table 1 reports CC = 1.53. These cannot all be true. Sec. 3.4 concedes that "our claim is therefore not that the Gumbel logit case is unrelated to temperature scaling", but no explanation is offered. So where does the main performance gain actually come from? If it comes from Gaussian/Uniform noise, then the theory does not cover the method that is actually used, and Sec. 3.4 titled "Theoretical Justification" does not justify it. If it comes from an implementation detail of the Gumbel case (e.g. how noise is shared, per-class scales, asymmetric perturbations), that detail must be specified. In short: the theory covers precisely the regime with no advantage, while the regime with the advantage (Gaussian/Uniform) lies outside the theory. Please either (i) revise the theoretical claim to state explicitly that the Gumbel case degenerates to TS and that the gain comes from Monte-Carlo discretization/nonlinearity (quantified), or (ii) explain the mechanism that makes the Gumbel implementation non-equivalent to TS.
- Direct numeric contradiction across tables, which affects confidence in the main claims. For the same configuration (ImageNet / ResNet-50 / Gaussian logit noise) three different values appear: Table 3 ("Consis. G") = 1.27, Table 4 ("Gaussian") = 1.53, and Table 1 ("CC (ours)") = 1.53. If 1.27 is correct, Table 1 should report 1.27, and a validation-ECE selection procedure should also have selected it rather than 1.53. In addition, Table 4's caption calls the three noise families "comparable", yet on ResNet-50 they span 0.87-1.71 (43-49% relative gap), which is a clear overstatement; and Table 4's own text claims CC beats TS for all three families while Sec. 3.4 implies Gumbel-CC is equivalent to TS. These three statements conflict. Request: provide the complete per-configuration raw numbers (including per-run values) and explain the protocol difference between Tables 3 and 4 (different validation splits? different noise parameterization? different ? different bin count?).
Questions for authors
- Please explain the contradiction between Table 4 (Gumbel-CC = 0.87, far better than TS = 2.09) and Theorem 1 / Eq. (10) (Gumbel implies , i.e. equivalent to TS). Is the Gumbel noise i.i.d. per class with a single scale? Is TS fitted on the same validation set, the same search grid and the same ECE criterion? If Gumbel-CC is indeed equivalent to TS, which noise family produces CC's gains in Table 1, and does the theory cover it?
- Why do Table 3 (Consis. G = 1.27), Table 4 (Gaussian = 1.53) and Table 1 (CC = 1.53) disagree for ImageNet/ResNet-50? Please provide the per-configuration raw values.
- What are the selected noise family, and for every dataset-model configuration in the main experiments? How consistent is the validation ECE-versus- curve with the test curve (i.e. is the validation-based selection overfitting)? Why when Appendix D argues 16 suffices?
- What are the measured inference time, memory and random-number-generation overhead compared with TS, GC and FC?
- Have you tried smoothing the frequency vector (Laplace/Jeffreys/temperature post-processing) to fix NLL? Does the smoothed version still beat TS on ECE? Can you provide reliability diagrams and risk-coverage / AURC?
- What is the ECE of a single-scalar monotone recalibration of the top1-minus-top2 logit gap in each configuration? If it is close to CC, what is the necessity of perturbation counting? Can ProCal / Dirichlet / matrix scaling / local TS results be added?
- What varies across the 5 runs? Please report standard deviations or bootstrap confidence intervals for all main tables.
- At the selected , what is the ratio ? Is the theorem still "tight" at the practical operating point?
- What notion of uncertainty do data-level and logit-level CC respectively correspond to, and why call them one framework?
4 comments
Q1. Matched Gumbel–TS comparison and the original ImageNet gap
We reran the comparison with identical validation data, the grid , and the same ECE criterion. Gumbel draws are i.i.d. per class with a single scale. Three independent validation/test seed pairs give:
| Method | Selected , three runs | Mean validation ECE (%) | Test ECE (%), mean ± SD |
|---|---|---|---|
| Analytic TS | 2.48 | 1.2733 | 1.4369 |
| Gumbel CC, | 2.75 / 2.67 / 2.60 | 0.9208 | 1.1383 ± 0.2456 |
| Gumbel CC, | 2.50 / 2.50 / 2.50 | 1.0400 | 1.3867 ± 0.0062 |
| Gumbel CC, | 2.49 / 2.50 / 2.49 | 1.1774 | 1.3950 ± 0.0339 |
| Gumbel CC, | 2.48 / 2.49 / 2.48 | 1.2761 | 1.4134 ± 0.0002 |
At fixed , mean absolute probability error decreases from 0.004279 to 0.001458 to 0.000461 for . The log–log slope is −0.484, consistent with Monte Carlo convergence to TS.
The recovered TS component uses NLL fitting, whereas Appendix B describes ECE selection. This identifies a protocol distinction but does not fully reconcile the original ImageNet 0.87-versus-2.09 comparison. We therefore do not use that gap as evidence of an intrinsic Gumbel advantage.
The recovered main implementation uses Gaussian/Uniform perturbations. The deterministic Uniform result directly supports gains from a different link, while the Gumbel theorem remains scoped to its stated noise model.
Q2. Per-run evidence for the cross-table discrepancy
The five recovered values are listed in W2. Their mean is 1.528%, matching the rounded 1.53 in Tables 1 and 4, while Table 3's 1.27 matches one run.
For this configuration, the available records support an aggregation distinction rather than choosing 1.27 as a replacement for the mean. This explanation concerns the specific Gaussian configuration identified in the review.
Q3. Configuration details, selection stability, and the choice of
The new controlled study uses the following explicit search protocol:
| Map | Validation search | Selected parameter at |
|---|---|---|
| TS / Gumbel | –, step 0.01 | TS: 2.48; Gumbel: 2.49/2.50/2.49 |
| Uniform | Total width 0.1–30.0, step 0.1 | Width 16.1 |
| Gaussian | Standard deviation 0.1–15.0, step 0.1 |
Uniform width 16.1 is equivalent to centered noise on . Uniform has lower validation ECE than Gaussian in this comparison. This table specifies the new searches; a fully verified per-entry configuration inventory for all historical main-table results is not yet available.
To test selection sensitivity, we used 20 validation bootstrap resamples with deterministic probability evaluation. Uniform widths selected from 12–20 in steps of 0.5 lie between 16.0 and 16.5. Test ECE is 1.0061 ± 0.0923%, compared with 1.3864 ± 0.1844% for refitted TS.
can provide useful calibration at lower cost, while offers finer probability resolution and lower sampling variance. These are different operating points, rather than a claim that 1,000 is always necessary or always yields lower ECE. Q1 quantifies this distinction.
Q4. Measured time, memory, and random-number generation
On cached CIFAR-10 logits, single-thread CPU NumPy evaluation with batch size 128 and float64 gives:
| Operation | Median time per batch |
|---|---|
| TS | 0.0132 ms |
| CC, | 10.365 ms |
| RNG and scaling, separate microbenchmark | 1.980 ms |
These measurements exclude the network forward pass and parameter fitting. The RNG measurement is a separate microbenchmark, not an additive profiler breakdown.
For a controlled 1,000-class logit-shaped workload with 20-draw chunks, measured whole-process peak CPU RSS is 85.15 MiB for CC versus 48.93 MiB for TS. These are CPU process-memory measurements, not GPU-memory or end-to-end network measurements. Chunking bounds temporary storage without changing the vote estimator.
The measured comparison is with TS; GC/FC latency and GPU memory are outside this measurement. The efficiency claim is that logit-level CC avoids additional model forward passes, while its post-processing cost scales with and the class count.
Q5. Smoothing, reliability, and selective prediction
We tested additive smoothing,
,
as well as blending with TS. The CC rows below report mean ± sample SD over three test-noise seeds, with the fitted Uniform width fixed at 16.1.
| Method | Test ECE (%) | Test NLL |
|---|---|---|
| TS, validation-NLL fitted | 1.3730 | 0.20381 |
| CC + Jeffreys, | 0.7695 ± 0.0464 | 0.18278 ± 0.00049 |
| CC + Laplace, | 1.1238 ± 0.0510 | 0.18496 ± 0.00038 |
| CC + validation-selected | 0.8003 ± 0.0321 | 0.18286 ± 0.00079 |
| CC + 5% TS blending | 0.9022 ± 0.0270 | 0.18327 ± 0.00075 |
For tuned additive smoothing and blending, strengths are selected by validation NLL; Jeffreys and Laplace use their fixed standard values.
Temperature post-processing after Jeffreys smoothing selects temperature 0.9377 and yields ECE 1.1229%, NLL 0.18379 in a single-seed check. These positive-support variants demonstrate simultaneous ECE and NLL improvements in this experiment.
For selective prediction, all methods rank the same original classifier decisions, with ties averaged consistently:
| Confidence score | AURC × 1,000 |
|---|---|
| TS-NLL | 10.7519 |
| Uniform CC | 8.6302 ± 0.0886 |
| Scalar margin | 7.9836 |
Lower AURC is better. Calibration quality and ranking quality therefore provide complementary assessments; the strongest calibration result need not give the strongest ranking.
Q6. Scalar margin and stronger calibration baselines
The scalar baseline
,
fitted on the same temperature grid by validation ECE, selects and gives ECE 1.5285%. Deterministic Uniform CC gives 0.9101%.
This comparison supports the value of the induced link in the tested configuration. Monte Carlo counting is a convenient estimator rather than a mathematical necessity when integration is tractable.
We also implemented matrix and Dirichlet scaling with ODIR regularization following [2], selecting regularization by five-fold cross-validation within the validation set and refitting on all validation examples:
| Method | Test ECE (%) | Test NLL |
|---|---|---|
| Matrix-ODIR | 1.4105 | 0.19347 |
| Dirichlet-ODIR | 1.4628 | 0.19422 |
| CC + Jeffreys | 0.7695 ± 0.0464 | 0.18278 ± 0.00049 |
These are additional implementations on one configuration. ProCal [3] requires feature-space neighborhoods, which the available cached logits do not provide; no ProCal or local-TS result is claimed here.
Q7. Run variation and uncertainty estimates
We distinguish the sources of variation explicitly:
- Historical records: W2 reports the five available Gaussian ImageNet values and their sample SD. Complete training/split/RNG provenance is not established for every main-table entry.
- Sampling variation: the three-seed CC results hold the checkpoint, validation split, and fitted parameters fixed, varying only test-time perturbations.
- Selection variation: 20 validation bootstrap resamples refit the calibration parameters.
- Test-sample variation: 1,000 paired test bootstrap resamples assess differences between fixed fitted maps.
The paired 95% interval for ECE(CC − TS-NLL) is [−0.675, −0.185] percentage points. For ECE(Jeffreys-CC − TS-NLL), it is [−1.002, −0.141].
These intervals condition on the checkpoint and fitted maps. They quantify the new evidence without conflating Monte Carlo seeds or bootstrap resamples with independent model training.
Q8. Practical size of the theorem's gap
At validation-selected Gumbel scale , we evaluated 512 randomly selected test examples using 5,000 perturbations each and three seeds. For
we obtain:
| Quantity | Mean value |
|---|---|
| Entropy term | 0.09185 nats |
| Estimated residual | 0.02271 nats |
| Their sum, estimating | 0.11456 nats |
| Ratio of means | 19.8% |
The entropy term therefore accounts for approximately 80.2% of mean Bregman Information at this operating point. This provides a quantitative practical characterization rather than describing the bound as uniformly tight; the theorem's worst-case additive allowance remains .
Q9. Meaning of uncertainty at different perturbation levels
Data-level CC measures prediction sensitivity to the chosen input transformations. Feature-level CC measures sensitivity in representation space. Logit-level CC measures competition between classes under a specified output perturbation.
They share the same operational construction: perturb a representation, propagate through the remaining model, and estimate class win probabilities. They nevertheless probe different sensitivities and incur different costs.
Calling them one framework refers to this shared construction and calibration procedure. It does not identify all three with the same physical uncertainty source or imply an automatic epistemic/aleatoric decomposition.
References
[1] Berthet et al. Learning with Differentiable Perturbed Optimizers. NeurIPS, 2020.
[2] Kull et al. Beyond Temperature Scaling: Obtaining Well-Calibrated Multi-Class Probabilities with Dirichlet Calibration. NeurIPS, 2019.
[3] Xiong et al. Proximity-Informed Calibration for Deep Neural Networks. NeurIPS, 2023.
Response to Reviewer 1Vvx
Thank you for the careful reading and for recognizing the breadth of the empirical study. We address the theoretical distinction, numerical provenance, and requested additional analyses separately below.
Unless explicitly labeled as historical ImageNet records, all new accuracy/calibration experiments use one CIFAR-10/ResNet-50 checkpoint, 5,000 validation examples, and 10,000 test examples. ECE is reported in percent with 15 equal-width bins. Hyperparameters are selected using validation data only.
W1. Gumbel equivalence and the source of the calibration gains
Gumbel-CC and TS have the same population probability map under i.i.d. per-class Gumbel noise with one common scale. Section 3.4 already identifies this relationship. The theorem gives a structural entropy–Bregman characterization of that special case; it does not establish an advantage over TS or a general calibration guarantee for Gaussian/Uniform perturbations.
However, equivalence in the Gumbel case does not imply that gains from other noise families must arise from Monte Carlo discretization. We tested this directly: deterministic integration of Uniform win probabilities gives ECE 0.9101%, versus 1.4369% for ECE-fitted TS, at a width selected using validation data. A different induced link can therefore improve calibration even after sampling error is removed.
We also explicitly credit the perturbed-maximizer construction in Berthet et al. [1]. The contribution is the calibration formulation, structural analysis, and empirical investigation across perturbation levels and training methods.
W2. The 1.27 versus 1.53 values and the noise-family comparison
The recovered ImageNet/ResNet-50 Gaussian records support a distinction between a single-run result and the aggregate:
The 1.27 entry matches the first run, while 1.53 matches the five-run mean. These require consistent aggregation labels; selecting the lowest test result would not be the appropriate correction.
Likewise, noise-family selection depends on validation ECE, so a smaller test value alone does not determine which family should be selected. The precise interpretation is that the families provide alternative calibration links with configuration-dependent performance, rather than numerically interchangeable results.