acceptodds
Under review as a conference paper at ICLR 2027

Consistency Calibration: Post-hoc Confidence from Prediction Stability

Abstract

Calibration is critical for deploying deep learning models in safety-critical applications where reliable confidence estimates guide downstream decisions. Existing studies predominantly view calibration from the reliability perspective, aligning predicted confidence with empirical accuracy, while neglecting whether model predictions remain consistent under perturbations. In this work, we propose Consistency Calibration (CC), a post-hoc method that complements reliability-based calibration by measuring prediction stability. Prior works have explored consistency mainly at the model level, through ensembles or dropout sampling, which aggregate multiple perturbed outputs to smooth predictions but incur substantial computational overhead. In contrast, CC interprets the perturbation process itself as an uncertainty signal, replacing confidence with the empirical stability of predictions under data-, feature-, or logit-level perturbations. We further establish a theoretical connection showing that, under Gumbel logit perturbations, the consistency vector recovers a temperature-scaled softmax distribution and is linked to the Bregman Information of the logit distribution through Renyi entropy. This clarifies the relationship between CC and Temperature Scaling while providing a unified perturbation-consistency interpretation. Extensive experiments on CIFAR-10/100, ImageNet, and ImageNet-LT demonstrate that CC achieves competitive calibration performance compared to both post-hoc and training-time methods, while remaining simple and efficient.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1VvxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Consistency Calibration (CC), a post-hoc method that replaces softmax confidence with the empirical frequency with which each class wins the argmax under TT i.i.d. perturbations applied at the data, feature, or logit level. Noise family and strength are selected on a validation set by minimizing ECE. A theory section shows that under Gumbel$(0,\tau)$ logit noise the consistency vector equals softmax(z0/τ)\mathrm{softmax}(z_0/\tau) and that the Bregman Information of the perturbed logits is sandwiched by (τ−1)Hτ(c)(\tau-1)H_\tau(c) and (τ−1)Hτ(c)+ln⁡K(\tau-1)H_\tau(c)+\ln K. Experiments cover CIFAR-10/100, ImageNet-1K and ImageNet-LT across six architectures, comparing against eight post-hoc and eight training-time calibration methods

Critical strengths
  1. Broad and well-organized empirical study. Four datasets and six backbones (ResNet-50/110, Wide-ResNet, DenseNet-121, Swin-B, ViT-B/16, ViT-B/32), compared against 8 post-hoc methods (TS/ETS/PTS/CTS/GC/FC) and 8 training-time methods (Brier/MMCE/LS/FLSD/FL/CLAS/ACLS), with supplementary AdaECE / CECE / accuracy results. CC attains the lowest ECE in 14 of 16 dataset-model combinations in Table 1, and improves 63 of 64 train-time-plus-CC pairs in Table 2. This is a substantial amount of work.
  2. The method is simple and the motivation is clear. Argmax-flip frequency under perturbation requires one forward pass plus TT argmax operations, no retraining or ensembling.
Critical weaknesses
  1. The theory covers exactly the case that provably coincides with the baseline, so the theory and the headline empirical claim contradict each other. Theorem 1 and Eq. (10) state that under i.i.d. Gumbel$(0,\tau)$ perturbations the expected consistency vector is exactly c=softmax(z0/τ)c=\mathrm{softmax}(z_0/\tau). This is a direct consequence of the Gumbel-Max trick, i.e. logit-level CC under Gumbel noise is mathematically the same single-parameter family as Temperature Scaling with temperature . Since the paper also states that TS's temperature is selected by minimizing validation ECE (Appendix B), both methods optimize the same scalar family on the same criterion, so their optimal validation ECE must coincide up to Monte-Carlo error (order O(T−1/2)≈0.01O(T^{-1/2})\approx 0.01). Yet Table 4 (ImageNet / ResNet-50) reports TS = 2.09 vs. Gumbel-CC = 0.87 (a 58% relative gap), while Table 1 reports CC = 1.53. These cannot all be true. Sec. 3.4 concedes that "our claim is therefore not that the Gumbel logit case is unrelated to temperature scaling", but no explanation is offered. So where does the main performance gain actually come from? If it comes from Gaussian/Uniform noise, then the theory does not cover the method that is actually used, and Sec. 3.4 titled "Theoretical Justification" does not justify it. If it comes from an implementation detail of the Gumbel case (e.g. how noise is shared, per-class scales, asymmetric perturbations), that detail must be specified. In short: the theory covers precisely the regime with no advantage, while the regime with the advantage (Gaussian/Uniform) lies outside the theory. Please either (i) revise the theoretical claim to state explicitly that the Gumbel case degenerates to TS and that the gain comes from Monte-Carlo discretization/nonlinearity (quantified), or (ii) explain the mechanism that makes the Gumbel implementation non-equivalent to TS.
  2. Direct numeric contradiction across tables, which affects confidence in the main claims. For the same configuration (ImageNet / ResNet-50 / Gaussian logit noise) three different values appear: Table 3 ("Consis. G") = 1.27, Table 4 ("Gaussian") = 1.53, and Table 1 ("CC (ours)") = 1.53. If 1.27 is correct, Table 1 should report 1.27, and a validation-ECE selection procedure should also have selected it rather than 1.53. In addition, Table 4's caption calls the three noise families "comparable", yet on ResNet-50 they span 0.87-1.71 (43-49% relative gap), which is a clear overstatement; and Table 4's own text claims CC beats TS for all three families while Sec. 3.4 implies Gumbel-CC is equivalent to TS. These three statements conflict. Request: provide the complete per-configuration raw numbers (including per-run values) and explain the protocol difference between Tables 3 and 4 (different validation splits? different noise parameterization? different TT? different bin count?).
Questions for authors
  1. Please explain the contradiction between Table 4 (Gumbel-CC = 0.87, far better than TS = 2.09) and Theorem 1 / Eq. (10) (Gumbel implies softmax(z0/τ)\mathrm{softmax}(z_0/\tau), i.e. equivalent to TS). Is the Gumbel noise i.i.d. per class with a single scale? Is TS fitted on the same validation set, the same search grid and the same ECE criterion? If Gumbel-CC is indeed equivalent to TS, which noise family produces CC's gains in Table 1, and does the theory cover it?
  2. Why do Table 3 (Consis. G = 1.27), Table 4 (Gaussian = 1.53) and Table 1 (CC = 1.53) disagree for ImageNet/ResNet-50? Please provide the per-configuration raw values.
  3. What are the selected noise family, ϵ\epsilon and TT for every dataset-model configuration in the main experiments? How consistent is the validation ECE-versus-ϵ\epsilon curve with the test curve (i.e. is the validation-based selection overfitting)? Why T=1000T=1000 when Appendix D argues 16 suffices?
  4. What are the measured inference time, memory and random-number-generation overhead compared with TS, GC and FC?
  5. Have you tried smoothing the frequency vector (Laplace/Jeffreys/temperature post-processing) to fix NLL? Does the smoothed version still beat TS on ECE? Can you provide reliability diagrams and risk-coverage / AURC?
  6. What is the ECE of a single-scalar monotone recalibration of the top1-minus-top2 logit gap in each configuration? If it is close to CC, what is the necessity of perturbation counting? Can ProCal / Dirichlet / matrix scaling / local TS results be added?
  7. What varies across the 5 runs? Please report standard deviations or bootstrap confidence intervals for all main tables.
  8. At the selected τ\tau, what is the ratio (τ−1)Hτ(c)/Bz(\tau-1)H_\tau(c)/B_z? Is the theorem still "tight" at the practical operating point?
  9. What notion of uncertainty do data-level and logit-level CC respectively correspond to, and why call them one framework?
4 comments

Response to Reviewer 1Vvx

Thank you for the careful reading and for recognizing the breadth of the empirical study. We address the theoretical distinction, numerical provenance, and requested additional analyses separately below.

Unless explicitly labeled as historical ImageNet records, all new accuracy/calibration experiments use one CIFAR-10/ResNet-50 checkpoint, 5,000 validation examples, and 10,000 test examples. ECE is reported in percent with 15 equal-width bins. Hyperparameters are selected using validation data only.

W1. Gumbel equivalence and the source of the calibration gains

Gumbel-CC and TS have the same population probability map under i.i.d. per-class Gumbel noise with one common scale. Section 3.4 already identifies this relationship. The theorem gives a structural entropy–Bregman characterization of that special case; it does not establish an advantage over TS or a general calibration guarantee for Gaussian/Uniform perturbations.

However, equivalence in the Gumbel case does not imply that gains from other noise families must arise from Monte Carlo discretization. We tested this directly: deterministic integration of Uniform win probabilities gives ECE 0.9101%, versus 1.4369% for ECE-fitted TS, at a width selected using validation data. A different induced link can therefore improve calibration even after sampling error is removed.

We also explicitly credit the perturbed-maximizer construction in Berthet et al. [1]. The contribution is the calibration formulation, structural analysis, and empirical investigation across perturbation levels and training methods.

W2. The 1.27 versus 1.53 values and the noise-family comparison

The recovered ImageNet/ResNet-50 Gaussian records support a distinction between a single-run result and the aggregate:

Recorded runTest ECE (%)
11.27
21.33
31.65
41.73
51.66
Mean ± sample SD1.528 ± 0.211

The 1.27 entry matches the first run, while 1.53 matches the five-run mean. These require consistent aggregation labels; selecting the lowest test result would not be the appropriate correction.

Likewise, noise-family selection depends on validation ECE, so a smaller test value alone does not determine which family should be selected. The precise interpretation is that the families provide alternative calibration links with configuration-dependent performance, rather than numerically interchangeable results.

Q1. Matched Gumbel–TS comparison and the original ImageNet gap

We reran the comparison with identical validation data, the grid τ=0.10,0.11,…,10.00\tau=0.10,0.11,\ldots,10.00, and the same ECE criterion. Gumbel draws are i.i.d. per class with a single scale. Three independent validation/test seed pairs give:

MethodSelected τ\tau, three runsMean validation ECE (%)Test ECE (%), mean ± SD
Analytic TS2.481.27331.4369
Gumbel CC, T=16T=162.75 / 2.67 / 2.600.92081.1383 ± 0.2456
Gumbel CC, T=100T=1002.50 / 2.50 / 2.501.04001.3867 ± 0.0062
Gumbel CC, T=1,000T=1,0002.49 / 2.50 / 2.491.17741.3950 ± 0.0339
Gumbel CC, T=10,000T=10,0002.48 / 2.49 / 2.481.27611.4134 ± 0.0002

At fixed τ=2.48\tau=2.48, mean absolute probability error decreases from 0.004279 to 0.001458 to 0.000461 for T=100,1,000,10,000T=100,1,000,10,000. The log–log slope is −0.484, consistent with Monte Carlo convergence to TS.

The recovered TS component uses NLL fitting, whereas Appendix B describes ECE selection. This identifies a protocol distinction but does not fully reconcile the original ImageNet 0.87-versus-2.09 comparison. We therefore do not use that gap as evidence of an intrinsic Gumbel advantage.

The recovered main implementation uses Gaussian/Uniform perturbations. The deterministic Uniform result directly supports gains from a different link, while the Gumbel theorem remains scoped to its stated noise model.

Q2. Per-run evidence for the cross-table discrepancy

The five recovered values are listed in W2. Their mean is 1.528%, matching the rounded 1.53 in Tables 1 and 4, while Table 3's 1.27 matches one run.

For this configuration, the available records support an aggregation distinction rather than choosing 1.27 as a replacement for the mean. This explanation concerns the specific Gaussian configuration identified in the review.

Q3. Configuration details, selection stability, and the choice of TT

The new controlled study uses the following explicit search protocol:

MapValidation searchSelected parameter at T=1,000T=1,000
TS / Gumbelτ=0.10\tau=0.10–10.0010.00, step 0.01TS: 2.48; Gumbel: 2.49/2.50/2.49
UniformTotal width 0.1–30.0, step 0.1Width 16.1
GaussianStandard deviation 0.1–15.0, step 0.1σ=3.8\sigma=3.8

Uniform width 16.1 is equivalent to centered noise on [−8.05,8.05][-8.05,8.05]. Uniform has lower validation ECE than Gaussian in this comparison. This table specifies the new searches; a fully verified per-entry configuration inventory for all historical main-table results is not yet available.

To test selection sensitivity, we used 20 validation bootstrap resamples with deterministic probability evaluation. Uniform widths selected from 12–20 in steps of 0.5 lie between 16.0 and 16.5. Test ECE is 1.0061 ± 0.0923%, compared with 1.3864 ± 0.1844% for refitted TS.

T=16T=16 can provide useful calibration at lower cost, while T=1,000T=1,000 offers finer probability resolution and lower sampling variance. These are different operating points, rather than a claim that 1,000 is always necessary or always yields lower ECE. Q1 quantifies this distinction.

Q4. Measured time, memory, and random-number generation

On cached CIFAR-10 logits, single-thread CPU NumPy evaluation with batch size 128 and float64 gives:

OperationMedian time per batch
TS0.0132 ms
CC, T=1,000T=1,00010.365 ms
RNG and scaling, separate microbenchmark1.980 ms

These measurements exclude the network forward pass and parameter fitting. The RNG measurement is a separate microbenchmark, not an additive profiler breakdown.

For a controlled 1,000-class logit-shaped workload with 20-draw chunks, measured whole-process peak CPU RSS is 85.15 MiB for CC versus 48.93 MiB for TS. These are CPU process-memory measurements, not GPU-memory or end-to-end network measurements. Chunking bounds temporary storage without changing the vote estimator.

The measured comparison is with TS; GC/FC latency and GPU memory are outside this measurement. The efficiency claim is that logit-level CC avoids additional model forward passes, while its post-processing cost scales with TT and the class count.

Q5. Smoothing, reliability, and selective prediction

We tested additive smoothing,

qk=(nk+α)/(T+Kα)q_k=(n_k+\alpha)/(T+K\alpha),

as well as blending with TS. The CC rows below report mean ± sample SD over three test-noise seeds, with the fitted Uniform width fixed at 16.1.

MethodTest ECE (%)Test NLL
TS, validation-NLL fitted1.37300.20381
CC + Jeffreys, α=0.5\alpha=0.50.7695 ± 0.04640.18278 ± 0.00049
CC + Laplace, α=1\alpha=11.1238 ± 0.05100.18496 ± 0.00038
CC + validation-selected α=0.1\alpha=0.10.8003 ± 0.03210.18286 ± 0.00079
CC + 5% TS blending0.9022 ± 0.02700.18327 ± 0.00075

For tuned additive smoothing and blending, strengths are selected by validation NLL; Jeffreys and Laplace use their fixed standard values.

Temperature post-processing after Jeffreys smoothing selects temperature 0.9377 and yields ECE 1.1229%, NLL 0.18379 in a single-seed check. These positive-support variants demonstrate simultaneous ECE and NLL improvements in this experiment.

For selective prediction, all methods rank the same original classifier decisions, with ties averaged consistently:

Confidence scoreAURC × 1,000
TS-NLL10.7519
Uniform CC8.6302 ± 0.0886
Scalar margin7.9836

Lower AURC is better. Calibration quality and ranking quality therefore provide complementary assessments; the strongest calibration result need not give the strongest ranking.

Q6. Scalar margin and stronger calibration baselines

The scalar baseline

q=sigmoid⁡((z(1)−z(2))/t)q=\operatorname{sigmoid}((z_{(1)}-z_{(2)})/t),

fitted on the same temperature grid by validation ECE, selects t=3.48t=3.48 and gives ECE 1.5285%. Deterministic Uniform CC gives 0.9101%.

This comparison supports the value of the induced link in the tested configuration. Monte Carlo counting is a convenient estimator rather than a mathematical necessity when integration is tractable.

We also implemented matrix and Dirichlet scaling with ODIR regularization following [2], selecting regularization by five-fold cross-validation within the validation set and refitting on all validation examples:

MethodTest ECE (%)Test NLL
Matrix-ODIR1.41050.19347
Dirichlet-ODIR1.46280.19422
CC + Jeffreys0.7695 ± 0.04640.18278 ± 0.00049

These are additional implementations on one configuration. ProCal [3] requires feature-space neighborhoods, which the available cached logits do not provide; no ProCal or local-TS result is claimed here.

Q7. Run variation and uncertainty estimates

We distinguish the sources of variation explicitly:

  • Historical records: W2 reports the five available Gaussian ImageNet values and their sample SD. Complete training/split/RNG provenance is not established for every main-table entry.
  • Sampling variation: the three-seed CC results hold the checkpoint, validation split, and fitted parameters fixed, varying only test-time perturbations.
  • Selection variation: 20 validation bootstrap resamples refit the calibration parameters.
  • Test-sample variation: 1,000 paired test bootstrap resamples assess differences between fixed fitted maps.

The paired 95% interval for ECE(CC − TS-NLL) is [−0.675, −0.185] percentage points. For ECE(Jeffreys-CC − TS-NLL), it is [−1.002, −0.141].

These intervals condition on the checkpoint and fitted maps. They quantify the new evidence without conflating Monte Carlo seeds or bootstrap resamples with independent model training.

Q8. Practical size of the theorem's gap

At validation-selected Gumbel scale τ=2.48\tau=2.48, we evaluated 512 randomly selected test examples using 5,000 perturbations each and three seeds. For

Bz=(τ−1)Hτ(c)+G,B_z=(\tau-1)H_\tau(c)+G,

we obtain:

QuantityMean value
Entropy term (τ−1)Hτ(c)(\tau-1)H_\tau(c)0.09185 nats
Estimated residual GG0.02271 nats
Their sum, estimating BzB_z0.11456 nats
Ratio of means G‾/B‾z\overline G/\overline B_z19.8%

The entropy term therefore accounts for approximately 80.2% of mean Bregman Information at this operating point. This provides a quantitative practical characterization rather than describing the bound as uniformly tight; the theorem's worst-case additive allowance remains ln⁡K\ln K.

Q9. Meaning of uncertainty at different perturbation levels

Data-level CC measures prediction sensitivity to the chosen input transformations. Feature-level CC measures sensitivity in representation space. Logit-level CC measures competition between classes under a specified output perturbation.

They share the same operational construction: perturb a representation, propagate through the remaining model, and estimate class win probabilities. They nevertheless probe different sensitivities and incur different costs.

Calling them one framework refers to this shared construction and calibration procedure. It does not identify all three with the same physical uncertainty source or imply an automatic epistemic/aleatoric decomposition.

References

[1] Berthet et al. Learning with Differentiable Perturbed Optimizers. NeurIPS, 2020.

[2] Kull et al. Beyond Temperature Scaling: Obtaining Well-Calibrated Multi-Class Probabilities with Dirichlet Calibration. NeurIPS, 2019.

[3] Xiong et al. Proximity-Informed Calibration for Deep Neural Networks. NeurIPS, 2023.

Reviewer_ZrXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "Consistency Calibration," a method that applies random perturbations to inputs, features, or logits, and uses the frequency with which each class emerges as the argmax winner—following these perturbations—as the predicted probability. The default logit-level implementation requires no additional model forward passes. The authors further demonstrate that, under Gumbel perturbations, this winning frequency is equivalent to a temperature-scaled softmax, and they establish a connection between its Rényi entropy and the Bregman Information of the perturbed logits. Experiments on CIFAR, ImageNet, and ImageNet-LT show that Consistency Calibration (CC) outperforms various post-processing and training-time calibration methods in terms of ECE, AdaECE, and CECE.

Critical strengths

1、The method is simple and easy to apply, consistently improving top-label calibration metrics across various datasets and architectures. 2、It provides a unified perspective on perturbation stability—encompassing data, features, and logit layers—includes thorough ablation studies, and transparently reports any degradation in NLL.

Critical weaknesses

1、The proposed method is closely related to classical random utility models, "perturb-and-MAP," and randomized smoothing; in the specific case of the Gumbel distribution, it reduces strictly to Temperature Scaling, suggesting that the work’s novelty is not sufficiently established. 2、While the method is heavily optimized for binned Expected Calibration Error (ECE), it significantly degrades Negative Log-Likelihood (NLL) on the ImageNet dataset. The theoretical analysis addresses only the dispersion of perturbations and offers no guarantee of true probability calibration; furthermore, the evaluation lacks assessments regarding selective prediction, performance under distribution shift, and actual computational efficiency.

Questions for authors

What is the relationship between CC and random utility models, "perturb-and-MAP," and randomized smoothing? Beyond using class win rates for ECE calculation, what are the novel technical contributions of this method?

When NLL on ImageNet degrades by more than 48%, can the result still be considered "probabilistic calibration"? Is it possible to improve both ECE and NLL by incorporating Dirichlet/Laplace smoothing or by blending the results with the original softmax outputs?

Theorem 1 only establishes a link between perturbation dispersion and win-rate entropy; at what point is this quantity connected to the true probability of correctness? Are there any calibration guarantees that hold under conditions beyond "toy" (simplified/synthetic) scenarios?

2 comments

Thank you for recognizing the method's simplicity, the breadth of the evaluation, and the perspective across perturbation levels. Your questions motivated additional experiments on smoothing, proper scoring, selective prediction, and computational cost.

The new experiments use one CIFAR-10/ResNet-50 checkpoint with 5,000 validation and 10,000 test examples. ECE uses 15 equal-width bins unless specified otherwise and is reported in percent. All fitted parameters use validation data only.

W1. Prior work and the role of the Gumbel case

The underlying perturb-and-select construction has clear precedents. Berthet et al. [1] formalize the perturbed maximizer; perturb-and-MAP [2] studies perturbation-based sampling; randomized smoothing [3] constructs class probabilities through input perturbations. We explicitly position CC relative to these constructions.

Gumbel logit perturbations recover the TS population map. This is the tractable special case analyzed in Section 3.4, while Gaussian and Uniform perturbations induce other links. Their calibration behavior therefore needs empirical assessment rather than being inferred from Gumbel equivalence.

Our new deterministic Uniform evaluation gives ECE 0.9101%, compared with 1.4369% for ECE-fitted TS. The improvement persists after removing Monte Carlo sampling, supporting the practical value of the induced link.

W2. Evaluation beyond one ECE estimate

We expanded the analysis in four directions:

  1. Proper scoring: smoothing and blending improve both ECE and NLL relative to TS in the controlled experiment; see Q2.
  2. Selective prediction: holding the original classifier decisions fixed, Uniform CC achieves AURC × 1,000 of 8.6302 ± 0.0886, versus 10.7519 for NLL-fitted TS. Lower is better; the variation is across three noise seeds.
  3. Binning sensitivity: the calibration improvement persists across different bin counts, as shown below.
  4. Measured computation: on cached logits, a CPU NumPy implementation takes 10.365 ms per batch of 128 for CC with T=1,000T=1,000, versus 0.0132 ms for TS. Both exclude the network forward pass. CC avoids additional network evaluations while incurring sampling and counting overhead.
Number of ECE binsTS-NLLUniform CC
101.34330.8098
151.37300.8505
201.46440.8638
301.83390.9192
502.17831.0270

The binning comparison uses fixed fitted parameters and one test-noise seed, without retuning for each bin count. These new results concern in-distribution calibration and selective prediction; they do not establish distribution-shift robustness.

Q1. Relationship to existing methods and technical contribution

The common ingredient is prediction under perturbation, but the objectives differ. Random-utility and perturbed-optimization constructions define choice probabilities or smooth optimization operators. Perturb-and-MAP targets sampling through perturbed optimization. Randomized smoothing primarily studies robustness of the resulting classifier. CC studies how a validation-selected perturbation response can serve as a calibrated confidence map.

The contribution comprises:

  • A calibration formulation across representation levels, together with an efficient logits-only implementation requiring no retraining or extra network forward passes.
  • An entropy–Bregman characterization of the Gumbel case, explicitly identifying TS within this framework.
  • Empirical evidence about useful perturbation-induced links, including their compatibility with calibration-aware training. The deterministic Uniform experiment further isolates the link function from its Monte Carlo estimator.

Thus, the contribution concerns the calibration formulation, analysis, and evidence, rather than invention of the perturbed-argmax operator.

Q2. Can smoothing improve both ECE and NLL?

Yes, in the new controlled experiment. Better top-label ECE alone does not establish better full-distribution probabilities: NLL also penalizes very small probabilities assigned to the true class. Empirical frequencies can contain zeros, and bounded perturbations can also yield zero population probabilities for some classes. Positive smoothing or blending directly addresses this support issue.

We tested additive smoothing,

qk=nk+αT+Kα,q_k=\frac{n_k+\alpha}{T+K\alpha},

and blending q=(1−λ)c+λpTSq=(1-\lambda)c+\lambda p_{\mathrm{TS}}. Uniform width 16.1 and T=1,000T=1,000 are fixed. The following CC results are mean ± sample SD over three test-time noise seeds.

MethodTest ECE (%)Test NLL
TS, validation-NLL fitted1.37300.20381
CC + Jeffreys, α=0.5\alpha=0.50.7695 ± 0.04640.18278 ± 0.00049
CC + Laplace, α=1\alpha=11.1238 ± 0.05100.18496 ± 0.00038
CC + validation-selected α=0.1\alpha=0.10.8003 ± 0.03210.18286 ± 0.00079
CC + 5% TS blending0.9022 ± 0.02700.18327 ± 0.00075

The additive grid is {0.01,0.1,0.5,1,2,5,10}\{0.01,0.1,0.5,1,2,5,10\}, and the blending grid is {0.01,0.05,0.1,0.25,0.5}\{0.01,0.05,0.1,0.25,0.5\}; selection uses validation NLL. Jeffreys and Laplace are fixed standard choices.

We also tested blending with the original softmax, as requested. Validation selects a 10% softmax contribution, yielding ECE 1.1804% and NLL 0.20017 in a single-seed check.

These results demonstrate that improved top-label calibration and improved NLL can coexist within the CC approach. Their scope is the new CIFAR-10 experiment, rather than a completed ImageNet rerun.

Q3. How does Theorem 1 connect to correctness?

Theorem 1 characterizes the relationship between the Gumbel win vector and the dispersion of perturbed logits. It is a structural result, not a universal guarantee that a win probability equals the true probability of correctness.

This distinction is already explicit in Section 3.2, which states that consistency alignment is not equivalent to calibration and gives a counterexample. The connection to correctness in the method comes through labeled validation data: the perturbation family and scale are chosen against observed calibration error, then assessed on held-out test data.

We additionally tested sensitivity to this selection using 20 bootstrap resamples of the validation set, refitting each scalar map and evaluating the unchanged test set:

MethodTest ECE (%), mean ± SD across validation resamples
TS1.3864 ± 0.1844
Deterministic Uniform CC1.0061 ± 0.0923

This provides empirical evidence beyond the toy setting, conditional on the fixed checkpoint and split. The theoretical characterization and the empirical calibration evidence serve distinct, complementary purposes.

References

[1] Berthet et al. Learning with Differentiable Perturbed Optimizers. NeurIPS, 2020.

[2] Papandreou and Yuille. Perturb-and-MAP Random Fields. ICCV, 2011.

[3] Cohen, Rosenfeld, and Kolter. Certified Adversarial Robustness via Randomized Smoothing. ICML, 2019.

[4] Gruber and Buettner. Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition. AISTATS, 2023.

Reviewer_HyTROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies post-hoc calibration from the perspective of prediction stability. The proposed method, Consistency Calibration (CC), perturbs a sample at the input, feature, or logit level and uses the frequency with which each class wins the argmax across perturbations as the calibrated confidence. The logit-level variant, used in all main experiments, adds i.i.d. noise (Gaussian, uniform, or Gumbel) to the logits, with the noise type and strength selected by minimizing ECE on a validation set. On the theoretical side, the authors show that with Gumbel noise the expected consistency vector equals a temperature-scaled softmax, and they derive bounds relating the Bregman Information of the perturbed logits to the Rényi entropy of the consistency vector, with a gap of at most ln K. Experiments on CIFAR-10/100, ImageNet, and ImageNet-LT with CNN and transformer backbones report lower ECE than TS-based and grouping-based post-hoc methods in most settings, as well as further improvements when CC is applied on top of training-time calibration methods. The authors also acknowledge that CC substantially increases NLL on ImageNet.

Critical strengths
  1. Simple and easy to use. The logit-level variant needs only the logits, has two hyperparameters (noise type and strength), requires no retraining or extra forward passes, and can be applied on top of any trained model, including models trained with calibration-aware losses.
Critical weaknesses

W1. The logit-level method coincides with an existing construction, which limits novelty. Logit-level CC (Eq. 7–8) is exactly the perturbed maximizer of Berthet et al. (2020), Learning with Differentiable Perturbed Optimizers (NeurIPS 2020): the expectation of the one-hot argmax under additive noise, y*_ε(θ) = E[argmax_y ⟨y, θ + εZ⟩]. When the feasible set is the set of one-hot vectors, this is precisely the vector of win probabilities c_k in Eq. 8. The correspondence extends to the details:

  • Berthet et al. also estimate this quantity by Monte Carlo, averaging the argmax over M noise draws (their Definition 3.1), which is Eq. 7 of this paper.
  • They state explicitly that with Gumbel noise the perturbed maximizer is the softmax and the perturbed maximum is log-sum-exp. These are the two identities (Eq. 10 and Eq. 14) on which Theorem 1 rests.
  • Their Proposition 2.2 shows y*_ε(θ) = y*_1(θ/ε) for any admissible noise, i.e., the noise amplitude acts exactly as a temperature. This makes logit-level CC a one-parameter calibration map in the same family as TS, differing only in the link function induced by the noise distribution.

The paper does not cite this work. The remaining contribution is applying this known operator as a post-hoc confidence estimate, which should be stated as such, and the theoretical section should be positioned relative to these existing results.

W2. The central intuition is not an insight at the logit level. That a prediction which stays unchanged under logit perturbations is more likely to be confident is close to a tautology: under i.i.d. logit noise, the probability that the top class keeps winning is a monotone function of the gaps between the top logit and the others, which is the same information softmax confidence already encodes. Appendix F confirms this: the explanation offered for why CC works is that misclassified overconfident samples have smaller top-1/top-2 margins. This is a property of the logits that any margin-based or temperature-based calibrator also exploits. The "consistency" framing would carry new information only for perturbations applied before a nonlinear part of the network (data level or deep feature level), but these are not the variants used in the main experiments.

W3. The comparison with TS is suspicious. Under Gumbel noise, CC equals TS in expectation, and both are tuned by validation ECE (Appendix B). Yet Table 4 reports ECE 0.87 for Gumbel-CC and 2.09 for TS on ResNet-50 / ImageNet. A gap of this size cannot come from the method itself, since the two differ only by Monte Carlo error (T = 1000). It suggests that the TS temperature search differs from the CC search (grid, range, or objective), which would also affect every TS-vs-CC comparison in Tables 1, 6 and 7. In addition, if Gumbel gives 0.87, it is unclear why Table 1 reports 1.53 for this setting when the noise type is supposedly selected on validation data. More generally, the noise type, strength, and search grid used for each entry of the main tables are not reported, which makes the results difficult to verify.

Questions for authors

1. Can the authors directly compare Gumbel-CC with TS? Since Gumbel noise with scale τ yields softmax(z/τ) in expectation, Gumbel-CC and TS with temperature τ should give the same ECE up to Monte Carlo error. Concretely, I suggest:

  • sweeping τ over the same grid for both methods and plotting validation and test ECE of TS and Gumbel-CC against τ on the same axes;
  • reporting the τ selected by each method under an identical search procedure, together with the resulting test ECE;
  • reporting Gumbel-CC for several values of T (e.g., 16, 100, 1000, 10000), which should converge to the TS curve at rate O(1/√T).

If the two curves coincide, the gap in Table 4 must come from the tuning protocol, and the TS baseline in the main tables should be re-tuned accordingly. If they do not coincide, the authors should explain where the difference comes from, since it would contradict Eq. 10.

6 comments

Thank you for the detailed review, particularly the connection to perturbed optimizers and the request for a matched Gumbel–TS comparison. We examined the cited work and conducted additional experiments addressing both points. The results distinguish the known Gumbel special case from the practical value of other perturbation-induced calibration maps.

Unless stated otherwise, the new experiments use one CIFAR-10/ResNet-50 checkpoint, 5,000 validation examples, 10,000 test examples, and 15 equal-width ECE bins. ECE is reported in percent, and hyperparameters are selected using validation data only.

W1. Relationship to perturbed optimizers and the contribution

The correspondence with Berthet et al. [1] is precise: on the one-hot simplex, their perturbed maximizer gives the same expected win-probability vector as logit-level CC. Their Monte Carlo estimator and scale relation also apply. We explicitly credit this construction and its Gumbel–softmax identity.

The paper's contribution should therefore be assessed at the level of calibration methodology and evidence:

  1. Calibration through perturbation-induced links: selecting the perturbation family and scale using labeled validation data and studying their usefulness as confidence maps.
  2. A structural analysis: relating the consistency vector's Rényi entropy to the Bregman Information of perturbed logits, with an explicit residual in the Gumbel case.
  3. A systematic empirical investigation: studying input-, feature-, and logit-level variants, different architectures and datasets, and compatibility with calibration-aware training.

The scale identity makes each fixed noise family a one-parameter map. It does not make every such map a softmax map. Our new deterministic experiment directly tests whether this distinction has practical calibration value.

W2. What does logit-level consistency add beyond margins?

Logit-level CC uses information already present in the logits. Its benefit is the mapping from that information to confidence: different noise distributions induce different functions of the competing logit gaps. A top-two-gap calibrator additionally compresses those gaps to a single statistic.

We tested a monotone scalar margin baseline,

q=sigmoid⁡((z(1)−z(2))/t)q=\operatorname{sigmoid}((z_{(1)}-z_{(2)})/t),

selecting tt by validation ECE. We also computed Uniform-CC probabilities by deterministic integration at the validation-selected width, removing Monte Carlo sampling from evaluation.

Confidence mapTest ECE (%)
TS, fitted by validation NLL1.3730
TS, fitted by validation ECE1.4369
Scalar top-two-margin calibration1.5285
Uniform CC, deterministic integration0.9101
Uniform CC, T=1,000T=1,0000.8854 ± 0.0408

The last row reports mean ± sample SD over three test-time noise seeds with fixed fitted parameters. Uniform width is 16.1, equivalently centered noise on [−8.05,8.05][-8.05,8.05].

The deterministic result is particularly informative: the improvement persists without sampling discretization. This supports the value of the induced calibration map without requiring additional information beyond the logits.

W3. The ImageNet comparison and noise-family selection

For i.i.d. Gumbel noise with a common scale, the population win vector is exactly TS. The ImageNet values 0.87 and 2.09 therefore should not be interpreted as an intrinsic advantage of the Gumbel population map.

The recovered TS component uses NLL fitting, whereas Appendix B describes ECE selection. This identifies a protocol distinction, but does not by itself establish the cause of the full ImageNet gap. Our controlled experiment below uses explicitly identical ECE fitting; it does not by itself reconcile the historical ImageNet comparison.

Regarding 0.87 versus the main-table value 1.53, selecting a noise family must depend on validation ECE. A lower test ECE alone does not imply that the corresponding family should have been selected. Reconstructing that historical selection requires the corresponding validation records. The new comparison provides an explicit search protocol and selected parameters.

Q1. Matched temperature sweeps and convergence with TT

We used the same logits, validation set, ECE implementation, and grid τ∈{0.10,0.11,…,10.00}\tau\in\{0.10,0.11,\ldots,10.00\} for TS and Gumbel-CC. Gumbel noise is i.i.d. across classes with one common scale.

We ran three independent validation/test seed pairs, using common standard-noise draws across candidate temperatures and nested draws across TT. Test draws are independent of validation draws.

MethodSelected τ\tau, three runsMean validation ECE (%)Test ECE (%), mean ± SD
Analytic TS2.481.27331.4369
Gumbel CC, T=16T=162.75 / 2.67 / 2.600.92081.1383 ± 0.2456
Gumbel CC, T=100T=1002.50 / 2.50 / 2.501.04001.3867 ± 0.0062
Gumbel CC, T=1,000T=1,0002.49 / 2.50 / 2.491.17741.3950 ± 0.0339
Gumbel CC, T=10,000T=10,0002.48 / 2.49 / 2.481.27611.4134 ± 0.0002

At fixed τ=2.48\tau=2.48, mean absolute probability error relative to analytic TS decreases as follows:

TTMean absolute probability error
1000.004279
1,0000.001458
10,0000.000461

The fitted log–log slope is −0.484, consistent with T−1/2T^{-1/2} Monte Carlo convergence. Binned ECE itself need not improve monotonically with TT.

These results support Eq. 10. The evidence for a useful alternative calibration map comes from the non-Gumbel comparison in W2, including its deterministic evaluation.

References

[1] Berthet et al. Learning with Differentiable Perturbed Optimizers. NeurIPS, 2020. Definitions 2.1 and 3.1; Proposition 2.2.

[2] Guo et al. On Calibration of Modern Neural Networks. ICML, 2017.

Thank you for the response and the additional experiments. Two concerns remain.

1. Under Gumbel noise, the advantage over TS comes from unconverged sampling. The new experiment shows that Gumbel-CC converges to TS as T increases, and any gain over TS appears only at small T. Since both methods optimize the same one-parameter family on the same validation set, this gain reflects fitting Monte Carlo noise rather than better calibration. The Gumbel results in Table 4 therefore cannot support an advantage of CC. At T = 1,000, the sampling effect is also far too small to explain the gaps in Table 4, which points to the TS baseline, fitted by NLL rather than by ECE as stated in Appendix B.

2. The final settings of the reported results are not provided. The response gives settings only for the new CIFAR-10 experiment. For the main tables, the noise type, noise strength, T, TS fitting objective, and whether each value is a single run or a mean remain unspecified, so the main results cannot be verified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.