acceptodds
Under review as a conference paper at ICLR 2027

Neighborhood Smoothing for Calibration

Abstract

Modern neural networks are often miscalibrated, with a tendency to overconfidence. Existing train-time calibration methods largely modify task losses or calibration penalties, leaving neighborhood structure in learned representations underexploited. We introduce graph smoothing as a general principle for train-time calibration, which encourages similar predictive distributions across neighboring samples in representation space. We analyze the effects of graph smoothing, deriving bounds that connect predictive divergence between neighboring samples to local confidence variation and to the propagation of pointwise calibration error, and characterize the conditions under which smoothing can or cannot improve calibration. In light of this analysis, we propose NeSCal, a graph-based train-time regularizer that penalizes the Jensen–Shannon divergence between predictive distributions of neighboring samples. We present a thorough empirical analysis, showing that across standard calibration benchmarks, NeSCal improves predictive quality, and the improvement is complementary to post-hoc calibration: after temperature scaling, NeSCal attains the lowest NLL of all evaluated train-time methods in seven of the eight image and tabular settings. These findings demonstrate the value of graph smoothing over learned representations for neural network calibration.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
confidence calibration, uncertainty quantification, train-time calibration, graph-based regularization, manifold regularization, train-time regularization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AspLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author of the paper define graph smoothing as a general principle for train-time calibration, as it encourages similar predictive distributions across neighboring samples in representation space. They finally proposes NeSCal, a graph-based train-time regularizer that penalizes a certain divergence between predictive distributions of neighboring samples in the embedding space. Empirically after temperature scaling, NeSCal attains the lowest NLL of all evaluated methods in both image and tabular settings.

Critical strengths
  • The method is quite simple with no additional inference cost.
  • In some cases, the empirical improvements in NLL persist even after temperature scaling.
  • The theory identifies conditions under which neighborhood smoothing helps or hurts calibration.
Critical weaknesses
  • The methodological novelty is limited and it's relative to neighbor consistency regularization, with no direct NCR comparison (even if it is cited).
  • The theoretical analysis of idealized averaging does not establish guarantees for the actual training procedure.
  • Since the paper is mostly focused on confidence calibration, alongside NLL in the main results there should be a calibration metric, as NLL does not distinguish calibration gains from improvements in predictive power. Although ECE is reported in the appendix, its improvements are not consistent across settings, making the evidence for better calibration less clear than that for improved NLL.
  • Some improvements are small relative to seed variability, and results on text are less competitive.
Questions for authors
  • Can you compare your method also with NCR which seems the most similar method with respect to yours? Both before and after TS
  • Since NLL does not isolate calibration gains from improvements in predictive power, could you clarify how the ECE and other calibration results in the appendix support the paper’s main claims, given their mixed improvements across settings?
  • Are the small gains over the strongest baselines statistically robust across seeds?
6 comments

We thank you for your review. Your Questions 1–2 and Weaknesses 1–3 are shared with the other reviewers and answered in the general response above: NCR before and after TS in Section 1, calibration beyond accuracy in Section 2, and the analysis of the procedure as implemented in Section 3. As you suggest in W3, the main results table will now report ECE and post-TS ECE alongside NLL. For Q2, the appendix result that speaks most directly to your question is the calibration–refinement decomposition: NeSCal's refinement term, the component that recalibration cannot improve, is lower than CE's in ten of eleven settings (Table 10).

Q3 / W4. Are the gains statistically robust across seeds? Yes. In the image settings no baseline is ever significantly better than NeSCal after TS, and across the eight image and tabular settings NeSCal has the lower post-TS NLL against each baseline in at least seven. We test this at two levels.

Within each setting. In each of the five end-to-end image settings, we paired NeSCal with every train-time baseline over the four seeds and computed a 95% t interval on the difference: 36 setting–baseline comparisons (the paper's six baselines and NCR in all five settings, plus Mixup on CIFAR-10/R50). After TS, NeSCal is never significantly worse: its NLL is significantly better in 21 comparisons and its ECE in 14, and all others are ties. Before TS, its NLL is significantly better in 23 and worse in only two, both on CIFAR-10/WRN, and neither survives TS. Against the runner-up after TS, the margin is significant on CIFAR-100/R50 (FL, −0.051 [−0.086, −0.016]) and within seed variation in the other four settings, which is why we also test across settings.

Across settings (Table 9 of the paper). NeSCal's post-TS NLL is lower than that of FL, LS, MMCE and MbLS in all eight image and tabular settings, and lower than that of MDCA and s-AvUC in seven; exact Wilcoxon signed-rank tests on the per-setting means give p = 0.008–0.023 (reported as exploratory in Appendix E.2).

W4. Text results. The smaller gains on text are a property of these benchmarks, not of the modality, and Proposition 3 explains them: smoothing can only remove error that differs between neighbors, so where neighboring predictions already agree there is little for it to remove. The four settings with the least disagreement at activation are two text and two tabular ones (Amazon binary and 5-class, Covertype, Jannis; mean edge JS 0.003–0.013), and all four gain 0.4–0.8% in NLL. The image settings, where disagreement is larger (0.026–0.254), gain 3.9–11.7% (Table 2). 20NG, whose disagreement matches CIFAR-10's, gains less than its disagreement would suggest (1.3%); Section 6.4 attributes this to a model that learns over frozen word embeddings and so has little room to adapt its features. NeSCal still lowers NLL relative to CE in all three text settings.

We believe these results resolve each of your concerns: NCR is now compared directly, before and after TS, in seven settings; calibration will be reported in the main results and separated from accuracy; and the gains are robust within and across settings. We would be grateful if you would consider raising your score.

Thank you for the detailed response and the additional experiments. The NCR comparison addresses one of my concerns. However, the calibration claim remains weakly supported:

  • Base-model calibration. NeSCal’s base-model ECE (Table 7) is best only on Tiny-ImageNet, ranks 3rd–6th of eight methods elsewhere, and is higher than plain CE on Jannis, Covertype and both Amazon tasks.
  • Contribution beyond TS. Compared with CE+TS, NeSCal+TS leaves post-TS ECE essentially unchanged in three of five image settings (e.g., 1.63 vs. 1.61 on CIFAR-100/R50). The remaining NLL gain mainly reflects better refinement and accuracy rather than calibration. The framing as a train-time calibration method (title, abstract, contribution (i)) is therefore not supported by calibration-specific metrics. The evidence mainly shows improved predictive quality, with calibration provided by post-hoc recalibration, consistent with the revised positioning in the rebuttal.
  • Unexplained losses. Specific baselines outperform NeSCal in several settings, e.g., FL on base NLL for CIFAR-10/WRN, MbLS/LS on 20NG, and MMCE on Amazon (binary). Neither the paper nor the rebuttal explains why. Among the five end-to-end image settings, the margin over the runner-up after TS is significant only on CIFAR-100/R50.

Finally, the currently available PDF does not yet include the promised revisions.

Thank you for the follow-up, and for confirming that the NCR comparison resolves that concern. We take your remaining points in turn.

Base-model calibration. We agree that base-model ECE is not where NeSCal leads on most benchmarks, as Section 6.1 states. Three points qualify it.

  • NeSCal's base-model ECE is below plain CE's in seven of the eleven settings, and it is the best of all methods on the two largest datasets: Tiny-ImageNet (Table 7, now part of Table 1) and ImageNet-1K, where it is the best of nine methods (0.95%, against 3.92% for CE; Table 28).
  • The four settings where it is higher than plain CE (Jannis, Covertype and both Amazon tasks) are exactly the four with the least disagreement between neighbors at activation (mean edge JS ≤ 0.013, Table 2). Smoothing can only remove error that differs between neighbors (Proposition 3), so there it has little to act on. Section 6.4 now states this.
  • All methods are selected by validation NLL, not ECE. Selecting NeSCal by ECE instead lowers its ECE by 0.85 points on average, at a cost of 0.017 nats of NLL (Appendix E.9).

Contribution beyond TS. Aggregate post-TS ECE compares models at different accuracies, and TS compresses it into a narrow band for every method (a spread of 0.5–1.1 points across methods on CIFAR). Two calibration-specific comparisons that hold accuracy fixed separate NeSCal+TS from CE+TS:

  • On the test points where NeSCal and CE predict the same label, so correctness is identical sample by sample, NeSCal's post-TS ECE is significantly lower in five of seven settings, including CIFAR-100/R50 (−1.79 [−2.56, −1.02] points), and its NLL is significantly lower in six of seven (Section 6.2, Table 37). On these points, the NLL gain cannot come from accuracy.
  • TS does not change predicted labels, so the NLL it removes measures confidence-scale miscalibration at each model's own accuracy. TS removes 28–90% less NLL from NeSCal than from CE across the five image settings (0.008 vs. 0.029 nats on CIFAR-100/R50).

Part of the remaining gain is refinement, which a single temperature cannot supply: NeSCal's refinement term is lower than CE's in ten of eleven settings (Table 9). This is the complementarity the revised framing claims. We agree with the positioning you describe: NeSCal improves predictive quality and is designed to be composed with post-hoc recalibration, not to replace it. The revised abstract, introduction (including the contributions) and conclusion now say this directly. We kept the title because calibration remains the goal; the revision states how NeSCal contributes to it.

Unexplained losses.

  • FL, base NLL on CIFAR-10/WRN: a confidence-scale effect. TS removes 0.0004 nats from FL and 0.019 from NeSCal, after which the two are tied (0.1568 vs. 0.1566, Table 1).
  • MbLS/LS on 20NG: an accuracy effect. Both are about 6 points more accurate than CE there (83.2% and 82.7% vs. 76.9%, Table 6). NeSCal's gain on 20NG is limited by the model, a convolutional network over frozen word embeddings with little room to adapt its features (Section 6.4).
  • MMCE on Amazon (binary): one of the four low-disagreement settings above. It has the highest edge agreement of all settings (0.987, Table 2), so there is little for smoothing to act on.
  • Margin over the runner-up after TS: four seeds give limited power within a single setting, so we also test across settings. NeSCal's post-TS NLL is lower than that of each baseline in at least seven of the eight image and tabular settings (Table 8, p ≤ 0.023). In 37 paired comparisons across the five end-to-end image settings, it is never significantly worse after TS (Appendix F.9). On ImageNet-1K, it is significantly better than every baseline, including the runner-up, on NLL before and after TS and on base-model ECE (Table 29).

Revised PDF. The revised PDF is now uploaded, with changes in blue. ECE before and after TS is in Table 1, the label-fixed comparison in Section 6.2, the low-disagreement settings in Section 6.4, and NCR in Section 6.5.

We hope the revision addresses your remaining concerns, and would be glad to clarify anything further before the discussion period closes.

Reviewer_MJe8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces NeSCal, a method that constructs a kkNN graph based on model representations during training and employs the Jensen–Shannon divergence to constrain the full predictive distributions of neighboring samples. The authors establish a relationship between graph smoothing and the propagation of point-wise calibration errors, and they analyze the conditions under which idealized averaging improves or impairs calibration. Experiments cover image, text, and tabular classification tasks; NeSCal improves the negative log-likelihood (NLL) in all 11 primary experimental settings, and when combined with Temperature Scaling, it achieves the lowest NLL in 7 out of 8 image/tabular classification scenarios.

Critical strengths

The method is simple and incurs no additional overhead during inference. It yields consistent improvements in NLL, Brier score, accuracy, and failure prediction across image-based tasks. The experimental design is rigorous; proper scoring rules are employed for model selection, and the observed gains persist even after temperature scaling (TS)—indicating that the improvements stem from more than just a global adjustment of confidence scales.

Critical weaknesses

The method is highly similar to classical manifold regularization and NCR; it lacks a primary experimental comparison against the most direct baseline—neighborhood consistency training—which limits its novelty. The theoretical analysis focuses on idealized training graphs and two-point averaging, which deviates from the practical implementation of NeSCal (characterized by fixed graphs, cached targets, and one-sided gradients). Furthermore, the approach lacks validation at the ImageNet scale, and the scalability of exact kkNN and O(nK)O(nK) caching remains unclear.

Questions for authors

Under the same training protocol, how does NeSCal compare to NCR, pairwise KL, symmetric KL, and logit pairing? If the improvement in NLL is primarily driven by an increase in accuracy, does NeSCal still improve calibration when accuracy is matched or when predicted labels are held constant? Could you evaluate the method on ImageNet-1K—or at least on one modern pre-trained backbone—and report the overhead associated with ANN graph construction and distributed caching?

4 comments

We thank you for your precise and constructive review. Your points answered in the general response above: calibration with predicted labels held fixed, as you suggested (Section 2); the fixed graph, cached targets and one-sided gradients (Section 3); and ImageNet-1K, a modern backbone, ANN graph construction and caching (Section 4). NCR is covered in Section 1. Measured all-gather timings for distributed caching will follow in our next post.

Q1. Pairwise KL, symmetric KL and logit pairing. We replaced JS with each on NeSCal's graph and protocol (same onset, schedule and seeds). Because their natural scales differ by orders of magnitude, each was tuned over five penalty magnitudes. JS gives the best post-TS NLL of the four (Table R7):

  • Pairwise KL, NCR's divergence, is significantly worse before TS (+0.014 [+0.001, +0.027]).
  • Symmetric KL buys a lower base ECE at the price of a significantly higher post-TS ECE (+0.56 points [+0.28, +0.85]).
  • Centered logit pairing, the classical graph-Laplacian regularizer is significantly worse both before and after TS (+0.033 and +0.037 nats).

This supports JS as a principled choice that is sufficient, though, as the paper states, not uniquely optimal.

Table R7. Divergences on NeSCal's graph, CIFAR-100/ResNet-50, mean ± SD over four seeds.

DivergenceNLL ↓NLL+TS ↓ECE % ↓ECE+TS % ↓ΔNLL+TS vs JS
Pairwise KL (NCR's)0.8166 ± 0.01980.8036 ± 0.02004.24 ± 0.121.53 ± 0.37+0.0092 [−0.0022, +0.0207]
Symmetric KL0.7988 ± 0.01920.7982 ± 0.01942.79 ± 0.352.19 ± 0.36+0.0038 [−0.0019, +0.0095]
Centered logit pairing0.8358 ± 0.02590.8309 ± 0.02573.49 ± 0.521.95 ± 0.43+0.0365 [+0.0217, +0.0513]
JS (NeSCal)0.8026 ± 0.01550.7944 ± 0.01913.95 ± 0.671.62 ± 0.24—

W1. Relation to manifold regularization. NeSCal does descend from manifold regularization (Belkin et al., 2006), as the paper acknowledges, but it differs in what it assumes, what it regularizes and what it proves. Manifold regularization is motivated and analyzed under the manifold assumption: the data lie near a low-dimensional manifold along which the target varies smoothly. NeSCal assumes neither. Our propagation bounds hold on every edge of any graph, under only a condition on the divergence; where smoothness fails, the failure appears explicitly as the correctness gap ∣ai−aj∣|a_i-a_j|, which Propositions 1–3 use to state when smoothing helps calibration and when it cannot. NeSCal also needs no unlabeled data, builds its graph over learned representations at a selected point in training, and penalizes a bounded divergence between full predictive distributions rather than squared differences of function values. Under our protocol, the manifold regularizer itself (centered logit pairing, Table R7) is significantly worse than NeSCal before and after TS.

We hope these results answer your questions. If they do, we would be grateful if you would consider raising your score.

References. Belkin, M., Niyogi, P., & Sindhwani, V. (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. JMLR, 7, 2399–2434.

Follow-up on Q3: distributed caching and a fine-tuned modern backbone

Distributed caching. We measured the all-gather that keeps the prediction cache consistent across data-parallel workers, at ImageNet-1K size (Table R9). Synchronizing at every step costs 0.37 ms per step, or 0.22 s per epoch. Synchronizing the full table once per epoch costs 0.23 s. Both move the same data, so either way the cost is about 0.2 s per pass over 1.23M images, small next to the 8.1 s the penalty itself adds (Table R5). Once-per-epoch synchronization corresponds to the "targets published once per epoch" arm of Table R6, which changes no metric.

Table R9. Cache synchronization, ImageNet-1K size (n = 1.23M, K = 1000, fp32), two GPUs, NCCL.

SynchronizationPer stepPer epoch
Every step (4.1 MB per worker)0.37 ms0.22 s
Once per epoch (full 4.92 GB table)—0.23 s

Fine-tuned backbone. In addition to the frozen DINOv2 and ViT-B/16 heads, we fine-tuned an ImageNet-pretrained ViT-B/16 end to end on CIFAR-100 at 224 px (Table R8). NeSCal is the most accurate model and has the lowest post-TS NLL and ECE. After TS, its NLL is 0.021 [0.017, 0.026] below CE's and 0.037 [0.021, 0.053] below NCR's. Its accuracy is 0.9 points [0.1, 1.8] above CE's.

Table R8. ViT-B/16 fine-tuned on CIFAR-100, 224 px, 30 epochs. Mean ± SD over four seeds.

MethodNLL ↓NLL+TS ↓ECE % ↓ECE+TS % ↓Acc % ↑
CE0.4489 ± 0.00180.4420 ± 0.00213.51 ± 0.702.07 ± 0.2986.86 ± 0.34
NCR0.4736 ± 0.00670.4576 ± 0.00632.84 ± 0.172.75 ± 0.1487.45 ± 0.16
NeSCal0.4438 ± 0.00780.4207 ± 0.00364.10 ± 0.241.43 ± 0.5487.80 ± 0.14

Together with the general response, this answers each part of Q3. If it resolves your remaining concerns, we would be grateful if you would consider raising your score.

Thank you again for your review. The revised PDF is now up, with changes in blue. It includes everything from our replies to you: the comparison of divergences and the relation to manifold regularization (Appendix F.3 and Related Work), the comparison with predicted labels held fixed that you suggested (Section 6.2 and Appendix F.10), and ImageNet-1K, approximate graphs, a fine-tuned ViT-B/16 and distributed caching (Appendices F.4–F.7). We would be grateful if you could take a look before the discussion period closes, and we are happy to answer any remaining questions.

Thank you again for your review. As the discussion period closes tomorrow, we wanted to check whether our responses and the revised PDF address your concerns. If anything remains unclear, we would be glad to answer it before the deadline.

Reviewer_uUh2OpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes graph smoothing as a principle for train-time calibration and instantiates it as NeSCal. NeSCal adds to the cross-entropy loss a Jensen–Shannon divergence penalty between the predictive distributions of neighboring samples in a fixed kNN graph built over learned representations, with no inference-time cost. The authors derive bounds showing that pointwise calibration error propagates along graph edges, governed by predictive divergence and by the gap in true correctness probability between neighbors. They also characterize when idealized averaging helps: smoothing can reduce error that differs between neighbors but not error they share, which motivates its complementarity with temperature scaling. Experiments span eleven image, text, and tabular settings against six train-time calibration baselines. NeSCal improves NLL over cross-entropy in all settings and, after temperature scaling, attains the lowest NLL in seven of the eight image and tabular settings, although its base-model ECE is not competitive and its gains on text and tabular data are small.

Critical strengths
  1. Clear motivation. Existing train-time calibration methods mostly modify the loss based on model outputs and labels, leaving the geometry of learned representations largely unexploited. Using neighborhood structure in representation space to regularize predictive behavior is a reasonable and relatively underexplored angle. The analysis in Proposition 3 gives a useful explanation of why neighborhood smoothing and global temperature scaling are complementary: smoothing can only reduce the component of error that differs between neighbors, not the shared component.

  2. Honest and transparent experimental reporting. The authors openly report settings where the method does not help or is harmful. These include the text benchmarks, where NeSCal does not attain the lowest NLL; label noise, where ECE improves but NLL and accuracy degrade in three of four conditions; and the synthetic geometric scenario, where NeSCal is worse than CE. Section 6.4 also analyzes when the method is expected to help. This level of transparency is appreciated.

Critical weaknesses
  1. Missing experimental comparison with NCR (Iscen et al., 2022). NCR is the most closely related prior method: it also penalizes disagreement between a sample's prediction and its neighbors' predictions in representation space. The paper distinguishes the two methods only in the related work section, citing per-edge vs. mixture-target penalties, a fixed full-dataset kNN graph vs. minibatch neighbors, and calibration vs. label-noise robustness as the target. These differences are design choices whose practical impact should be demonstrated empirically. Discussion alone is not sufficient to establish the contribution over NCR. I would expect a direct comparison on at least the main CIFAR settings, evaluated both before and after temperature scaling.

  2. Lack of discussion on scalability. The exact kNN graph construction is O(n²l), and the method additionally caches an O(nK) prediction table. Even at Tiny-ImageNet scale, peak GPU memory rises from 3.89 GB to 6.89 GB (Table 18). The largest dataset evaluated has about 100k training samples, and there are no experiments or discussion at ImageNet scale or beyond, nor on large-K settings. I am concerned the method may be impractical on larger datasets. Several design choices raise further concerns at scale:

    • The graph is frozen at activation while representations keep evolving.
    • Cached neighbor predictions are refreshed only once per epoch.

    The authors should discuss, or ideally evaluate, approximate nearest-neighbor search, graph rebuilding strategies, and the resulting cost–performance trade-offs.

  3. The positioning should be generalization / predictive smoothing rather than calibration. The current evidence suggests that NeSCal's "calibration" benefit is realized mainly through temperature scaling rather than by the method itself:

    • Base-model ECE is not competitive. For example, NeSCal reaches 2.29% on CIFAR-10/R50, compared with 0.83% for MMCE and 1.14% for FL. The paper itself acknowledges that base-model ECE is where NeSCal is least competitive.
    • NLL gains track accuracy gains. NLL improvements largely coincide with accuracy improvements, for example 75.29% → 77.19% on CIFAR-100/R50. Accuracy improves in nine of eleven settings, and failure-prediction AUROC/AURC also improve.
    • Post-TS NLL mostly reflects refinement. The headline result, lowest NLL after temperature scaling, primarily measures refinement and discrimination once the confidence scale has been corrected globally.

    Taken together, NeSCal behaves more like a regularizer that improves generalization and the quality of predictive distributions, with calibration obtained by composing it with TS. I suggest reframing the title and main claims accordingly. Alternatively, the authors could provide evidence that calibration improves beyond what is explained by accuracy, for example through accuracy-matched comparisons.

Questions for authors
  1. Could the authors add comparisons with data-augmentation-based methods such as Mixup? Mixup (Zhang et al., 2018; Thulasidasan et al., 2019) and its calibration-oriented variants (e.g., RankMixup) are well known to improve both calibration and generalization. They are therefore natural baselines, especially if the method is positioned around generalization and predictive smoothing. It would also be informative to report whether NeSCal is complementary to Mixup, i.e., whether combining the two yields further gains.
5 comments

We thank you for the detailed review, and for recognizing Proposition 3's account of why neighborhood smoothing and temperature scaling are complementary. Your three weaknesses are answered in the general response above:

  • W1 (NCR): Section 1. After TS, NeSCal has lower mean NLL and ECE than NCR in all seven settings, with NLL significantly lower in five, and neither metric is ever significantly higher. As you asked, the 2×2 factorial measures the practical impact of each design difference (Table R2): at NeSCal's λ, replacing the fixed, full-dataset graph with minibatch neighbors costs 0.12–0.13 nats of post-TS NLL, while the choice of target stays within seed variation.
  • W2 (scalability): Section 4. On ImageNet-1K (1.23M training images, K = 1000), the one-off exact graph takes 159 s on one GPU and NeSCal gives its clearest gains; the section also covers approximate search, graph rebuilding, cache staleness, distributed caching, and a correction to Table 18.
  • W3 (positioning): We take your second option: evidence that calibration improves beyond what accuracy explains. Section 2 of the general response answers your four points. Base-model ECE: agreed, and the revision will state the claim precisely. NLL tracking accuracy: with predicted labels held fixed, the strictest form of the accuracy-matched comparison you suggest, NeSCal's ECE and NLL are still lower, and a rebuilt-graph variant that keeps NeSCal's accuracy loses its calibration gain (points 1, 3 and 4). Calibration realized through TS: in the five end-to-end settings TS has 28–90% less to remove from NeSCal than from CE, and on ImageNet-1K nothing at all (point 2). Refinement: agreed; it is what a single temperature cannot supply (point 5).

Q1. Mixup and RankMixup. Mixup is a data-augmentation protocol rather than a calibration objective, which is why the paper discusses it in related work rather than as a baseline. It is orthogonal to NeSCal, and the combination you propose is the right test of complementarity. We have added both methods under our protocol. Post-TS metrics matter here, since Wang et al. (2023) find that Mixup "usually makes models less calibratable than vanilla empirical risk minimization."

  • RankMixup (Noh et al., 2023), CIFAR-100/R50: NeSCal is clearly better. Its NLL is lower by 0.125 [0.066, 0.183] before TS and 0.099 [0.049, 0.148] after, its base ECE by 2.5 points [1.3, 3.7], and its accuracy is higher (77.2% vs. 74.4%).
  • Mixup (Zhang et al., 2018), CIFAR-10/R50: Mixup is the more accurate model (95.1% vs. 94.3%) and has a lower base NLL, though not significantly (+0.010 [−0.003, +0.023]). After TS, NeSCal matches its NLL (−0.005 [−0.015, +0.006]) and has lower ECE (−0.69 points [−1.08, −0.31]); TS leaves the Mixup model unchanged (T = 1.00), consistent with Wang et al.

We will post Mixup on CIFAR-100 and NeSCal combined with Mixup in the coming days. With W1–W3 addressed here, we would be grateful if you would consider updating your score now, and we will gladly answer further questions once the remaining Mixup results are posted.

References. Noh, J., Park, H., Lee, J., & Ham, B. (2023). RankMixup: Ranking-based mixup training for network calibration. ICCV. Wang, D.-B., Li, L., Zhao, P., Heng, P.-A., & Zhang, M.-L. (2023). On the pitfall of mixup for uncertainty calibration. CVPR. Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. ICLR.

Thank you for the detailed response and the additional experiments. I have two follow-up questions.

1. Why does the method still rely on TS? The response says TS has 28–90% less to remove from NeSCal than from CE, and nothing on ImageNet-1K. But base-model ECE is still not competitive in most settings: on CIFAR-10/R50, for example, NeSCal is at 2.29% vs. 0.83% for MMCE and 1.14% for FL. So I'm still not sure what to make of the calibration claim. If the regularizer improves calibration on its own, why does it consistently need TS to become competitive on ECE? Is there something in the method (e.g., keeping CE as the only supervised term, or decaying λ toward zero by the end of training) that leaves the shared overconfidence in place by design? If so, I think the paper should say this directly and frame NeSCal as something meant to be used together with TS, rather than as a standalone calibration method. It would also help to see how NeSCal's base ECE on ImageNet-1K compares with the other train-time baselines, not just CE.

2. Peak GPU memory on ImageNet-1K. The response reports the graph construction time (159 s) but not the memory cost. What is the peak GPU memory on ImageNet-1K?

Thank you for the follow-up questions.

1. TS and the calibration claim. You are right that NeSCal leaves the shared overconfidence to TS, and this is by design. Proposition 3 shows that smoothing can only remove the part of the error that differs between neighbors. The part they share, the systematic overconfidence deep networks are known for, is exactly what a single temperature corrects (Section 3.3). Of the two choices you name, keeping CE as the only supervised term is deliberate: it keeps the objective strictly proper (Section 4.2). FL and LS lower confidence globally, a correction TS also provides, and after TS both fall behind NeSCal in all eight image and tabular settings (Table 1). Decaying λ is not the cause: with a constant λ, base ECE is higher in three of the four CIFAR settings (Table 15).

How much of the error neighbors share depends on the data. On CIFAR, as you note, the strongest calibration objectives have lower base ECE. On the two largest datasets, NeSCal has the lowest base ECE of every method we ran: on Tiny-ImageNet (Table 7) and on ImageNet-1K, where we have now run all six train-time baselines (Table R10). On ImageNet-1K, NeSCal ranks first on all five metrics, and its NLL before and after TS and its base ECE are significantly lower than those of every baseline. On the smaller CIFAR-100 head (frozen ViT-B/16), NeSCal again has the lowest NLL before and after TS and the lowest post-TS ECE, and it is not significantly worse than any baseline on any metric; the revision will include this table.

We agree the paper should say this directly. The revision abstract will state the following: "NeSCal's calibration gains are best realized when composed with post-hoc recalibration: it reduces the error that differs between neighbors, and leaves the confidence-scale error they share to a single temperature." The introduction will add: "NeSCal is therefore a complement to post-hoc recalibration, and not necessarily a direct substitute for it."

Table R10. Train-time baselines on ImageNet-1K (frozen DINOv2 ViT-B/14 features, MLP head). Each baseline is tuned by validation NLL over a subset of the paper's search space. Mean ± SD over four seeds.

MethodNLL ↓NLL+TS ↓ECE % ↓ECE+TS % ↓Acc % ↑
CE0.6565 ± 0.00290.6440 ± 0.00273.92 ± 0.101.84 ± 0.1383.09 ± 0.10
FL0.6925 ± 0.00130.6686 ± 0.00177.54 ± 0.041.25 ± 0.0382.06 ± 0.04
LS0.6669 ± 0.00120.6357 ± 0.00134.15 ± 0.052.81 ± 0.1283.71 ± 0.05
MMCE0.6550 ± 0.00370.6437 ± 0.00353.79 ± 0.101.85 ± 0.1583.03 ± 0.10
MDCA0.6557 ± 0.00330.6432 ± 0.00303.90 ± 0.061.86 ± 0.1083.14 ± 0.09
s-AvUC0.6556 ± 0.00260.6432 ± 0.00243.95 ± 0.071.84 ± 0.1083.08 ± 0.06
MbLS0.6525 ± 0.00140.6332 ± 0.00172.83 ± 0.052.69 ± 0.0783.65 ± 0.04
NCR0.6327 ± 0.00220.6250 ± 0.00261.39 ± 0.032.34 ± 0.1183.65 ± 0.04
NeSCal0.6051 ± 0.00160.6055 ± 0.00170.95 ± 0.041.08 ± 0.0983.77 ± 0.03

2. Peak memory on ImageNet-1K. Table R5 gave the overall peak, 14.1 GB against 4.6 GB for CE. Table R11 breaks it down. During training NeSCal uses 9.5 GB: CE's 4.6 GB plus the 4.9 GB prediction cache, which is the only memory the method adds. The 14.1 GB peak is a one-off at activation, when the cache is seeded from the full table of logits and its softmax held at once. Seeding the cache in chunks gives an identical model (the same validation NLL) and lowers the overall peak to 10.5 GB; the revision will report this version. The exact kNN graph is built once beforehand and peaks at 10.4 GB (Table R5), or uses no GPU memory with an IVF graph built on CPU (Table R4).

Table R11. Peak GPU memory (GB) on ImageNet-1K (n = 1.23M, K = 1000; the cache is 4.92 GB in fp32), seed 18.

PhaseCENeSCalNeSCal, cache seeded in chunks
Before activation4.564.554.55
Activation (one-off)—14.1110.48
Training with the penalty—9.529.52
Overall peak4.5614.1110.48

Mixup on CIFAR-100 and NeSCal combined with Mixup are running and will follow in our next update. We hope this resolves both questions; if it does, we would be grateful if you would consider raising your score.

Comments on the paper

8 comments

We thank the reviewers for their careful reading and constructive questions. Several points are raised by multiple reviewers and addressed below, while points raised by a single reviewer are answered under that review. We will upload a revised PDF with all changes marked in blue, and post a short note when it is up.

Summary of new evidence

  • NCR: in seven settings up to ImageNet-1K, after temperature scaling (TS) NeSCal has lower mean NLL and ECE than NCR in every setting, significantly lower NLL in five, and is never significantly worse. A target × graph factorial attributes the gap to NeSCal's fixed, full-dataset graph.
  • Improvements Beyond Accuracy: on test points where NeSCal and CE agree on the label, NeSCal's NLL is significantly lower in six of seven settings; TS has 28–90% less to correct on NeSCal than on CE end to end, and nothing on ImageNet-1K.
  • Seed robustness: in 36 paired comparisons with every train-time baseline in the five end-to-end image settings, NeSCal is never significantly worse after TS.
  • Scale: on ImageNet-1K, NeSCal cuts NLL by 7.8% and ECE from 3.92% to 0.95% versus CE and raises accuracy, after a 159 s one-off graph build; approximate graphs match exact ones; graph rebuilding and cache staleness are tested.
  • Divergences: of four divergences, JS gives the best post-TS NLL; the classical graph-Laplacian regularizer is significantly worse.

1. NeSCal vs. NCR

Raised by AspL (Q1, W1), MJe8 (Q1, W1) and uUh2 (W1).

We ran NCR (Iscen et al., 2022) on all five end-to-end image settings and both pre-trained backbones, under the protocol used for every other method in the paper: the loss of the authors' released implementation, its onset and weight tuned on a grid built around the authors' reported settings, and selection by validation NLL (full details will appear in the revision).

After TS, NeSCal has a lower mean NLL and a lower mean ECE than NCR in all seven settings. Paired over seeds (95% t intervals), its post-TS NLL is significantly lower in five: CIFAR-100/R50 (−0.066 [−0.085, −0.047]), CIFAR-10/R50 (−0.027 [−0.032, −0.021]), CIFAR-100/WRN (−0.032 [−0.058, −0.007]), Tiny-ImageNet (−0.042 [−0.055, −0.028]) and ImageNet-1K (−0.020 [−0.022, −0.017]). Its post-TS ECE is significantly lower on CIFAR-100/R50, CIFAR-100/WRN and ImageNet-1K. Neither is ever significantly higher. Before TS, NCR has the lower mean ECE on the four CIFAR settings and the ViT head, while NeSCal has it on the two largest datasets, Tiny-ImageNet (2.02% vs. 4.18%) and ImageNet-1K (0.95% vs. 1.39%). A single validation-fitted temperature removes global over- or under-confidence, and NCR's base-ECE advantage does not survive it. This is the distinction raised Section 6.2 from Proposition 3: smoothing acts on errors that differ between neighbors, which a global temperature cannot reach.

NeSCal's advantage is also not an accuracy effect. Where NCR is the more accurate model (CIFAR-10/R50, CIFAR-100/WRN and CIFAR-10/WRN, by 0.1–0.9 points), NeSCal still has the lower post-TS NLL, significantly so on CIFAR-10/R50 and CIFAR-100/WRN. Across all seven settings, the only significant NLL difference in NCR's favor is the base NLL on CIFAR-10/WRN (+0.014 [+0.007, +0.021]), and it disappears after TS. The full results appear in Table R1 below.

NeSCal and NCR differ on several axes: the target, the graph, the divergence and the gradient flow. To isolate the two design choices uUh2 names, we ran a 2×2 factorial on CIFAR-100/R50 (Table R2). It changes only the target and the graph and holds every other component, including λ, at NeSCal's selected values. The graph is the decisive component: with minibatch neighborhoods, post-TS NLL rises by 0.118 [0.086, 0.150] with per-edge targets and by 0.132 [0.105, 0.159] with mixture targets, and ECE rises with it before and after TS, while accuracy is unchanged (76.7–77.1% vs. 77.2%). The target matters less: on the frozen graph, the mixture target is within seed variation of NeSCal on NLL before and after TS.

NeSCal's contribution relative to NCR is therefore the fixed, full-dataset neighborhood graph, which carries the empirical gain, together with an analysis of when smoothing on such a graph helps or harms calibration; NCR, designed for label noise, provides no such analysis. Of the two remaining axes, the divergence is tested in our reply to MJe8 (Table R7): NCR's KL worsens base NLL by 0.014 and ties after TS. Gradient flow is not varied separately.

References. Iscen, A., Valmadre, J., Arnab, A., & Schmid, C. (2022). Learning with neighbor consistency for noisy labels. CVPR. Released implementation: https://github.com/google-research/scenic/tree/main/scenic/projects/ncr

2. Calibration gains beyond accuracy

Raised by AspL (Q2, W3), MJe8 (Q2) and uUh2 (W3).

We agree that base-model ECE is not where NeSCal leads against other calibration methods: MMCE, MDCA and s-AvUC optimize a calibration surrogate directly, and much of the base-ECE advantage of methods that lower confidence directly is recoverable by a single temperature (Section 6.2). Relative to CE, however, NeSCal has lower mean base ECE in every image setting, including both pre-trained backbones (Tables 7 and R3), and the four settings where it does not are exactly the four with the least neighbor disagreement at activation (mean edge JS ≤ 0.013; Table 2), where there is little for smoothing to remove. All methods are selected by validation NLL, which does not target base ECE; selecting NeSCal by ECE instead lowers its test ECE by 0.85 points on average, at a cost of 0.017 nats (Appendix E.9). Our aim is not a method that leads on every calibration metric. The contribution is graph smoothing as a principle for train-time calibration, with an analysis of when it can and cannot help (Propositions 1–3). NeSCal is one instantiation of it: it reduces miscalibration relative to CE wherever neighbors disagree, improves proper scores, and composes with post-hoc recalibration. The revised abstract and introduction will state the claim in exactly these terms.

The reviewers ask whether the NLL improvement is just an accuracy improvement. It is a fair question, and we answer it in six ways.

  1. Holding predicted labels fixed (MJe8's suggestion). We restrict each comparison to the test points on which NeSCal and the other method predict the same label. Correctness is then identical sample by sample, so any difference in ECE or NLL comes only from how probability is assigned.
    • Against CE: NeSCal's ECE is significantly lower in four of seven settings before TS and five of seven after, and never significantly higher. Its NLL is significantly lower in six of seven (Table R3).
    • Against FL and NCR, two of the strongest baselines on base ECE, NeSCal's post-TS NLL on the shared-label points is significantly lower in seven of ten comparisons and never higher, and its post-TS ECE is never higher.
  2. The method, not TS, removes the confidence-scale error. The NLL that a validation-fitted temperature removes measures how much of a model's miscalibration is a global confidence-scale error. It needs no binning, and because TS never changes predicted labels, it is measured at each model's own accuracy. If NeSCal's calibration came from TS, TS would have as much to remove from NeSCal as from CE. It has 28–90% less across the five end-to-end image settings (CIFAR-100/R50: 0.029 → 0.008), and on ImageNet-1K nothing at all is left to remove (Table R3). The exception is the CIFAR-100 ViT head, where CE leaves only 0.002 to remove.
  3. At matched accuracy, calibration still improves. On Tiny-ImageNet, NeSCal's accuracy is indistinguishable from CE's (+0.21 points [−0.25, +0.67]), yet ECE falls from 5.75% to 2.02% (−3.73 [−4.08, −3.37]), and the NLL that TS removes falls from 0.029 to 0.003.
  4. Similarly, when the graph is rebuilt periodically instead of frozen (Table R6), accuracy is unchanged (77.3% vs. 77.2%), still two points above CE, but base ECE returns to CE's level (5.29–6.02% vs. 5.90%), and so does the NLL that TS removes (0.026–0.040 vs. 0.029 for CE and 0.008 for NeSCal). The calibration effect comes from a specific design choice, not from accuracy.
  5. Post-TS NLL mainly measures refinement, because TS has already corrected the global confidence scale; refinement is what a train-time method can add on top of TS. In the five end-to-end image settings, NeSCal's post-TS NLL and ECE are never significantly worse than those of any train-time baseline we ran (see reply to AspL, Q3), and on CIFAR-100/R50 even the runner-up after TS, FL, is significantly behind (−0.051 [−0.086, −0.016]). In the submitted comparison, NeSCal is first or second on post-TS ECE in five of eight image and tabular settings (Table 7), and its post-TS NLL is 6.3% and 6.0% below the strongest baseline on CIFAR-10/R50 and CIFAR-100/R50.
  6. The calibration gains over CE concentrate where neighboring predictions disagree: CIFAR-100 and Tiny-ImageNet, with mean edge JS at activation of 0.19–0.25. They are small on CIFAR-10 (≈0.03), where there is little disagreement to remove (Section 6.4, Table 2). This is consistent with Proposition 3: smoothing reduces the error that differs between neighbors, not the error they share.

3. The analysis and the implemented training procedure

Raised by AspL (W2) and MJe8 (W2).

MJe8 identifies three gaps between the analysis and the implementation: one-sided gradients, cached targets and a fixed graph. The bounds of Section 3.2 are inequalities about a fixed predictor on a fixed graph, so they apply to whatever model training produces; Appendix E.11 evaluates them on the trained models with current predictions. The question is therefore whether training, as implemented, reduces the quantities that appear in them. We take the three points in turn.

One-sided gradients. Training penalizes JS(pi∥p^j)\mathrm{JS}(p_i\Vert\hat p_j), where p^j\hat p_j is a detached copy of the neighbor's prediction (Section 4.3, Algorithm 1), so each step moves only pip_i. That step still points toward the target at any level of disagreement: the inner product of the gradient with respect to pip_i with pi−p^jp_i-\hat p_j equals a symmetric KL divergence between pip_i and the midpoint of the two distributions, which is positive whenever they differ and, by Pinsker's inequality, at least ∥pi−p^j∥12/(4ln⁡2)\Vert p_i-\hat p_j\Vert_1^2/(4\ln 2). On reciprocal edges (43.5% of edges on CIFAR-10/R50 and 43.9% on CIFAR-100/R50; Table 19), both endpoints move toward each other, and to first order the pair moves toward its average, which is the operation Proposition 3 analyzes.

Cached targets. Neighbor targets are cached predictions rather than current ones. Because the confidence-smoothness bound (Proposition 5) holds for any two distributions, the triangle inequality splits the smoothness term of Proposition 1 in two, with the same constant: ∣ci−cj∣≤12∥pi−pj∥1≤2ln⁡2 (JS(pi∥p^j)+JS(p^j∥pj))|c_i-c_j| \le \tfrac12\Vert p_i-p_j\Vert_1 \le \sqrt{2\ln 2}\,\big(\sqrt{\mathrm{JS}(p_i\Vert\hat p_j)}+\sqrt{\mathrm{JS}(\hat p_j\Vert p_j)}\big). The first term lies across the edge, between sample ii and its neighbor's cached prediction, and is what each step penalizes; the second stays at the neighbor, between its cached and current predictions, and measures staleness. On CIFAR-100/R50, averaged over training, the first is 0.345 and the second 0.147, and the staleness term shrinks as training converges, from 0.215 in the first ten epochs after activation to 0.092 in the last ten. Even this overstates what training sees: it compares a sample's own cached entry, written about an epoch earlier on a different augmentation, with its current prediction, whereas the neighbor entries training reads are 0.58 epochs old on average and never older than two epochs. Publishing targets only every 5 epochs raises the staleness term only to 0.155 and changes no metric (post-TS NLL −0.001 [−0.007, +0.006]; Table R6), so most of it is augmentation noise rather than drift.

Fixed graph. Freezing the graph does not invalidate the bounds, which hold on whatever graph is used. It can only change which samples are connected, and hence the correctness gaps ∣ai−aj∣|a_i-a_j| on the edges. Rebuilding the graph every 10 or 25 epochs does not improve post-TS NLL and removes most of NeSCal's effect on base calibration (Section 2 above, point 4), so the frozen graph is the configuration that carries the effect.

Theory and experiment. The analysis is idealized by design: it identifies the quantities that govern calibration under smoothing, predictive divergence and the correctness gap, and the experiments test its predictions under the procedure as implemented, where its assumptions hold only approximately. The predictions hold. NeSCal lowers the across-seed dispersion of neighboring errors, the quantity Proposition 3 says averaging acts on (Table 19); its gains survive TS, as Proposition 3 anticipates (Table 1), and are small wherever neighbors barely disagree (Table 2); kNN graphs yield lower NLL than random graphs of the same degree, consistent with the role of the correctness gap (Table 14); and in the synthetic study, smoothing helps when neighboring errors differ and hurts when neighbors share a bias across a correctness gap, the failure mode the analysis predicts (Appendix E.11).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.