- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The author of the paper define graph smoothing as a general principle for train-time calibration, as it encourages similar predictive distributions across neighboring samples in representation space. They finally proposes NeSCal, a graph-based train-time regularizer that penalizes a certain divergence between predictive distributions of neighboring samples in the embedding space. Empirically after temperature scaling, NeSCal attains the lowest NLL of all evaluated methods in both image and tabular settings.
Critical strengths
- The method is quite simple with no additional inference cost.
- In some cases, the empirical improvements in NLL persist even after temperature scaling.
- The theory identifies conditions under which neighborhood smoothing helps or hurts calibration.
Critical weaknesses
- The methodological novelty is limited and it's relative to neighbor consistency regularization, with no direct NCR comparison (even if it is cited).
- The theoretical analysis of idealized averaging does not establish guarantees for the actual training procedure.
- Since the paper is mostly focused on confidence calibration, alongside NLL in the main results there should be a calibration metric, as NLL does not distinguish calibration gains from improvements in predictive power. Although ECE is reported in the appendix, its improvements are not consistent across settings, making the evidence for better calibration less clear than that for improved NLL.
- Some improvements are small relative to seed variability, and results on text are less competitive.
Questions for authors
- Can you compare your method also with NCR which seems the most similar method with respect to yours? Both before and after TS
- Since NLL does not isolate calibration gains from improvements in predictive power, could you clarify how the ECE and other calibration results in the appendix support the paper’s main claims, given their mixed improvements across settings?
- Are the small gains over the strongest baselines statistically robust across seeds?
6 comments
Thank you for the detailed response and the additional experiments. The NCR comparison addresses one of my concerns. However, the calibration claim remains weakly supported:
- Base-model calibration. NeSCal’s base-model ECE (Table 7) is best only on Tiny-ImageNet, ranks 3rd–6th of eight methods elsewhere, and is higher than plain CE on Jannis, Covertype and both Amazon tasks.
- Contribution beyond TS. Compared with CE+TS, NeSCal+TS leaves post-TS ECE essentially unchanged in three of five image settings (e.g., 1.63 vs. 1.61 on CIFAR-100/R50). The remaining NLL gain mainly reflects better refinement and accuracy rather than calibration. The framing as a train-time calibration method (title, abstract, contribution (i)) is therefore not supported by calibration-specific metrics. The evidence mainly shows improved predictive quality, with calibration provided by post-hoc recalibration, consistent with the revised positioning in the rebuttal.
- Unexplained losses. Specific baselines outperform NeSCal in several settings, e.g., FL on base NLL for CIFAR-10/WRN, MbLS/LS on 20NG, and MMCE on Amazon (binary). Neither the paper nor the rebuttal explains why. Among the five end-to-end image settings, the margin over the runner-up after TS is significant only on CIFAR-100/R50.
Finally, the currently available PDF does not yet include the promised revisions.
Thank you for the follow-up, and for confirming that the NCR comparison resolves that concern. We take your remaining points in turn.
Base-model calibration. We agree that base-model ECE is not where NeSCal leads on most benchmarks, as Section 6.1 states. Three points qualify it.
- NeSCal's base-model ECE is below plain CE's in seven of the eleven settings, and it is the best of all methods on the two largest datasets: Tiny-ImageNet (Table 7, now part of Table 1) and ImageNet-1K, where it is the best of nine methods (0.95%, against 3.92% for CE; Table 28).
- The four settings where it is higher than plain CE (Jannis, Covertype and both Amazon tasks) are exactly the four with the least disagreement between neighbors at activation (mean edge JS ≤ 0.013, Table 2). Smoothing can only remove error that differs between neighbors (Proposition 3), so there it has little to act on. Section 6.4 now states this.
- All methods are selected by validation NLL, not ECE. Selecting NeSCal by ECE instead lowers its ECE by 0.85 points on average, at a cost of 0.017 nats of NLL (Appendix E.9).
Contribution beyond TS. Aggregate post-TS ECE compares models at different accuracies, and TS compresses it into a narrow band for every method (a spread of 0.5–1.1 points across methods on CIFAR). Two calibration-specific comparisons that hold accuracy fixed separate NeSCal+TS from CE+TS:
- On the test points where NeSCal and CE predict the same label, so correctness is identical sample by sample, NeSCal's post-TS ECE is significantly lower in five of seven settings, including CIFAR-100/R50 (−1.79 [−2.56, −1.02] points), and its NLL is significantly lower in six of seven (Section 6.2, Table 37). On these points, the NLL gain cannot come from accuracy.
- TS does not change predicted labels, so the NLL it removes measures confidence-scale miscalibration at each model's own accuracy. TS removes 28–90% less NLL from NeSCal than from CE across the five image settings (0.008 vs. 0.029 nats on CIFAR-100/R50).
Part of the remaining gain is refinement, which a single temperature cannot supply: NeSCal's refinement term is lower than CE's in ten of eleven settings (Table 9). This is the complementarity the revised framing claims. We agree with the positioning you describe: NeSCal improves predictive quality and is designed to be composed with post-hoc recalibration, not to replace it. The revised abstract, introduction (including the contributions) and conclusion now say this directly. We kept the title because calibration remains the goal; the revision states how NeSCal contributes to it.
Unexplained losses.
- FL, base NLL on CIFAR-10/WRN: a confidence-scale effect. TS removes 0.0004 nats from FL and 0.019 from NeSCal, after which the two are tied (0.1568 vs. 0.1566, Table 1).
- MbLS/LS on 20NG: an accuracy effect. Both are about 6 points more accurate than CE there (83.2% and 82.7% vs. 76.9%, Table 6). NeSCal's gain on 20NG is limited by the model, a convolutional network over frozen word embeddings with little room to adapt its features (Section 6.4).
- MMCE on Amazon (binary): one of the four low-disagreement settings above. It has the highest edge agreement of all settings (0.987, Table 2), so there is little for smoothing to act on.
- Margin over the runner-up after TS: four seeds give limited power within a single setting, so we also test across settings. NeSCal's post-TS NLL is lower than that of each baseline in at least seven of the eight image and tabular settings (Table 8, p ≤ 0.023). In 37 paired comparisons across the five end-to-end image settings, it is never significantly worse after TS (Appendix F.9). On ImageNet-1K, it is significantly better than every baseline, including the runner-up, on NLL before and after TS and on base-model ECE (Table 29).
Revised PDF. The revised PDF is now uploaded, with changes in blue. ECE before and after TS is in Table 1, the label-fixed comparison in Section 6.2, the low-disagreement settings in Section 6.4, and NCR in Section 6.5.
We hope the revision addresses your remaining concerns, and would be glad to clarify anything further before the discussion period closes.
We thank you for your review. Your Questions 1–2 and Weaknesses 1–3 are shared with the other reviewers and answered in the general response above: NCR before and after TS in Section 1, calibration beyond accuracy in Section 2, and the analysis of the procedure as implemented in Section 3. As you suggest in W3, the main results table will now report ECE and post-TS ECE alongside NLL. For Q2, the appendix result that speaks most directly to your question is the calibration–refinement decomposition: NeSCal's refinement term, the component that recalibration cannot improve, is lower than CE's in ten of eleven settings (Table 10).
Q3 / W4. Are the gains statistically robust across seeds? Yes. In the image settings no baseline is ever significantly better than NeSCal after TS, and across the eight image and tabular settings NeSCal has the lower post-TS NLL against each baseline in at least seven. We test this at two levels.
Within each setting. In each of the five end-to-end image settings, we paired NeSCal with every train-time baseline over the four seeds and computed a 95% t interval on the difference: 36 setting–baseline comparisons (the paper's six baselines and NCR in all five settings, plus Mixup on CIFAR-10/R50). After TS, NeSCal is never significantly worse: its NLL is significantly better in 21 comparisons and its ECE in 14, and all others are ties. Before TS, its NLL is significantly better in 23 and worse in only two, both on CIFAR-10/WRN, and neither survives TS. Against the runner-up after TS, the margin is significant on CIFAR-100/R50 (FL, −0.051 [−0.086, −0.016]) and within seed variation in the other four settings, which is why we also test across settings.
Across settings (Table 9 of the paper). NeSCal's post-TS NLL is lower than that of FL, LS, MMCE and MbLS in all eight image and tabular settings, and lower than that of MDCA and s-AvUC in seven; exact Wilcoxon signed-rank tests on the per-setting means give p = 0.008–0.023 (reported as exploratory in Appendix E.2).
W4. Text results. The smaller gains on text are a property of these benchmarks, not of the modality, and Proposition 3 explains them: smoothing can only remove error that differs between neighbors, so where neighboring predictions already agree there is little for it to remove. The four settings with the least disagreement at activation are two text and two tabular ones (Amazon binary and 5-class, Covertype, Jannis; mean edge JS 0.003–0.013), and all four gain 0.4–0.8% in NLL. The image settings, where disagreement is larger (0.026–0.254), gain 3.9–11.7% (Table 2). 20NG, whose disagreement matches CIFAR-10's, gains less than its disagreement would suggest (1.3%); Section 6.4 attributes this to a model that learns over frozen word embeddings and so has little room to adapt its features. NeSCal still lowers NLL relative to CE in all three text settings.
We believe these results resolve each of your concerns: NCR is now compared directly, before and after TS, in seven settings; calibration will be reported in the main results and separated from accuracy; and the gains are robust within and across settings. We would be grateful if you would consider raising your score.