acceptodds
Under review as a conference paper at ICLR 2027

Sparsifying Label Smoothing to Better Calibrate Classifiers

Abstract

Label smoothing is widely used to reduce neural network overconfidence. Existing variants, whether uniform or adaptive, differ in how much smoothing mass they assign and how they weight it, but all spread that mass densely across the label space, so semantically plausible alternatives and unrelated labels alike receive probability mass. We argue that effective label smoothing depends not only on the smoothing strength but also on the support over which smoothing mass is assigned: smoothing should be sparse and similarity-aware. We instantiate this principle with Sparse Similarity-Aware Label Smoothing (SSA-LS), a simple and practical method that uses latent-space class neighborhoods to redistribute probability mass only toward plausible alternatives. Across multiple datasets and architectures, SSA-LS consistently improves confidence calibration while preserving, and sometimes improving, accuracy. Beyond SSA-LS, similarity-aware sparsity acts as a general wrapper, improving the calibration of existing adaptive label smoothing methods without hurting accuracy. We further show that strong representations do not by themselves guarantee well-calibrated predictions, and through sparsity ablations and neighborhood corruption studies, that the gains arise specifically from structured, similarity-aware sparsity, establishing it as a critical ingredient for label smoothing.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
label smoothing, confidence calibration, neural network calibration, classifier

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rcobOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Sparse Similarity-Aware Label Smoothing (SSA-LS) to improve confidence calibration. Unlike conventional label smoothing, SSA-LS assigns smoothing probability only to the top-k similar classes based on pretrained feature representations. Experiments on CIFAR-100, TinyImageNet, and ImageNet-1k show improved calibration while maintaining comparable classification accuracy.

Critical strengths
  1. The paper addresses an important problem in confidence calibration and provides a simple method that can be easily applied to existing classifiers.

  2. The proposed method shows consistent improvements in ECE across multiple datasets and model architectures.

  3. The ablation studies on sparsity, class similarity, and feature representations provide useful insights into the proposed method.

  4. The method can also be applied to existing label smoothing approaches without changing the model architecture.

Critical weaknesses
  1. The methodological novelty is limited. Similarity-based label smoothing has already been studied in previous work, such as Liu and JaJa (2021) [1]. The main difference appears to be selecting only the top-k similar classes.

  2. The most closely related similarity-based label smoothing methods are not included in the main experiments. Direct comparisons are needed to show the benefit of the proposed sparsification.

  3. The method assigns the same soft label to all images in the same class. However, natural images often contain multiple objects, which may cause different levels of uncertainty. The current method does not directly consider this image-specific ambiguity.

  4. The ImageNet-1k experiments only train a small MLP on frozen CLIP features. This limits the evidence for the effectiveness of SSA-LS in large-scale end-to-end classifier training.

[1] Liu, C., & JaJa, J. (2021). Class-Similarity Based Label Smoothing for Confidence Calibration. In Artificial Neural Networks and Machine Learning – ICANN 2021, pp. 190–201. Springer.

Questions for authors
  1. What is the main difference between SSA-LS and the class-similarity-based label smoothing method proposed by Liu and JaJa (2021) [1], beyond selecting the top-k classes?

  2. Can the authors compare SSA-LS with existing similarity-based label smoothing methods under the same experimental settings?

  3. How does SSA-LS handle images containing multiple objects when only one class label is given? Can the authors explain how class-wise smoothing reflects the uncertainty of individual images?

  4. Can the authors provide results using end-to-end training on ImageNet-1k, rather than only training an MLP on frozen CLIP features?

[1] Liu, C., & JaJa, J. (2021). Class-Similarity Based Label Smoothing for Confidence Calibration. In Artificial Neural Networks and Machine Learning – ICANN 2021, pp. 190–201. Springer.

Reviewer_BEoUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates restricting label smoothing to semantically related classes to improve classifier calibration. It proposes Sparse Similarity-Aware Label Smoothing (SSA-LS). It constructs class neighborhoods using pretrained or learned representations and redistributes the surplus (smoothing) probability only among nearby classes. The authors also show that the sparsification procedure can be applied to existing adaptive label smoothing methods.

Critical strengths

a. Simple and straightforward method to implant. Paper written well. Demonstrate good results on some models like DenseNet-121 on CIFAR-100. b. Useful ablation demonstrating the role of smoothing support. Results show that calibration depends on neighborhood size k. The latent space comparisons along with corruption experiments reveal the importance of the selecting the smoothing support.

Critical weaknesses

a. Limited Novelty - There already exists class-similarity based label smoothing for confidence calibration (Liu and JaJa - https://arxiv.org/abs/2006.14028). The experiments do not show the role of sparsity when compared to this method. b. Although it can be established that sparsity in the neighborhood helps with Calibration Error, the experiments do not establish how structured semantic sparsity is actually responsible for the gains.

Questions for authors
  1. Relation to previous Research (Liu and JaJa, 2021) - how is your paper different from this paper. Can you experimentally compare.
  2. Tease out the role of semantic sparsity. What experiments can you conduct to separate the effects of sparsity, semantic neighborhood selection, and similarity-based weighting.
Reviewer_FrrwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Sparse Similarity-Aware Label Smoothing (SSA-LS), which restricts smoothing mass to a small set of semantically similar classes rather than distributing it across all non-target classes.

  • Class neighborhoods are precomputed from distances between class centroids in a fixed representation space.
  • The same sparsification procedure can also be applied to existing adaptive label smoothing methods.
  • Experiments on CIFAR-100, TinyImageNet, and frozen-feature ImageNet-1k classification report lower expected calibration error with generally comparable accuracy.
Critical strengths
  • The method is simple and easy to incorporate into an existing training pipeline. Class neighborhoods are computed once, and the encoder is not needed during classifier training or inference.

  • The ablations are a useful part of the paper. Varying neighborhood size and replacing semantic neighbors with random classes helps establish when restricting the smoothing support improves calibration. The comparisons using CLIP, DINOv2, and β-VAE features also suggest that the gains do not depend on a single pretrained encoder.

  • The improvements for LS++ and OLS are encouraging, as they show that the proposed sparsification can benefit existing target construction methods.

Critical weaknesses
  • The experiments do not fully separate the benefit of sparse support from that of concentrating smoothing mass. The dense-support ablation fixes the temperature, so it remains unclear whether a well-tuned dense semantic smoothing baseline could achieve similar results.

  • Hyperparameter selection is uneven across methods. SSA-LS selects kk for each dataset–architecture pair, while the margin-based baselines reuse parameters selected on ResNet-18 across architectures. Appendix H addresses smoothing-strength selection only in frozen-feature MLP experiments.

  • The claim of preserving accuracy is overstated. On ImageNet-1k, SSA-LS improves ECE over LS from 2.25 to 1.14, but accuracy falls from 74.73% to 74.34%. Similar results elsewhere suggest an accuracy–calibration trade-off that deserves a more careful discussion.

  • The CIFAR-100 frozen-feature results need clarification. CE accuracy is 78.12% in Table 13 and 79.92% in Table 14, despite the apparently similar setup. The paper should explain the source of this difference.

Questions for authors
  • Does SSA-LS still outperform dense similarity-based smoothing when the dense baseline’s temperature is tuned independently?

  • How do the comparisons change if the margin-based baselines receive the same per-architecture hyperparameter search as SSA-LS?

  • What accounts for the difference in CIFAR-100 frozen-feature CE accuracy between Tables 13 and 14 (78.12% versus 79.92%)?

  • Do the calibration gains persist under classwise calibration metrics and different ECE binning schemes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.