- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors aim to explain why logit regularization, which includes the widely applied label smoothing technique, can improve model generalization. They point out that many previous works have attributed the benefits of label smoothing mainly to preventing overconfidence. They argue that this explanation is incomplete and propose a new one: logit regularization encourages the logits of training samples to cluster around their corresponding class-specific optimal values (i.e., logit clustering), thereby changing the model's preference for certain solutions (i.e., implicit bias). Specifically, the authors prove that, for Gaussian data or any data distribution with a quadratic per-sample loss, the optimal weight direction aligns with the LDA direction. Using a signal-plus-noise model, the authors further investigate how this implicit bias improves generalization.
In their experiments, the authors validate their theoretical predictions using synthetic Gaussian data and ResNet-18 embeddings extracted from the CIFAR-10 images. They show that the later real-world feature representations align with the signal-plus-noise model, providing empirical support for their analysis.
Critical strengths
- With detailed proof, the authors propose a new explanation that logit regularization leads to logit clustering and thus aligns the optimal weight direction with the LDA direction with better generalization.
- The authors verify their theoretical analysis not only with synthetic data but also with real-world ResNet embedding features.
- The connection to grokking is interesting and provides an appealing interpretation of delayed generalization under logit regularization.
Critical weaknesses
- The authors do not clearly explain how this work connects to related work, like Neural Collapse. They only state that their results are reminiscent of neural collapse without any further discussion. I wonder if confidence reduction and neural collapse are the consequences of logit clustering. I also wonder if this work is complementary to previous works or an alternative.
- The authors do not clearly state how broadly their results can be applied. To my knowledge, label smoothing works for many tasks, many of which are not well captured by the simple signal-plus-noise model. As with the first weakness, a detailed comparison between previous works might help clarify the scope and limitations of the proposed explanation.
Questions for authors
- What is the connection between this work and the previous works on confidence reduction and neural collapse?
- Is the LDA direction the main reason behind the effectiveness of label smoothing, or just one factor?
- Is there any case where the max-margin direction is better than the LDA direction? Have you ever considered such cases and checked if label smoothing works for them? If so, can the proposed framework be extended?
2 comments
I thank the authors for their response. However, it seems that some of my concerns remain open for further investigation. I will maintain my score.
We thank the reviewer for the constructive and positive feedback. Below, we address the concerns and clarify the points that have been raised.
Regarding Q1 / W1: Thank you for this important question. We agree that there are good reasons to expect a relationship between logit clustering and neural collapse. Our findings suggest a possible mechanism: when features are trainable, fitting common class-specific logit targets may encourage within-class feature clustering along directions visible to the final classifier. Moreover, for multiclass label smoothing, the optimal centered logit targets form a regular simplex, suggesting a connection to the geometry associated with NC2.
For example, this is consistent with Guo et al., "Cross Entropy versus Label Smoothing: A Neural Collapse Perspective" (https://arxiv.org/abs/2402.03979), who observe faster and stronger NC1 and NC2 under label smoothing.
However, further experiments are required to clarify this interplay. We thus view this as a promising direction for future work and will mention it in the revised discussion.
Regarding confidence reduction: lowering confidence (that is, increasing the label-smoothing parameter) moves the preferred centered logits toward zero. Our results predict that the results are quite robust to the location of the per-sample minimum (see section 3.4 for example), as long that the minimum is not too flat. Thus we expect to obtain such robustness to the confidence. We will address it also in the revised version.
Regarding W2: The key results of Sec. 3 (logit clustering and, in particular, the connection to the LDA direction) do not rely on the specific signal-plus-noise model. The mechanism presented there should be applicable quite broadly: logit regularizers that yield finite per-sample minima encourage the last-layer logits to cluster around these targets. Only in Sec. 4, building on the conclusions of Sec. 3, we introduce this model to study the consequences for generalization. We will revise the text to emphasize this distinction.
Regarding Q2: In our setup, the implicit bias toward the LDA direction (or a logit-clustering direction that is close to it) is indeed the main factor behind the effectiveness of label smoothing. However, more generally, there may be other factors: our results suggest that this implicit bias is shared by a broad class of logit regularizers, of which label smoothing is one particular instance.
There may therefore be additional qualities that make label smoothing an effective choice within this class. Identifying these factors and determining if label smoothing is indeed the best choice among all of the possible logit-regularization functions is an interesting direction for future research.
Regarding Q3: Yes, depending on the data, there are cases where the max-margin solution is better than the LDA (logit clustering) direction. A direct example of this can be seen in the bottom-left panel of Fig. 4. While further study is needed, our results suggest that in the presence of spurious features, logit regularization could be in particularly beneficial. The reason is that the max-margin training may learn the wrong direction (see for example Sagawa et al. https://arxiv.org/abs/2005.04345), and logit regularization may mitigate this effect (as our results show it implies robustness to orthogonal noise amplitude, see Prop. 6.1).