acceptodds
Under review as a conference paper at ICLR 2027

The Implicit Bias of Logit Regularization

Abstract

Label smoothing is known empirically to improve calibration and generalization in classification tasks, but its mechanism remains under-explored. Mathematically, label smoothing can be viewed as a special case of a broader class of logit regularization: the addition of a convex penalty directly in logit space. In this work, we analyze these logit regularizers in the context of linear classification, and demonstrate that they induce an implicit bias of **logit clustering** around finite per-sample targets. Remarkably, this reveals a novel connection between logit regularization and Linear Discriminant Analysis (LDA): for Gaussian data, we prove that such regularization drives the weight vector to align exactly with the LDA direction, a result which extends approximately to any distribution when the logits are sufficiently clustered. To demonstrate the consequences, we study a simple signal-plus-noise model in which this transition has dramatic effects: Logit regularization halves the critical sample complexity and induces grokking in the small-noise limit, while making generalization robust to noise. Our results extend the theoretical understanding of label smoothing and highlight the efficacy of a broader class of logit regularization methods.

Primary area:
learning theory
Keywords:
Logit regularization, Grokking, Implicit bias, Label smoothing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ruzcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors aim to explain why logit regularization, which includes the widely applied label smoothing technique, can improve model generalization. They point out that many previous works have attributed the benefits of label smoothing mainly to preventing overconfidence. They argue that this explanation is incomplete and propose a new one: logit regularization encourages the logits of training samples to cluster around their corresponding class-specific optimal values (i.e., logit clustering), thereby changing the model's preference for certain solutions (i.e., implicit bias). Specifically, the authors prove that, for Gaussian data or any data distribution with a quadratic per-sample loss, the optimal weight direction aligns with the LDA direction. Using a signal-plus-noise model, the authors further investigate how this implicit bias improves generalization.

In their experiments, the authors validate their theoretical predictions using synthetic Gaussian data and ResNet-18 embeddings extracted from the CIFAR-10 images. They show that the later real-world feature representations align with the signal-plus-noise model, providing empirical support for their analysis.

Critical strengths
  1. With detailed proof, the authors propose a new explanation that logit regularization leads to logit clustering and thus aligns the optimal weight direction with the LDA direction with better generalization.
  2. The authors verify their theoretical analysis not only with synthetic data but also with real-world ResNet embedding features.
  3. The connection to grokking is interesting and provides an appealing interpretation of delayed generalization under logit regularization.
Critical weaknesses
  1. The authors do not clearly explain how this work connects to related work, like Neural Collapse. They only state that their results are reminiscent of neural collapse without any further discussion. I wonder if confidence reduction and neural collapse are the consequences of logit clustering. I also wonder if this work is complementary to previous works or an alternative.
  2. The authors do not clearly state how broadly their results can be applied. To my knowledge, label smoothing works for many tasks, many of which are not well captured by the simple signal-plus-noise model. As with the first weakness, a detailed comparison between previous works might help clarify the scope and limitations of the proposed explanation.
Questions for authors
  1. What is the connection between this work and the previous works on confidence reduction and neural collapse?
  2. Is the LDA direction the main reason behind the effectiveness of label smoothing, or just one factor?
  3. Is there any case where the max-margin direction is better than the LDA direction? Have you ever considered such cases and checked if label smoothing works for them? If so, can the proposed framework be extended?
2 comments

We thank the reviewer for the constructive and positive feedback. Below, we address the concerns and clarify the points that have been raised.

  1. Regarding Q1 / W1: Thank you for this important question. We agree that there are good reasons to expect a relationship between logit clustering and neural collapse. Our findings suggest a possible mechanism: when features are trainable, fitting common class-specific logit targets may encourage within-class feature clustering along directions visible to the final classifier. Moreover, for multiclass label smoothing, the optimal centered logit targets form a regular simplex, suggesting a connection to the geometry associated with NC2.

    For example, this is consistent with Guo et al., "Cross Entropy versus Label Smoothing: A Neural Collapse Perspective" (https://arxiv.org/abs/2402.03979), who observe faster and stronger NC1 and NC2 under label smoothing.

    However, further experiments are required to clarify this interplay. We thus view this as a promising direction for future work and will mention it in the revised discussion.

    Regarding confidence reduction: lowering confidence (that is, increasing the label-smoothing parameter) moves the preferred centered logits toward zero. Our results predict that the results are quite robust to the location of the per-sample minimum (see section 3.4 for example), as long that the minimum is not too flat. Thus we expect to obtain such robustness to the confidence. We will address it also in the revised version.

  2. Regarding W2: The key results of Sec. 3 (logit clustering and, in particular, the connection to the LDA direction) do not rely on the specific signal-plus-noise model. The mechanism presented there should be applicable quite broadly: logit regularizers that yield finite per-sample minima encourage the last-layer logits to cluster around these targets. Only in Sec. 4, building on the conclusions of Sec. 3, we introduce this model to study the consequences for generalization. We will revise the text to emphasize this distinction.

  3. Regarding Q2: In our setup, the implicit bias toward the LDA direction (or a logit-clustering direction that is close to it) is indeed the main factor behind the effectiveness of label smoothing. However, more generally, there may be other factors: our results suggest that this implicit bias is shared by a broad class of logit regularizers, of which label smoothing is one particular instance.

    There may therefore be additional qualities that make label smoothing an effective choice within this class. Identifying these factors and determining if label smoothing is indeed the best choice among all of the possible logit-regularization functions is an interesting direction for future research.

  4. Regarding Q3: Yes, depending on the data, there are cases where the max-margin solution is better than the LDA (logit clustering) direction. A direct example of this can be seen in the bottom-left panel of Fig. 4. While further study is needed, our results suggest that in the presence of spurious features, logit regularization could be in particularly beneficial. The reason is that the max-margin training may learn the wrong direction (see for example Sagawa et al. https://arxiv.org/abs/2005.04345), and logit regularization may mitigate this effect (as our results show it implies robustness to orthogonal noise amplitude, see Prop. 6.1).

I thank the authors for their response. However, it seems that some of my concerns remain open for further investigation. I will maintain my score.

Reviewer_7EPuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors have characterized the implicit bias of convex logit regularization, such as label smoothing, in linear classification. They showed that such regularization results in logit clustering around the finite minima of the per-sample loss function. This provides a straightforward geometric explanation for the known label smoothing effect, and demonstrates that the phenomenon is not unique to label smoothing, but is a general consequence of any loss that can be decomposed as a sum of per-sample losses with a finite minimum. The theoretical results in this paper are quite interesting, but their practical application remains unclear.

Critical strengths
  1. This paper presents a unified formulation for convex logit regularization techniques, such as label smoothing, expressed on a per-sample basis as (1−α)ℓCE+αf(z)(1-\alpha)\ell_{CE} + \alpha f(z). Compared to viewing label smoothing merely as the use of soft labels, this interpretation is more general and encompasses a wider range of logit penalties.
  2. This paper reveals a novel connection between logit regularization and Linear Discriminant Analysis (LDA), which is insightful.
  3. The theoretical results of this paper help in understanding the source of robustness in logit regularization.
Critical weaknesses
  1. While the results from the linear probe in the paper support the qualitative predictions, they are insufficient to demonstrate that the conclusions hold generally for deep networks.
  2. LDA alignment is exact only under Gaussian assumptions or with a purely quadratic loss. While the paper’s Limitations section acknowledges this as an open question, the phrasing used in the main text and abstract—stating that it "extends approximately to any distribution"—is stronger than the evidence warrants; such a claim may be inappropriate.
Questions for authors
  1. I understand that this is a theoretical paper, but could the author explain how these results might be useful for applications in the era of deep learning?
  2. Is the σn\sigma_n invariance (Prop 6.1) truly unique to logit regularization, or does it hold for any per-sample loss at its unique optimal solution?
1 comment

We would like to thank the reviewer for the positive assessment and the detailed feedback. We will address each concern and question below.

  1. Regarding Q1 / W1: Thank you for raising this issue. We agree that extending these results to deep networks requires further work, but we see two potential practical implications:

    • Our analysis shows that label smoothing is one member of a broader family of logit regularizers. This motivates exploring whether other logit regularizaters may work better in particular settings.
    • By linking the effects of regularization to the structure of the data, our analysis may help explain when it helps or hurts, and guide the choice of regularization strength.

    We hope that this new theoretical perspective will encourage further research into these questions in deep networks.

  2. Regarding W2: We accept the criticism. We will revise the abstract and main text to clarify that the result extends to broader distributions only when they allow the logits to be sufficiently clustered (and therefore the loss becomes approximately quadratic around its minima).

  3. Regarding Q2: Indeed, the results hold for any per-sample loss with a unique finite solution. However, when the features are separable, logit regularization (or label smoothing) is the practical way to achieve such a loss: Not adding a regularization at all would not result in a unique minimum, and adding other kinds of regulatization would not result in a per-sample loss (and in such cases the theorem would be wrong, see for example Figure 17 in the appendix).

Reviewer_SvHcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how logit regularization changes linear classification. Finite preferred logits replace unbounded margin growth with target fitting. For Gaussian population data, the optimal direction follows linear discriminant analysis (LDA). A quadratic total loss gives a related result from first and second moments. A signal-plus-noise model then shows an additional regime of perfect generalization and invariance to matched orthogonal-noise scaling. Synthetic and frozen-feature experiments illustrate these effects.

Critical strengths

S1. The analysis explains when different logit penalties select the same direction. For Gaussian population data, convexity favors the smallest logit variance at a fixed positive mean. This yields the Fisher/LDA direction, while the loss determines the scale. The quadratic case provides a direct calculation. The result connects the known clustering effect to the classifier learned on fixed features, with clear exact cases.

S2. The signal model identifies a concrete generalization benefit. With a noiseless signal and a positive target, the full-rank problem has a unique pure-signal solution for d/N<1d/N<1. This adds the regime 1/2<d/N<11/2<d/N<1 to the earlier unregularized analysis. Figure 4 also identifies noise settings where regularization helps and where it hurts. These results explain a change in generalization under the same features and training data.

Critical weaknesses

W1. The converse of Proposition 5.1 needs a selection argument. For d/N>1d/N>1, the pure-signal minimizer still exists alongside other minimizers. A nontrivial null space does not establish which solution training selects. The optimizer, initialization, and positive-target assumption need to be stated. Zero-initialized convergent gradient descent offers a minimum-norm formulation that can support the intended converse. This is a fixable gap; the unique positive-target solution in 1/2<d/N<11/2<d/N<1 remains valid.

W2. The evidence for the non-Gaussian extension is indirect. Figure 2 compares the learned and LDA vectors through their cosine with the signal axis. Two vectors can have similar projections on that axis while pointing in different directions. Figure 7 directly measures their angle, but uses two-dimensional Gaussian samples. It therefore does not settle the broader claim for the heavy-tailed setting in Figure 2. This limits the empirical extension, rather than the exact Gaussian or quadratic results.

Questions for authors

Q1. Which optimizer and initialization select the minimizer in Proposition 5.1 for d/N>1d/N>1? Please state the positive-target assumption and justify the converse for that selected solution.

Q2. Is there a direct angle comparison between the learned and LDA vectors for Figure 2 among the existing results or saved solutions? If not, please distinguish its observed signal-axis alignment from the stronger directional claim. No new training is requested.

2 comments

We would like to thank the reviewer for the detailed and positive feedback. The reviewer has raised two small concerns, which we answer thoroughly below.

  1. Regarding Q1 / W1: We thank you for identifying this small gap. First, we assume a finite positive target z∗>0z^*>0 and full-batch gradient descent initialized at zero, with a step size ensuring convergence to a global minimum (we will revise Proposition 5.1 to mention it explicitly).

    Secondly, we agree that the existence of a nontrivial null space alone does not establish which minimizer training selects. The key point is that the weight vector can be decomposed into a component in the span of the training data and an orthogonal null-space component: S=Sspan+S⊥S=S_{\mathrm{span}}+S_\perp. Since the loss depends only on S⋅xiS\cdot x_i, the gradients lie in the training-data span, while S⊥S_\perp remains fixed at its initial value. Therefore, with zero initialization, GD selects the minimum-norm interpolant.

    In the noiseless-signal setting, the pure-signal vector Sp=(z∗/μf)e1S_p=(z^*/\mu_f)e_1 minimizes every per-sample loss. For N≥dN\geq d, full column rank makes this solution unique. For N<dN<d, it remains a minimizer, but GD instead selects S∞=PspanSpS_{\infty} =P_{span} S_p. Under nondegenerate Gaussian orthogonal noise, e1e_1 almost surely lies outside the training-data span in this regime. The selected solution consequently has a nonzero noise-coordinate component and strictly positive population classification error. We will revise the proposition and proof to make clear that the converse applies to this GD-selected solution.

  2. Regarding W2 / Q2: Thank you for pointing this out. We agree that comparing two vectors via their projection onto e1e_1 is a bit indirect. Following your suggestion, we directly computed the cosine similarity between the learned weight vector smin⁡(α)\boldsymbol{s}_{\min}(\alpha) and the sample's empirical LDA direction Σ^−1μ^\hat{\Sigma}^{-1}\hat{\boldsymbol{\mu}} for the exact same Student-tt setup. See the new figure here: https://ibb.co/TxmLztNK.

    As we can see, we obtain the exact same qualitative behavior: a sharp jump from α=0\alpha = 0 to α>0\alpha > 0, followed by a plateau very close to 11 (which becomes flatter and closer to 11 as ν\nu increases or as α→1\alpha \to 1, where it converges to 11 by Corollary 3.3). We will include this direct plot in the revised manuscript and clarify the distinction in Section 3.4.

Thank you for your response. I am satisfied with how you addressed my two questions by explaining which minimizer gradient descent selects from zero in Proposition 5.1 and directly comparing the learned weights with the empirical LDA direction in your new Student-t plot.

Generally, I find the paper's theoretical arguments sound, and I support acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.