The Implicit Bias of Logit Regularization

Label smoothing is known empirically to improve calibration and generalization in classification tasks, but its mechanism remains under-explored. Mathematically, label smoothing can be viewed as a special case of a broader class of logit regularization: the addition of a convex penalty directly in logit space. In this work, we analyze these logit regularizers in the context of linear classification, and demonstrate that they induce an implicit bias of **logit clustering** around finite per-sample targets. Remarkably, this reveals a novel connection between logit regularization and Linear Discriminant Analysis (LDA): for Gaussian data, we prove that such regularization drives the weight vector to align exactly with the LDA direction, a result which extends approximately to any distribution when the logits are sufficiently clustered. To demonstrate the consequences, we study a simple signal-plus-noise model in which this transition has dramatic effects: Logit regularization halves the critical sample complexity and induces grokking in the small-noise limit, while making generalization robust to noise. Our results extend the theoretical understanding of label smoothing and highlight the efficacy of a broader class of logit regularization methods.