acceptodds
Under review as a conference paper at ICLR 2027

A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification

Abstract

Hard-label classification is usually trained with smooth surrogate losses such as softmax cross-entropy. We isolate a mechanism by which this mismatch between smooth surrogate and discrete labels produces power-law learning curves for a one-layer softmax student trained by online SGD on Gaussian inputs with argmax teacher labels. In the thermodynamic limit, the dynamics reduce to a growing student-teacher alignment and a residual student variance sustained by SGD noise. At late times, only shrinking layers of examples around teacher decision boundaries remain active. This yields an law in training time not only for the test loss but also for the generalization error, one minus test accuracy. The error law is an online-noise effect. The loss law persists without gradient noise. Learning-rate annealing separates the two exponents, improving the error exponent from 1/3 toward 1/2 while slowing the loss. Simulations support the predictions. Correlated Gaussian inputs and whitened pretrained features change transients, not the late-time decay. An MNIST MLP and the largest Pythia model show compatible error scaling. The mechanism is asymptotic and complementary to spectral explanations and data-statistics explanations of neural scaling laws.

Primary area:
learning theory
Keywords:
neural scaling laws, learning dynamics, generalization, stochastic gradient descent, statistical mechanics of learning, multiclass classification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7BQZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the convergence behavior of a softmax classifier trained with cross-entropy loss and online SGD. The goal is to derive the so-call 1/31/3 power-law of the convergence speed. Specifically, the authors consider a single-layer teacher with orthogonal linear neurons producing deterministic hard labels (which can be seen as a special case of [Liu et al. 2026] with zero temperature). The student network, also having a single layer, is trained with cross-entropy loss. While the prior work in [Liu et al. 2026] has shown the 1/31/3 power-law for the training (cross-entropy) loss assuming perfect alignment of teacher and student weights, this paper aims to further study the scaling of the test error, which is meaningful when the above weight-alignment does not occur due to random initialization. The authors study the limit when the number of student neurons approach infinity. The main contribution appears to be identifying a boundary-layer mechanism: as training progresses, only data samples concentrated near the teacher's decision boundaries contribute to learning, while stochastic updates maintain a nonzero residual variance. By analyzing the resulting optimization dynamics, the paper derives a (t^{-1/3}) convergence rate for both the training loss and test error. Experiments with deep networks show comparable scaling.

Critical strengths
  1. The so-called boundary layer mechanism could be new. Specifically, as training progresses, only data concentrated near the teacher's decision boundaries contribute to learning.
Critical weaknesses
  1. The paper is difficult to read. The authors claims that the central observation is identifying the boundary layer mechanism. However, there is no clean explanation on what it is after the introduction. Similarly, while several proofs are presented, it is difficult for the readers to understand how they together show the effect of the boundary layer, e.g., how it leads to the 1/31/3 power law. Perhaps the proofs are produced by AI; I think some efforts are needed in order to present the proof and intuitions in a way that readers can understand.

  2. Sometimes, results are presented with notations not clearly defined. For example, what are g1g_1 and h1h_1 in Lemma 1?

Questions for authors
  1. In Assumption (A7), could you define precisely what "permutation-symmetric" means?

  2. What are g1g_1 and h1h_1 in Lemma 1?

  3. In Lemma 2, what does "fixed finite bb" refer to?

  4. What do the pair aa and bb refer to above (11)? Even further above, why is the boundary layer of width O(D−1)O(D^{-1})?

2 comments

We thank the reviewer for the specific questions.

Two corrections to the summary. (i) The limit we take is the input dimension N→∞N \to \infty at a fixed number KK of classes (A1, A6). The student has KK output units throughout. (ii) The misalignment that controls the late-time error is not a remnant of the random initialization. It is maintained by SGD noise: without this noise, the error decays faster than any inverse power of training time (Section 4).

Q1. “Permutation-symmetric” in (A7). All classes are statistically equivalent: the overlaps depend only on whether two class indices coincide, Ja⋅Ta′/N=Rδaa′+S(1−δaa′)J_a \cdot T_{a'} / N = R \delta_{aa'} + S (1 - \delta_{aa'}) and Ja⋅Ja′/N=Qδaa′+C(1−δaa′)J_a \cdot J_{a'} / N = Q \delta_{aa'} + C (1 - \delta_{aa'}). In the limit (A6), the dynamics preserves this form. The overlaps from independent, identically distributed student initialization approach it as the input dimension grows. (A7) now states these equations.

Q2 / W2. g1g_1 and h1h_1 in Lemma 1. They are the class-1 values of two quantities defined a few lines above the lemma (lines 240 and 246): hah_a is the centered student logit of class aa (Eq. 8), and ga=paT−pag_a = p_a^{\mathrm{T}} - p_a is the teacher's one-hot label minus the student's softmax output. By (A7), every class gives the same averages, which we write for class 1; the revised text says so.

Q3. “Fixed finite bb” in Lemma 2. bb is the minibatch size, introduced in (A5) as “fixed finite batch size bb” (line 196). Lemma 2 now repeats the words “batch size”.

Q4. The pair aa, bb above Eq. (11), and the width O(D−1)O(D^{-1}). In the submitted version, aa and bb label two classes at the decision boundary between them, where their teacher logits coincide, ua=ubu_a = u_b. The letter bb clashed with the batch size, so this index is now a′a'. The width follows from Eq. (8). Let GG be the gap between the two largest teacher logits for an example. The corresponding student logit gap is DGDG plus noise of size Δ\sqrt{\Delta}. For bounded Δ\Delta, appreciable deviations from the one-hot label occur mainly when DGDG is O(1)O(1), hence G=O(1/D)G = O(1/D).

W1. The mechanism, and how the proofs combine

The explanation in Section 4 and Fig. 1 (right) was not gathered in one place. The revision gives the chain of reasoning in two paragraphs, “Boundary layers of width 1/D1/D” and “Why one third”:

  1. At late times, only examples in the boundary layers, G=O(1/D)G = O(1/D), contribute appreciably to the gradient. They are a fraction ∝1/D\propto 1/D of all examples.
  2. The test loss is approximately this fraction times an O(1)O(1) loss per example, so it decays as 1/D1/D.
  3. The logit scale DD follows the negative gradient of this loss at fixed Δ\Delta: D˙=−ηK−1 ∂Lg/∂D\dot D = -\frac{\eta}{K-1}\,\partial \mathcal{L}_g/\partial D, by Lemma 1. Hence D˙∝η/D2\dot D \propto \eta/D^2 and D∝α1/3D \propto \alpha^{1/3} at fixed learning rate.
  4. Locally, an error occurs when noise reverses the class ordering. Typical errors therefore have gaps G=O(Δ/D)G = O(\sqrt{\Delta}/D), giving error ∝Δ/D\propto \sqrt{\Delta}/D.
  5. At fixed learning rate, SGD noise keeps Δ\Delta at a nonzero limiting value set by η/b\eta/b, giving an error ∝α−1/3\propto \alpha^{-1/3}.

Theorem 3 and Corollary 4 make these steps precise and supply the prefactors. Appendix A now opens with a roadmap: A.1 reduces the dynamics to (D,Δ)(D, \Delta), A.2 shows that DD diverges at bounded Δ\Delta, A.3 evaluates the averages that drive the flow, and A.4 and A.5 integrate the result.

On the use of AI

Our AI-use statement discloses that AI tools assisted in formulating and writing proofs. All derivations and proofs were checked step by step by the authors, and the revised statement says so. The results can also be tested: the predicted exponents and prefactors agree, without fitted parameters, with SGD simulations of the student (Figs. 2, 3, 6, 7).

Soundness

We recognize that the presentation made the analysis difficult to assess. The review does not identify a specific erroneous step. We would welcome clarification of any remaining technical concern and will address it.

I thank the authors for the response, which clarifies some of the confusions. However, I think my concern on the readability of the paper remains. Thus, at this point I will keep my score.

Reviewer_dTxNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how the mismatch between smooth surrogates and discrete labels leads to the one-third scaling law in online softmax classification through the boundary-layer mechanism. The authors extend the prior results in Liu et al. (2026) from testing loss to generalization error. The theoretical results are supported by numerical simulations.

Critical strengths
  1. The paper builds on the prior work (Liu et al., 2026) and extends their analysis to generalization error with the support of numerical simulations.
  2. The authors clearly acknowledge the limitations of their theoretical results in Section 7.
Critical weaknesses
  1. The paper is difficult to follow for readers. It uses many technical terms without first explaining their meaning or purpose in simple terms. For example, their contributions in Section 1 starts with

We derive an exact centered macroscopic closure for symmetric online K class softmax learning

None of "exact", "centered", "macroscopic", "closure" and "symmetric" are with intuitive explanation. This is not an isolated sentence, the entire paper is full of such sentences.

  1. The math is equally difficult to follow. In Section 1, it is said that

The late-time dynamics becomes transparent only in centered coordinates, which remove the softmax redundancy under a common shift of all logits (Section 3): the centered student-teacher overlap DD, the centered student squared norm QeffQ_{eff}, and the residual student variance Δ=Qeff−D2\Delta = Q_{eff} − D^2, which comes from fluctuations of the student orthogonal to the teacher.

There is no commonly accepted meaning of "student-teacher overlap" and "centered student squared norm". In fact, neither "student" nor "teacher" is defined, and it is not obvious what the norm is of. Similarly, this is not an isolated problem; the entire paper is full of such issues.

Other examples:

  • ξ\xi first appears in Eq. (2) (line 188), but is not defined until line 194.
  • μ\mu appears in Eq.(4) (line 199) without definition.
  • α\alpha emerges in Abstract while its definition is in Eq.(4) (line 199).
  • ...
  1. It is unclear what new understanding this paper provides beyond existing work and why these results matter. The abstract and introduction contain too many technical details instead of explaining this work's motivation and significance.
Questions for authors
  1. Could the authors substantially improve the readability of this paper? Provide intuitive explanations of technical terms and define variables when you first use them.
  2. Why do you extend the one-third scaling law to generalization error? Why does it matter?
1 comment

We thank the reviewer for the concrete examples. They were useful for the revision.

W1 / Q1. Readability

The quoted sentence was too compressed, and we have replaced it. The contributions paragraph and the abstract have been rewritten in plain language. For reference, the terms mean the following. Exact: starting from the large-input-dimension equations, the reduction to two variables introduces no further approximation. Macroscopic: the quantities are averages that become deterministic in that limit. Closure: a set of equations for these quantities alone. Centered: the mean over classes, which does not affect softmax probabilities, is subtracted. Symmetric: all classes are statistically equivalent.

W2. Definitions

  • Student and teacher. In the teacher-student framework (Seung et al., 1992; Saad & Solla, 1995; Goldt et al., 2019), a fixed teacher network generates the labels, and a student network of the same form is trained on them. The submission defined teacher, student, and labels in (A1)-(A3), lines 179-193, and the overlaps and norms in Eqs. (5)-(7). There, DD is the student-teacher inner product and QeffQ_{\mathrm{eff}} the student squared norm, both formed from the centered, rescaled weight vectors and divided by NN. The passage quoted in the review is a preview in the Introduction that points to Section 3. We agree that this preview used the symbols too early. The revised Introduction defines teacher and student at first mention and describes DD and Δ\Delta in words.
  • ξ\xi is now introduced in (A1), before Eq. (2).
  • μ\mu counts SGD updates. The training time α=μ/N\alpha = \mu/N is the number of updates per input dimension. This is now stated at Eq. (4).
  • In the abstract, α\alpha is introduced as the training time; Eq. (4) gives the precise definition.

W3 / Q2. What is new, and why the classification error matters

Accuracy directly measures classification performance. Its complement, the classification error, need not decay at the same rate as the loss. The closest prior theory (Liu et al., 2026) assumes a student that stays aligned with the teacher. Its classification error relative to the teacher's predicted class is zero by construction, so it cannot say how accuracy improves. We show that the error follows a one-third law of a different origin: it requires SGD noise, which maintains a nonzero residual variance at fixed learning rate. Such an error law was known for the binary hinge-loss perceptron (Sclocchi & Wyart, 2024); we obtain both laws in one model. Suitable learning-rate decay can therefore make the error fall faster and the loss more slowly in this model (Section 5, Fig. 3). The revised Introduction states what was known, and Table 1, now in Section 2, compares the three analyses.

Soundness

We recognize that unclear definitions and explanations can make soundness difficult to assess. The review does not identify a specific error in a derivation or result. We would welcome clarification of any remaining technical concern and will address it. The predicted exponents and prefactors are compared, without fitted parameters, with SGD simulations of the student (Figs. 2, 3, 6, 7).

Reviewer_gNTMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the learning dynamics of online softmax classification using a teacher student model. The authors explain how a boundary-layer mechanism leads to a 1/3 scaling law for both test loss and classification error. They also show that SGD noise affects classification error and that learning-rate annealing can improve its asymptotic decay rate. The theoretical results are supported by numerical simulations and experiments on MNIST and Pythia models.

Critical strengths

The paper provides a useful explanation of how decision boundaries and SGD noise affect classification error. The distinction between test loss and classification error is interesting, and the analysis extends existing results on one-third scaling.

The authors derive detailed learning dynamics and test their predictions under different learning rates, batch sizes, and numbers of classes. The numerical simulations generally agree with the theoretical results.

Critical weaknesses

The one-third scaling law has already been studied in previous work, including Liu et al. (2026) and Sclocchi & Wyart (2024). Although this paper extends the analysis to multiclass softmax classification and provides additional results on classification error, the main mechanism is closely related to earlier findings. I am not fully convinced that these extensions provide a sufficiently significant new contribution.

The theoretical results rely on restrictive assumptions, including Gaussian inputs, orthogonal teacher vectors, and a fixed number of classes in the high-dimensional limit.

The experiments on MNIST and Pythia show scaling exponents close to one-third, but this does not necessarily confirm the proposed boundary-layer mechanism. In particular, the Pythia results depend on a fitted error floor and the selected fitting window.

Without more direct evidence of the proposed mechanism in practical models, it is difficult to judge how widely the theoretical explanation applies.

Questions for authors
  1. Could the authors clarify the main theoretical contribution beyond Liu et al. (2026) and Sclocchi & Wyart (2024)? In particular, which new insights are specific to multiclass softmax classification?

  2. Can the authors provide more direct evidence of the boundary-layer mechanism in practical neural networks, rather than only fitting the scaling exponent? This would help establish whether the observed power laws come from the proposed mechanism.

1 comment

We thank the reviewer for the careful reading. The two questions led to the main changes in the revision.

Q1 / W1. Contribution beyond Liu et al. (2026) and Sclocchi & Wyart (2024)

Each paper establishes one of the two one-third laws, as acknowledged in the revised Introduction and Table 1 (now in Section 2).

  • Liu et al. Softmax cross-entropy with peaked soft targets under gradient flow. The student is assumed to stay aligned with the teacher, so that only its logit scale grows. This gives the one-third law for the loss. Classification error relative to the teacher's predicted class is zero by construction, and SGD noise is not included.
  • Sclocchi & Wyart. A binary perceptron with hinge loss and hard labels under online SGD. For Gaussian inputs, the test error decays with exponent one third when SGD noise dominates.

What we add:

  1. Both laws in one model, with different origins. For multiclass softmax cross-entropy with hard labels, the loss law holds with and without SGD noise. The one-third error law requires SGD noise: under gradient flow, the error decays faster than any inverse power of training time. Loss and error are controlled by two distinct boundary-layer widths, 1/D1/D and Δ/D\sqrt{\Delta}/D, where DD is the student logit scale along the teacher logits and Δ\Delta is the residual variance parameter (Section 4).
  2. Relative to Liu et al. Exact alignment is not assumed. SGD noise, which they leave open, maintains residual weight fluctuations that give the one-third error law.
  3. Relative to Sclocchi & Wyart. We treat softmax cross-entropy with KK classes in place of binary hinge loss, and obtain a law for the test loss, which they do not derive.
  4. Error scaling under a decaying learning rate. For a learning rate η∝α−γ\eta \propto \alpha^{-\gamma} with 0<γ<10 < \gamma < 1 in training time α\alpha, the error exponent increases to (2+γ)/6(2+\gamma)/6 while the loss exponent decreases to (1−γ)/3(1-\gamma)/3 (Corollary 5, Fig. 3). For the loss alone, this reduces to the rescaled time of Liu et al.

Specific to multiclass softmax cross-entropy:

  • Cross-entropy. In Theorem 3, the growth rate of the logit scale DD is proportional to π2/6+Δ\pi^2/6 + \Delta. The first term is present without SGD noise; the second is sustained by it at late times.
  • More than two classes. At leading order, only boundaries between two classes matter; near-ties of three classes are subleading. The number of classes KK therefore enters only through prefactors, and the exponents do not depend on KK.

W2. Assumptions

The orthonormal teacher assumption is motivated by the large-input-dimension limit at fixed KK: independent normalized isotropic random teacher vectors have overlaps between distinct vectors of order N−1/2N^{-1/2} (Section 3). Section 6 probes departures from the other assumptions: non-Gaussian inputs (whitened pretrained features with teacher labels, Fig. 4), correlated Gaussian inputs with bounded covariance spectra (Fig. 5), and up to K=100K = 100 classes at N=200N = 200 (Fig. 6). The revision states the limits more clearly: our results assume noiseless hard labels, and we make no claim for spectra not bounded away from zero as N→∞N\to\infty.

W3. MNIST and Pythia

We agree that an exponent alone does not confirm a mechanism. The revision reports both as exploratory fits outside assumptions (A1)-(A7) and states that these fits alone do not establish the mechanism (Section 6, Appendix C.6). For Pythia, we follow the checkpoint analysis of Liu et al. and extend it from the loss to the token top-1 error, using the heuristic time rescaling described in Appendix C.6.

Q2 / W4. Direct evidence in a trained network

We ran the test the reviewer requested on the MNIST MLP of Fig. 10 (Appendix C.6, new Fig. 11), following the test of the solvable model in Fig. 8. There is no explicit teacher in this experiment, so we use the final trained network, normalized to unit logit variance, as a reference. The gap between its largest and second-largest logits serves as a proxy for distance from the decision boundary. During training, both errors and squared gradient norms of the per-example loss concentrate at small gaps. Their median gaps shrink inversely with the current network's logit standard deviation, which we use as a proxy for DD: both roughly halve while the logit standard deviation doubles.

The errors and gradients are thus carried by examples close to the network's decision boundaries, as in the solvable model. This is evidence for the mechanism in a network with trained hidden layers. The fixed-feature theory makes no prediction for such a network, and the revised text says so.

We hope that the new experiment and the clarified contribution allow the reviewer to reconsider the assessment.

Comments on the paper

1 comment

We thank the reviewers for their reports. We have uploaded a revised PDF. This comment summarizes the changes; our individual replies address each point. Line numbers refer to the submitted version; figure and section numbers refer to the revision.

The result in plain terms. A one-layer softmax classifier, the student, is trained by online SGD on hard labels from a teacher of the same form. For large input dimension, the dynamics reduces to two coupled equations: one for the scale DD of the student logits along the teacher logits, and one for the residual variance parameter Δ\Delta. At fixed learning rate and late times, only examples in boundary layers of width 1/D1/D around the teacher's decision boundaries contribute appreciably to the gradient, and D∝α1/3D \propto \alpha^{1/3} in training time α\alpha. The test loss decays as 1/D1/D, with and without SGD noise. The classification error decays as Δ/D\sqrt{\Delta}/D; it follows the one-third law because SGD noise keeps Δ\Delta finite and nonzero. Annealing the learning rate as α−γ\alpha^{-\gamma} with 0<γ<10<\gamma<1 separates the two decay exponents: the error exponent increases to (2+γ)/6(2+\gamma)/6 and the loss exponent decreases to (1−γ)/3(1-\gamma)/3.

Changes in the revision

  • Readability (dTxN, 7BQZ). The abstract and contributions paragraph have been rewritten in plain language. Section 4 includes two explanatory paragraphs, “Boundary layers of width 1/D1/D” and “Why one third”, which explain how the boundary layers lead to the one-third laws. The symbols the reviewers asked about are now defined at first use. The class index bb, which clashed with the batch size, is now a′a'. Appendix A opens with a roadmap of the proofs.
  • Relation to prior work (gNTM, dTxN). Sclocchi & Wyart (2024) are now discussed in the Introduction as well. The comparison table has moved from the appendix to Section 2 (Table 1) and includes a column for their work.
  • Evidence in a trained network (gNTM). A new experiment on the MNIST MLP (Appendix C.6, new Fig. 11) shows that errors and gradient contributions concentrate on examples close to the network's decision boundaries, as in the solvable model.
  • Scope. The statement on correlated inputs is restricted to the covariance spectra tested. The revision also makes clear that the results assume noiseless hard labels; soft or noisy labels remain outside the present analysis.

Checks of the theory. The reports do not identify a specific error in a derivation or result. We recognize that the presentation made the analysis difficult to assess. The predicted exponents and prefactors are compared, without fitted parameters, with SGD simulations at finite input dimension across learning rates, schedules, batch sizes, and numbers of classes (Figs. 2, 3, 6, 7). We would welcome clarification of any remaining concern about a specific step or result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.