- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the convergence behavior of a softmax classifier trained with cross-entropy loss and online SGD. The goal is to derive the so-call power-law of the convergence speed. Specifically, the authors consider a single-layer teacher with orthogonal linear neurons producing deterministic hard labels (which can be seen as a special case of [Liu et al. 2026] with zero temperature). The student network, also having a single layer, is trained with cross-entropy loss. While the prior work in [Liu et al. 2026] has shown the power-law for the training (cross-entropy) loss assuming perfect alignment of teacher and student weights, this paper aims to further study the scaling of the test error, which is meaningful when the above weight-alignment does not occur due to random initialization. The authors study the limit when the number of student neurons approach infinity. The main contribution appears to be identifying a boundary-layer mechanism: as training progresses, only data samples concentrated near the teacher's decision boundaries contribute to learning, while stochastic updates maintain a nonzero residual variance. By analyzing the resulting optimization dynamics, the paper derives a (t^{-1/3}) convergence rate for both the training loss and test error. Experiments with deep networks show comparable scaling.
Critical strengths
- The so-called boundary layer mechanism could be new. Specifically, as training progresses, only data concentrated near the teacher's decision boundaries contribute to learning.
Critical weaknesses
-
The paper is difficult to read. The authors claims that the central observation is identifying the boundary layer mechanism. However, there is no clean explanation on what it is after the introduction. Similarly, while several proofs are presented, it is difficult for the readers to understand how they together show the effect of the boundary layer, e.g., how it leads to the power law. Perhaps the proofs are produced by AI; I think some efforts are needed in order to present the proof and intuitions in a way that readers can understand.
-
Sometimes, results are presented with notations not clearly defined. For example, what are and in Lemma 1?
Questions for authors
-
In Assumption (A7), could you define precisely what "permutation-symmetric" means?
-
What are and in Lemma 1?
-
In Lemma 2, what does "fixed finite " refer to?
-
What do the pair and refer to above (11)? Even further above, why is the boundary layer of width ?
2 comments
I thank the authors for the response, which clarifies some of the confusions. However, I think my concern on the readability of the paper remains. Thus, at this point I will keep my score.
We thank the reviewer for the specific questions.
Two corrections to the summary. (i) The limit we take is the input dimension N→∞ at a fixed number K of classes (A1, A6). The student has K output units throughout. (ii) The misalignment that controls the late-time error is not a remnant of the random initialization. It is maintained by SGD noise: without this noise, the error decays faster than any inverse power of training time (Section 4).
Q1. “Permutation-symmetric” in (A7). All classes are statistically equivalent: the overlaps depend only on whether two class indices coincide, Ja⋅Ta′/N=Rδaa′+S(1−δaa′) and Ja⋅Ja′/N=Qδaa′+C(1−δaa′). In the limit (A6), the dynamics preserves this form. The overlaps from independent, identically distributed student initialization approach it as the input dimension grows. (A7) now states these equations.
Q2 / W2. g1 and h1 in Lemma 1. They are the class-1 values of two quantities defined a few lines above the lemma (lines 240 and 246): ha is the centered student logit of class a (Eq. 8), and ga=paT−pa is the teacher's one-hot label minus the student's softmax output. By (A7), every class gives the same averages, which we write for class 1; the revised text says so.
Q3. “Fixed finite b” in Lemma 2. b is the minibatch size, introduced in (A5) as “fixed finite batch size b” (line 196). Lemma 2 now repeats the words “batch size”.
Q4. The pair a, b above Eq. (11), and the width O(D−1). In the submitted version, a and b label two classes at the decision boundary between them, where their teacher logits coincide, ua=ub. The letter b clashed with the batch size, so this index is now a′. The width follows from Eq. (8). Let G be the gap between the two largest teacher logits for an example. The corresponding student logit gap is DG plus noise of size Δ. For bounded Δ, appreciable deviations from the one-hot label occur mainly when DG is O(1), hence G=O(1/D).
W1. The mechanism, and how the proofs combine
The explanation in Section 4 and Fig. 1 (right) was not gathered in one place. The revision gives the chain of reasoning in two paragraphs, “Boundary layers of width 1/D” and “Why one third”:
Theorem 3 and Corollary 4 make these steps precise and supply the prefactors. Appendix A now opens with a roadmap: A.1 reduces the dynamics to (D,Δ), A.2 shows that D diverges at bounded Δ, A.3 evaluates the averages that drive the flow, and A.4 and A.5 integrate the result.
On the use of AI
Our AI-use statement discloses that AI tools assisted in formulating and writing proofs. All derivations and proofs were checked step by step by the authors, and the revised statement says so. The results can also be tested: the predicted exponents and prefactors agree, without fitted parameters, with SGD simulations of the student (Figs. 2, 3, 6, 7).
Soundness
We recognize that the presentation made the analysis difficult to assess. The review does not identify a specific erroneous step. We would welcome clarification of any remaining technical concern and will address it.