- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies neural scaling laws in a two-layer linear network trained with online minibatch SGD on a power-law task, with the goal of explaining how curvature and minibatch noise interact. The authors derive predictions for the loss, its exponent, and the compute-optimal scaling law, together with their dependence on the learning rate and batch size. Viewing feature learning as the evolution of the kernel , they show that the kernel scale grows with the norm of the learned function through , and propose this as the mechanism behind progressive sharpening.
Since , the growing kernel amplifies minibatch noise, which in turn limits the growth of and leads to self-stabilization. Using a two-variable closure, the authors predict the level at which the kernel self-stabilizes and connect it to the loss. Building on this, they explain several hyperparameter phenomena, including the widening learning-rate basin, the critical batch size, and the gains from learning-rate annealing.
Critical strengths
(1) Mechanistic and quantitative account of sharpening.
By reducing the kernel dynamics to a single scalar through , the paper derives progressive sharpening, which prior edge-of-stability analyses take as an empirical input, from the growth of the norm of the learned function. It further predicts that minibatch noise halts sharpening at a level that is independent of the learning rate, and validates this through data collapse across , , and .
(2) Connecting self-stabilization to loss scaling and the hyperparameter landscape.
The paper shows that self-stabilization caps the progress per sample, so the accelerated exponent of infinite-batch theory is only a transient and the compute-optimal exponent returns to its lazy value. This resolves an inconsistency in prior theory, namely that the infinite-batch rate violates a sample-complexity lower bound. The same feedback explains, within a single tractable model, the widening learning-rate basin, the linear scaling rule that emerges without tuning, and annealing gains of up to a factor of .
Critical weaknesses
Definition of feature learning.
The learned kernel retains the form , and because the contribution of the rank-one spike is small, the paper itself accounts for it only through a prefactor (Section A). Feature learning in this model therefore amounts essentially to a global rescaling of the initial kernel, i.e., an increase in the effective learning rate. This is qualitatively different from what is usually regarded as the core of feature learning, such as kernel--target alignment or sparse representations. Although this definition follows Bordelon et~al. (2025), the conclusions that the infinite-batch acceleration is only transient and that the compute-optimal exponent returns to its lazy value may depend heavily on this particular form of feature learning. The authors should either restrict the scope of their conclusions to this form, or discuss whether the results extend to settings in which the geometry of the kernel changes.
Questions for authors
Q1. Could the authors respond to the weakness described above?
Q2. In Figure 1(c) and Figure 3, does the behavior associated with the accelerated exponent remain intact when the learning rate is increased (e.g., or )?
Q3. Were all experiments conducted solely on the two-layer linear network set up for the analysis in the paper?
2 comments
My questions and concerns about the paper have been addressed.
I now have a clearer understanding of the scope of the paper's contributions. I will maintain my score of 3.
Thank you for the thoughtful review! Inspired by your comments, we provide additional results and clarifications here. We have uploaded a revised PDF with all changes marked in blue, with the new experiments in Appendix F, and we hope you will consider raising your score in light of our response.
Feature learning as a rescaling of the kernel (Q1). You are right that in this model feature learning acts mostly by rescaling the initial kernel, together with a rank-one spike along the learned function, which is itself a form of kernel-target alignment since w converges to the target. This is the model in which Bordelon et al. (2025) found the accelerated exponent, and our analysis shows that the acceleration comes from the rescaling. Our conclusions therefore concern this mechanism, and we restricted their stated scope accordingly in Section 7. The constraint we identify acts on the trace of the kernel: stability under minibatch noise requires ηTrK<2B, so the growth of the kernel cannot raise the progress per sample above a fixed rate. Feature learning that changes the geometry of the kernel is subject to the same constraint, but it can spend the trace more efficiently by concentrating it on the directions the target still needs, and could in principle improve the late-time exponent beyond β, up to the exponent αβ of the information-theoretic bound. We added this discussion to Section 7, together with a new experiment on a two-layer tanh network (Appendix F, Fig. 14), in which the geometry of the kernel changes during training, and which still shows the self-stabilization of the noise level, the wide learning-rate basin and the large annealing gains (see Q3).
Larger learning rates (Q2). No, as the speed limit predicts and as Fig. 1c anticipates. The kernel can grow only by the factor keq=2BK⋆/(ηTrΛ) before it self-stabilizes, and by Eq. 12 the accelerated window, from the onset of sharpening at t∼γ−1/(1−β)/η to t×, spans a factor (B/η)(1+β)/(1−β) in time, which shrinks quickly as η grows. At the setting of Fig. 1c (B=1, γ=3), keq=3.2 at η=0.05 and 1.6 at η=0.1, and the local exponent of the closure peaks at 0.26 and 0.24, below 2β/(1+β)=0.33. New SGD runs (4 seeds, N=216) confirm this. At η=0.05 the local exponent stays between 0.22 and 0.27 from 103 steps on (earlier, the loss at this learning rate is dominated by spikes from which SGD recovers), and at η=0.1 SGD diverges in all seeds, since the step of the gate, γη=0.3, lets minibatch noise drive q close to zero. Larger batches restore stability since keq∝B, and at η=0.1 with B∈{4,8} SGD follows the closure to within 0.03 in the local exponent, which peaks at 0.27. These runs are now in Appendix F (Fig. 15).
Experiments beyond the two-layer linear network (Q3). The experiments in the submission use the two-layer linear network, mostly with the scalar gate, and Appendix A compares the scalar and matrix parameterizations. Following the reviews, we added two sets of experiments. First, the matrix model of Eq. 2 at N=4096 (16.8M parameters) reproduces the final loss of the scalar gate to within 2.5 and its late-time exponent to within 0.015 across 14 settings, with a noise level 1 to 11% higher. Second, a two-layer tanh network, in which the geometry of the kernel changes during training, shows the same qualitative mechanism: its kernel grows until the noise level approaches a level that depends only weakly on η, its annealed loss varies by only 1.5× over an 8-fold range of η, and its annealing gains are large at all learning rates (Appendix F, Figs. 13 and 14, and Table 1).
Thank you again for your constructive feedback! We hope you can raise your score in light of our response.