acceptodds
Under review as a conference paper at ICLR 2027

How Nonlinear Dynamics Shape Neural Scaling Laws and the Hyperparameter Landscape

Abstract

How does evolving curvature constrain the scaling of neural network training? Solvable models of neural scaling laws are mostly linear, with a fixed curvature, so they cannot capture how the growing curvature of real networks interacts with minibatch noise or the hyperparameter phenomena that depend on it. We study a minimal nonlinear extension, a two-layer linear network trained with SGD on a power-law task, and derive why and how fast its curvature grows, as feature learning ties the scale of the kernel to the norm of the function learned so far. The growing kernel amplifies minibatch noise until the noise level self-stabilizes at a fraction of the stochastic edge of stability set by the task, at any learning rate and even without finite-step-size effects. This feedback sets a speed limit on training, so the accelerated exponent that prior infinite-batch analyses found for feature learning is only a transient, after which the loss respects an information-theoretic bound that infinite-batch theory violates and the compute-optimal exponent returns to its lazy value. The same feedback explains why the loss is insensitive to the learning rate over a range that widens during training, why the effective learning rate follows the linear scaling rule up to a critical batch size without tuning, and why annealing yields large gains that tolerate long decays, effects that in linear models hinge on the learning rate chosen by the practitioner. SGD experiments across tasks, model sizes, batch sizes and learning rates confirm our quantitative predictions, which connect scaling laws, curvature dynamics and the hyperparameter landscape within a single tractable model.

Primary area:
optimization
Keywords:
neural scaling laws, solvable models, edge of stability

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mUDrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies neural scaling laws in a two-layer linear network trained with online minibatch SGD on a power-law task, with the goal of explaining how curvature and minibatch noise interact. The authors derive predictions for the loss, its exponent, and the compute-optimal scaling law, together with their dependence on the learning rate and batch size. Viewing feature learning as the evolution of the kernel KK, they show that the kernel scale kk grows with the norm of the learned function through k(k−1)=γ∥w∥2k(k-1)=\gamma\|w\|^2, and propose this as the mechanism behind progressive sharpening.

Since K∝ηk\mathcal{K} \propto \eta k, the growing kernel amplifies minibatch noise, which in turn limits the growth of kk and leads to self-stabilization. Using a two-variable closure, the authors predict the level at which the kernel self-stabilizes and connect it to the loss. Building on this, they explain several hyperparameter phenomena, including the widening learning-rate basin, the critical batch size, and the gains from learning-rate annealing.

Critical strengths

(1) Mechanistic and quantitative account of sharpening.

By reducing the kernel dynamics to a single scalar through k(k−1)=γ∥w∥2k(k-1)=\gamma\|w\|^2, the paper derives progressive sharpening, which prior edge-of-stability analyses take as an empirical input, from the growth of the norm of the learned function. It further predicts that minibatch noise halts sharpening at a level K⋆=1−2−β\mathcal{K}_\star = 1-2^{-\beta} that is independent of the learning rate, and validates this through data collapse across η\eta, γ\gamma, and BB.

(2) Connecting self-stabilization to loss scaling and the hyperparameter landscape.

The paper shows that self-stabilization caps the progress per sample, so the accelerated exponent of infinite-batch theory is only a transient and the compute-optimal exponent returns to its lazy value. This resolves an inconsistency in prior theory, namely that the infinite-batch rate violates a sample-complexity lower bound. The same feedback explains, within a single tractable model, the widening learning-rate basin, the linear scaling rule that emerges without tuning, and annealing gains of up to a factor of 2β2^\beta.

Critical weaknesses

Definition of feature learning.

The learned kernel retains the form K=kΛ+γk(Λ1/2w)(Λ1/2w)⊤K = k\Lambda + \frac{\gamma}{k}(\Lambda^{1/2}w)(\Lambda^{1/2}w)^\top, and because the contribution of the rank-one spike is small, the paper itself accounts for it only through a prefactor (Section A). Feature learning in this model therefore amounts essentially to a global rescaling of the initial kernel, i.e., an increase in the effective learning rate. This is qualitatively different from what is usually regarded as the core of feature learning, such as kernel--target alignment or sparse representations. Although this definition follows Bordelon et~al. (2025), the conclusions that the infinite-batch acceleration is only transient and that the compute-optimal exponent returns to its lazy value may depend heavily on this particular form of feature learning. The authors should either restrict the scope of their conclusions to this form, or discuss whether the results extend to settings in which the geometry of the kernel changes.

Questions for authors

Q1. Could the authors respond to the weakness described above?

Q2. In Figure 1(c) and Figure 3, does the behavior associated with the accelerated exponent 2β/(1+β)2\beta/(1+\beta) remain intact when the learning rate is increased (e.g., η=0.05\eta=0.05 or 0.10.1 )?

Q3. Were all experiments conducted solely on the two-layer linear network set up for the analysis in the paper?

2 comments

Thank you for the thoughtful review! Inspired by your comments, we provide additional results and clarifications here. We have uploaded a revised PDF with all changes marked in blue, with the new experiments in Appendix F, and we hope you will consider raising your score in light of our response.

Feature learning as a rescaling of the kernel (Q1). You are right that in this model feature learning acts mostly by rescaling the initial kernel, together with a rank-one spike along the learned function, which is itself a form of kernel-target alignment since ww converges to the target. This is the model in which Bordelon et al. (2025) found the accelerated exponent, and our analysis shows that the acceleration comes from the rescaling. Our conclusions therefore concern this mechanism, and we restricted their stated scope accordingly in Section 7. The constraint we identify acts on the trace of the kernel: stability under minibatch noise requires ηTr⁡K<2B\eta\operatorname{Tr}K<2B, so the growth of the kernel cannot raise the progress per sample above a fixed rate. Feature learning that changes the geometry of the kernel is subject to the same constraint, but it can spend the trace more efficiently by concentrating it on the directions the target still needs, and could in principle improve the late-time exponent beyond β\beta, up to the exponent αβ\alpha\beta of the information-theoretic bound. We added this discussion to Section 7, together with a new experiment on a two-layer tanh network (Appendix F, Fig. 14), in which the geometry of the kernel changes during training, and which still shows the self-stabilization of the noise level, the wide learning-rate basin and the large annealing gains (see Q3).

Larger learning rates (Q2). No, as the speed limit predicts and as Fig. 1c anticipates. The kernel can grow only by the factor keq=2BK⋆/(ηTr⁡Λ)k_{\rm eq}=2B\mathcal K^\star/(\eta\operatorname{Tr}\Lambda) before it self-stabilizes, and by Eq. 12 the accelerated window, from the onset of sharpening at t∼γ−1/(1−β)/ηt\sim\gamma^{-1/(1-\beta)}/\eta to t×t_\times, spans a factor (B/η)(1+β)/(1−β)(B/\eta)^{(1+\beta)/(1-\beta)} in time, which shrinks quickly as η\eta grows. At the setting of Fig. 1c (B=1B=1, γ=3\gamma=3), keq=3.2k_{\rm eq}=3.2 at η=0.05\eta=0.05 and 1.61.6 at η=0.1\eta=0.1, and the local exponent of the closure peaks at 0.260.26 and 0.240.24, below 2β/(1+β)=0.332\beta/(1+\beta)=0.33. New SGD runs (4 seeds, N=216N=2^{16}) confirm this. At η=0.05\eta=0.05 the local exponent stays between 0.220.22 and 0.270.27 from 10310^3 steps on (earlier, the loss at this learning rate is dominated by spikes from which SGD recovers), and at η=0.1\eta=0.1 SGD diverges in all seeds, since the step of the gate, γη=0.3\gamma\eta=0.3, lets minibatch noise drive qq close to zero. Larger batches restore stability since keq∝Bk_{\rm eq}\propto B, and at η=0.1\eta=0.1 with B∈{4,8}B\in\{4,8\} SGD follows the closure to within 0.030.03 in the local exponent, which peaks at 0.270.27. These runs are now in Appendix F (Fig. 15).

Experiments beyond the two-layer linear network (Q3). The experiments in the submission use the two-layer linear network, mostly with the scalar gate, and Appendix A compares the scalar and matrix parameterizations. Following the reviews, we added two sets of experiments. First, the matrix model of Eq. 2 at N=4096N=4096 (16.8M parameters) reproduces the final loss of the scalar gate to within 2.52.5\\% and its late-time exponent to within 0.0150.015 across 14 settings, with a noise level 11 to 11%11\% higher. Second, a two-layer tanh network, in which the geometry of the kernel changes during training, shows the same qualitative mechanism: its kernel grows until the noise level approaches a level that depends only weakly on η\eta, its annealed loss varies by only 1.5×1.5\times over an 8-fold range of η\eta, and its annealing gains are large at all learning rates (Appendix F, Figs. 13 and 14, and Table 1).

Thank you again for your constructive feedback! We hope you can raise your score in light of our response.

My questions and concerns about the paper have been addressed.

I now have a clearer understanding of the scope of the paper's contributions. I will maintain my score of 3.

Reviewer_fTgoOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors study a two-layer linear network trained with SGD on a power-law regression task. They find that feature learning causes progressive sharpening and that the growing kernel amplifies minibatch noise until the noise kernel norm self-stabilizes at a task-dependent level below the stochastic edge of stability. It follows from their analysis that the accelerated exponent of Bordelon et al. (2025) holds only as a transient, with the compute-optimal exponent returning to its lazy value. The same mechanism accounts for other phenomena observed in real networks, such as insensitivity to the learning rate, the gains from annealing it, and the linear scaling of its optimal value with the batch size. The predictions are confirmed by simulations.

Critical strengths

The paper identifies a single mechanism, the feedback between progressive sharpening and minibatch noise, that connects several phenomena of feature learning. It yields precise, quantitative predictions that the simulations confirm closely across a wide range of hyperparameters, and that agree qualitatively with observations in real networks. Moreover, the paper corrects a recent claim of Bordelon et al. (2025) on the accelerated scaling exponent in the feature-learning regime, and clarifies its range of validity.

Critical weaknesses

I have no critical concern. A minor point: from the text, it is unclear whether the scalar-gate approximation covers the regime where the paper’s results are measured. The key experiments are run only on this model, which matches the matrix parameterization to first order in the learning rate. While the approximation is supported by the kernel traces of fig. 7, the authors could also compare some of their predictions with experiments on the matrix model of eq. 2, which is closer to real networks and to the setting of the prior work (Bordelon et al., 2025) the paper revisits. It would also help to clarify the approximation’s range of validity across learning phases and hyperparameters.

Questions for authors

See Weaknesses. Do simulations of the matrix model of eq. 2 confirm the paper's predictions (e.g. the plateau level or the late-time exponent)? I would also suggest moving a short summary of the scalar-gate approximation and its limits into the main text.

1 comment

Thank you for the supportive review and the thoughtful suggestions! We address your questions below. We have uploaded a revised PDF with all changes marked in blue, and the new results are in Appendix F.

Do simulations of the matrix model confirm the predictions? Yes. We trained the matrix model of Eq. 2 with N=4096N=4096 (16.8M parameters) alongside the scalar gate on identical minibatches for 14 settings, 12 of them from the noise-level experiment of Fig. 2c, each run for at least 4040 plateau times. The table compares the self-stabilized noise level, the late-time exponent (fit from 10 tp10\,t_p to the end of training) and the final loss, with the closure solved at the same NN.

α\alphaβ\betaBBη\etaK\mathcal K, closureK\mathcal K, scalar SGDηTr⁡K/2B\eta\operatorname{Tr}K/2B, matrix / scalarexponent, closureexponent, scalarexponent, matrixfinal loss, matrix / scalar
20.410.0490.2600.2621.070.450.450.450.997
20.510.0590.3070.3061.090.540.550.540.992
20.610.0690.3540.3441.110.660.670.670.985
30.210.0180.1380.1381.010.220.220.220.999
30.210.0360.1360.1361.020.210.210.210.998
30.310.0520.1900.1911.030.310.310.310.995
30.320.0520.1910.1901.020.310.320.310.997
30.410.0340.2440.2421.030.410.420.420.995
30.410.0670.2430.2461.050.410.410.400.993
30.420.0670.2430.2431.030.410.420.420.994
30.440.0670.2440.2421.020.410.420.420.997
30.480.0670.2450.2451.010.420.430.420.998
30.510.0810.2930.2911.070.510.500.500.985
30.610.0940.3400.3321.080.610.610.600.976

Here K\mathcal K is the noise kernel norm at the plateau, which the closure at N=4096N=4096 predicts slightly above 1−2−β1-2^{-\beta} for α=2\alpha=2 because of the finite-NN correction of Appendix D. The matrix model reproduces the loss of the scalar gate to within 9%9\% over the whole trajectory and 2.5%2.5\% at the end of training, and its late-time exponent to within 0.0150.015, and both follow the closure. Its noise level ηTr⁡K/2B\eta\operatorname{Tr}K/2B lies 11 to 11%11\% above the scalar gate's. This difference is second order in η\eta, since the first-order dynamics of MM are identical in the two models (Eq. 7), and at second order minibatch noise gives QQ⊤QQ^\top components outside the form kI+(γ/k)ww⊤kI+(\gamma/k)ww^\top that the scalar gate cannot represent.

Range of validity of the scalar-gate approximation. The two parameterizations share the same first-order dynamics in all phases of training, so their differences come from second-order terms. Over the range we tested, these terms affect the noise level and leave the loss nearly unchanged. During sharpening, the kernel traces agree to within 22 to 10%10\% (Fig. 7), and the largest loss differences over the trajectory (9%9\%) also occur early, before the plateau. After self-stabilization, the offset in the noise level grows with the noise level itself, from 1%1\% at K=0.14\mathcal K=0.14 to 11%11\% at K=0.35\mathcal K=0.35, rather than with the batch size: for α=3\alpha=3, β=0.4\beta=0.4 and η=0.067\eta=0.067, we ran B∈{1,2,4,8}B\in\{1,2,4,8\} across Bcrit≈5B_{\rm crit}\approx5, up to ηkλ1=1.2\eta k\lambda_1=1.2 at B=8B=8, and the two models' final losses agree to within 1%1\% at all four batch sizes. The two models also agree at both learning rates we ran for α=3\alpha=3 and β∈{0.2,0.4}\beta\in\{0.2,0.4\}. These results and this discussion are now in Appendix F (Fig. 13 and Table 1), referenced from Appendix A.

Summary of the approximation in the main text. We agree, and added to Section 3 a short summary stating that the two parameterizations share the same kernel dynamics to first order in η\eta, that they differ at second order mainly in the noise level, which leaves the loss and its exponent nearly unchanged, and pointing to the new matrix-model results.

Thank you again for your feedback. We are happy to answer any further questions.

Reviewer_3pC8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a changing kernel and minibatch noise jointly affect scaling laws in a two-layer linear network. Main finding: kernel growth amplifies noise, which eventually stops further growth and removes the faster loss decay from feature learning. The same model also explains reduced sensitivity to learning rate, batch-size scaling, and gains from learning-rate decay.

Critical strengths
  1. A useful connection between scaling laws and training dynamics. In particular, a clear explanation of why the faster rate from infinite-batch analysis may not last under finite-batch SGD.
  2. Several testable predictions from one simple model, including the noise level, crossover time, and learning-rate sensitivity. Broad numerical checks across task spectra, model sizes, and training settings.
Critical weaknesses
  1. Several uncontrolled approximations behind the main results. These include replacing most kernel evolution with a single scale, assuming fast adjustment of the noise covariance, and dropping higher-order update terms. Appendix C reports a breakdown of the norm relation after stabilization, precisely the regime central to the paper. Clearer conditions for the approximations needed.
  2. Limited evidence beyond the chosen model. Gaussian inputs, direct access to the top eigenfeatures, and mostly a scalar-gated linear predictor. Strong agreement within this setting, but no tests showing that the same mechanism explains learning-rate robustness or annealing gains in nonlinear networks.
  3. Overstated comparison with the sample-complexity bound. The bound in Appendix B averages over a Gaussian target prior, whereas the main setup specifies fixed target coefficients. Also, applying an infinite-batch rate at finite batch size does not establish a flaw in the original infinite-batch result. The distinction needs clearer wording.
  4. The compute-optimal claim depends on additional assumptions. Compute counted as NBtNBt, appropriate for the scalar model but not the dense matrix model with N2N^2 parameters. The finite-size loss also treated as an additive term, despite its effect on gradient noise. A clearer scope for the claimed exponent needed.
Questions for authors

N/A

2 comments

Thank you for the careful and constructive review! Inspired by your comments, we ran new experiments on the matrix model of Eq. 2 and on a nonlinear network, and we clarify the conditions and scope of our results below. We have uploaded a revised PDF with all changes marked in blue, with the new experiments in Appendix F, and we hope you will consider raising your score in light of our response.

Conditions for the approximations. We added to Appendix C the condition of validity of each approximation.

  1. Kernel reduction. M=kI+(γ/k)ww⊤M=kI+(\gamma/k)ww^\top is exact to first order in η\eta, since the first-order update (Eq. 5) preserves it for any sequence of minibatch gradients, and we quantify the second-order corrections below.
  2. Adiabatic noise covariance. The top modes, which carry nearly all of Tr⁡Σ\operatorname{Tr}\Sigma, relax within 1/(ηkλ1)1/(\eta k\lambda_1) steps, so the condition is ηkt≫1/λ1\eta kt\gg1/\lambda_1, which holds throughout the power-law phase of the loss.
  3. Lazy clock. The spike's share of the loss-decrease rate falls to a few percent once the kernel has grown (Fig. 7d).
  4. Discrete time. The closure keeps the exact discrete-time noise covariance for Gaussian features and remains accurate up to the stability limit (1+1/B)ηkλ1<2(1+1/B)\eta k\lambda_1<2, including above BcritB_{\rm crit} (Fig. 1a).

The norm relation after self-stabilization. k(k−1)=γ∥w∥2k(k-1)=\gamma\|w\|^2 is conserved by the first-order dynamics. At the plateau the first-order drive of the kernel vanishes (G=Tr⁡ΣG=\operatorname{Tr}\Sigma), so second-order terms become the leading change of the conserved quantity and ∥w∥2\|w\|^2 keeps growing at fixed kk, which is the departure in Appendix C (now explained there). These terms barely move the plateau: in the equation for kk, they are smaller than the noise term Tr⁡Σ\operatorname{Tr}\Sigma by the spike's share of Tr⁡K\operatorname{Tr}K, which we measure at 11 to 6%6\%. The norm relation enters the closure only through Eq. 10 and the prefactor of Eq. 8, which matter only during growth, while the plateau is the zero of G−Tr⁡ΣG-\operatorname{Tr}\Sigma. Consistently, the measured noise level agrees with the closure to within 3%3\% in all new runs below.

Evidence beyond the scalar-gated model.

  1. Matrix model (Eq. 2). With N=4096N=4096 (16.8M parameters) and 14 settings (α∈{2,3}\alpha\in\{2,3\}, β\beta from 0.20.2 to 0.60.6, BB from 11 to 88), the matrix model reproduces the final loss of the scalar gate to within 2.5%2.5\% and its late-time exponent to within 0.0150.015, with a noise level 11 to 11%11\% higher, a second-order effect (Appendix F, Fig. 13 and Table 1).

  2. Nonlinear network. We trained a two-layer tanh network f(x)=u⊤tanh⁡(Wx)/Hf(x)=u^\top\tanh(Wx)/\sqrt H (H=4096H=4096, Gaussian W(0)W(0), u(0)=0u(0)=0, learning rate γη\gamma\eta on WW) on the task with α=2\alpha=2, β=0.5\beta=0.5, N=2048N=2048, B=1B=1, either rich (γ=128\gamma=128) or lazy (γ=0\gamma=0). Its features are not the eigenfeatures of the task, and its kernel changes its geometry during training. Rich training shows the same mechanism (Appendix F, Fig. 14). The kernel grows until ηTr⁡K/2B\eta\operatorname{Tr}K/2B approaches a level nearly independent of η\eta, so the learned kernel scales as 1/η1/\eta (η\eta times its trace ends between 0.780.78 and 0.860.86 for η∈[0.2,0.8]\eta\in[0.2,0.8]), while lazy training's noise level grows in proportion to η\eta. After annealing, the rich loss varies by only 1.5×1.5\times over η∈[0.1,0.8]\eta\in[0.1,0.8], up to its stability limit, while the lazy loss keeps improving up to its largest stable η\eta and still ends 4.7×4.7\times higher. Annealing gains are 1.9×1.9\times to 7.5×7.5\times at all η\eta in rich training, but stay below 1.25×1.25\times in lazy training until η\eta approaches its stochastic edge. We do not claim the quantitative values 1−2−β1-2^{-\beta} and 2β2^\beta for this network.

    η=0.1\eta=0.1η=0.2\eta=0.2η=0.4\eta=0.4η=0.8\eta=0.8η=1.6\eta=1.6η=3.2\eta=3.2
    rich, noise level0.280.360.420.49unstablenot run
    rich, annealed loss (10−410^{-4})4.23.53.45.1unstablenot run
    rich, annealing gain1.92.74.07.5unstablenot run
    lazy, noise level0.020.030.070.140.280.55
    lazy, annealed loss (10−410^{-4})38.929.923.619.517.116.0
    lazy, annealing gain1.01.01.11.21.63.4

    Here the noise level is ηTr⁡K/2B\eta\operatorname{Tr}K/2B averaged over the last 20%20\% of 2202^{20} steps, and the annealing gain is the ratio between the constant-rate and annealed losses (linear decay over the last 20%20\%).

Comparison with the sample-complexity bound.

  1. Gaussian prior vs. fixed target. The infinite-batch learning curve is a sum over many modes and self-averages. At the setting of Fig. 4b, the curves of 8 targets drawn from the prior agree with the fixed-target curve to within 1%1\% for D≥103D\ge10^3, and their average falls below the average-case bound from D≈1.2×103D\approx1.2\times10^3 on, reaching 0.550.55 of it at D=3×104D=3\times10^4. We now state the bound and this check explicitly as an average over the prior in Appendix B.
  2. Infinite vs. finite batch. We agree and reworded the paper accordingly. The infinite-batch result is correct at infinite batch size, where the bound does not apply, and what fails is its use at finite batch size, where minibatch noise was argued to be subleading. In the abstract and Section 5, we replaced "a bound that infinite-batch theory violates" with "a bound that the infinite-batch rate would violate at finite batch size".

Scope of the compute-optimal claim.

  1. Compute. We follow Bordelon et al. (2025), who count C=NtC=Nt, to compare with their rich exponent on equal terms. The return to the lazy exponent holds for any cost c(N)c(N) per sample: after self-stabilization, the loss Lcap(D)+L∞(N)\mathcal L_{\rm cap}(D)+\mathcal L_\infty(N) is the same function of (N,D)(N,D) as for tuned lazy training up to constants, so the two share the exponent at fixed C=c(N)DC=c(N)D. For the dense matrix model (c=N2c=N^2) both give αβ/(α+2)\alpha\beta/(\alpha+2), and we now state this scope in Section 5.
  2. Finite-size loss and gradient noise. The unlearnable part acts as label noise of variance 2L∞(N)2\mathcal L_\infty(N), which lowers the plateau to G/(2L0+2L∞+G)G/(2\mathcal L_0+2\mathcal L_\infty+G). Along the frontier, L∞(N⋆)\mathcal L_\infty(N^\star) is a fixed fraction of the learnable loss, so this changes the prefactor and not the exponent. Our compute-optimal experiments include this effect (V=32NV=32N) and follow the predicted exponent (Figs. 4c, 12c), and we added this derivation to Appendix D.

Thank you again for your constructive feedback! We hope you can raise your score in light of our response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.