- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This is a largely theoretical paper that studies SGD optimization of linear random feature models with power law spectral/convergence properties. The paper uses a standard representation of the risk evolution in terms of Volterra iterations with suitable kernels. The first stated result is an equivalence between asymptotic power laws in the spectral data and the kernels (Theorem 3.1). Then, the paper presents a (largely known) phase diagram of different convergence regimes corresponding to constant batch/learning rate schedules. After that, the paper analyzes non-constant, power law schedules and finds the deviation of the noisy risk trajectories from the noiseless one (Theorem 5.1). Next, results on schedule optimization and applications to LLM training are presented.
Critical strengths
The paper is an advanced study of SGD in linear noisy models. It addresses several potentially important topics related to power-law SGD convergence. Some elements of the study are new and interesting. However, the material in the present submission is poorly organized, and significance of the specific contributions is not convincing (see below).
Critical weaknesses
Poor exposition
The paper is very hard to read. It is extremely dense and tries to cover numerous topics each of which is sufficiently complex to deserve a separate extended discussion: power-law spectral and convergence laws, a phase diagram with a large number of learning regimes, non-stationary schedules and associated convergence rates, deterministic equivalents and surrogates, schedule optimization, applications to LLMs,.. None of these topics seems to me sufficiently clearly explained.
Importantly, only two results of the paper are stated as proper theorems (Theorems 3.1 and 5.1), and these theorems rely on hidden assumptions not found in the main text.
The paper includes a large 80-page appendix, and its correspondence to the main text is complicated and confusing. In particular, I struggled to find the proof of one of the two main theorems, Theorem 5.1. Its assumptions and proof seem to be scattered across multiple sections, and some genuine research of the document is needed to identify the relevant pieces.
Marginal contributions
While the paper claims multiple new contributions, I struggle to find one that would be genuinely novel, interesting, convincing and well-described.
The first claimed result is the equivalence between power laws in the cumulative distribution and kernel asymptotics (Theorem 3.1). This essentially boils down to a classical asymptotic property of the Laplace transform and is literally a century-old topic in analysis. The distribution to Laplace transform direction is known as Abelian theorems, and the opposite as Tauberian (e.g., Karamata's Tauberian theorem). Though the paper has a special setting (a joint width-time limit, particular distributions, etc), the essense of Theorem 3.1 is exactly this Abelian/Tauberian power law duality. This duality is well-known and is not even mentioned.
Several versions of the phase diagram very similar to the 3+3(+2) diagram of the paper have already appeared in the literature (in particular the cited 4+3 diagram in Paquette et al. (2024)). There are some natural variations in the setting details, in the parameterization and in what to call a distinct regime, leading to different versions. I don't quite see a major contribution in the new version.
Theorem 5.1 on schedule transformation is the second of the two theorems presented in the main text. This theorem actually looks like a reasonable new result. However, as mentioned, the exposition is too dense; the theorem itself relies on multiple hidden assumptions scattered across the appendix, and it takes much effort to even locate the proof.
The application to LLMs in section 5.4 look very superficial to me. The paper essentially just fits some artificial parametric family of risk curves to the experimental curves. It is stated that these experiments test the theory, but I don't quite see how. The experiments do not directly reveal or estimate the spectral distributions, and it is not even clear how relevant is the linear random feature model studied in the paper to the LLMs in question.
Questions for authors
N/A
4 comments
Dear Reviewer henb,
Thank you again for your review. As the discussion period draws to a close, we would appreciate any feedback on whether the revised presentation makes the main results and their assumptions easier to follow. We would also be happy to clarify any remaining questions about the theory or the experiments.
Best regards,
The Authors
I thank the authors for their clarifications and for addressing my concerns. I acknowledge that the revised paper has improved in several respects. Most notably, it is now significantly easier to navigate between the main theorem statements and their assumptions and proofs in the appendix.
Nevertheless, my overall assessment has not fundamentally changed. I still find the paper overly dense, and I remain unconvinced that its contributions offer sufficiently novel or significant insights relative to the existing literature. In recognition of the improvements, I am raising my score from 1 to 2.
Dear Reviewer henb,
Thank you for taking the time to revisit our paper and response. We appreciate your recognition of the improvements and your updated assessment. Your feedback helped us improve the paper’s presentation and organization.
Best regards,
The Authors
We thank for Reviewer henb's feedback especially on the presentation. We have thoroughly polished this paper, with major revisions highlighted in red.
A1: We sincerely thank for the reviewer's writing suggestions. We have polished the main text, reorganized the appendices to follow the main text progression: spectral criteria in Appendix B, schedule analysis in Appendix C, PLRF risk bounds in Appendix D, and resource optimization in Appendix E.
Besides, we add two proof roadmaps in the appendix for better readability, see Figure 6 for Theorem 3.1, and Figure 11 for Theorem 5.1, respectively.
In our main text, we highlight Theorems 3.1 and 5.1 that answer the paper’s two central questions: when forcing and memory follow power laws, and how joint schedules change risk decay. The required assumptions, formal versions, and the proof are deferred to Appendices B.1 and C.1 for Theorems 3.1 and 5.1, respectively.
A2: We acknowledge Abelian–Tauberian duality and credit the Karamata theorem in the main text.
For the joint width–time setting, fixed-width positive modes eventually decay exponentially, while larger models can contain slower modes and allow longer power-law windows. The technical difficulty is obtaining estimates that remain valid as the spectrum changes. We establish this joint-limit criterion under explicit uniform assumptions in Appendix B.1, supported by the Tauberian result in Appendix B.3.
A3: The 4+3 map of [1] describes compute-optimal model size and risk with learning rate and batch size fixed, in a static sense. However, our 3+3(+2) map describes how target error decays and stochastic errors persist under different training schedules requirements, in a dynamic sense.
The SGD error comes from two parts: target error and SGD noise. We separate the phases by how fast each part decays and which part decays more slowly. This is now explained in Section 4. Time-varying learning rate and batch size schedules affect how fast SGD noise accumulate. So once we distinguish the 3+3(+2) phases, we show how to set the schedule in each phase, which is now explained in Section 5.
A4: The proxy model, linear random features, cannot match the structure and formulation of LLMs. However, some mechanisms of the proxy model and LLMs trained by SGD can match each other, e.g., the stochastic injections, joint LR–batch scheduling effects.
Our theory is
+\int_0^T \frac{F(u)+\sigma^2}{r(u)}\\,k(T-u)\\,du, \qquad r(u)=\frac{B(u)}{\eta(u)}.$$ Here, $F(T)$ is the unresolved target error, $1/r(u)$ controls stochastic injections, and $k(T-u)$ describes how much of an earlier injection survives until $T$. We formulate the LLM modeling by assuming $F(t) \approx (1+t)^{-q_F}$ and $k(T-u) \approx (T-u)^{-q_K}$ $$L(T)=L_\infty+A_F(1+T)^{-q_F} +\int_0^T \frac{A_1(1+u)^{-q_F}+A_0}{r(u)} \bigl(1+c_K(T-u)\bigr)^{-q_K}\\,du,$$ that recovers FSL’s loss formula [2], reparameterized by the forcing and memory exponents $q_F$ and $q_K$. The terms in the LLM loss model correspond to those in the theoretical risk as follows: $A_F(1+T)^{-q_F}$ models the target error $F(T)$; $A_1(1+u)^{-q_F}+A_0$ models the injection $F(u)+\sigma^2$; and $(1+c_K(T-u))^{-q_K}$ models the memory $k(T-u)$. We retain the schedule factor $1/r(u)$ and add $L_\infty$ as the loss floor. This model is used for cross-schedule prediction: we fit its parameters on one 8-1-1 trajectory, then freeze them and change only the input ratio path $r(T)$ to predict loss under other schedules. --- In general, our LLM experiments test if this LLM model predicts loss: i) Different LR–batch factorizations of the same $r(T)$ path produce coincident loss, consistent with the model’s dependence on $T$ and $r(T)$. ii) The model fits an 8-1-1 trajectory well and predicts other LR and batch-size schedules without refitting, testing whether the same parameters remain predictive after the schedule changes. --- We thank for the reviewer's time again and hope our response addressed the reviewer's concerns. >Reference [1] Elliot Paquette, et al. "4+3 Phases of Compute-Optimal Neural Scaling Laws." NeurIPS'24. [2] Binghui Li, et al. "Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules." NeurIPS'25.