acceptodds
Under review as a conference paper at ICLR 2027

From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

Abstract

Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random features. Conditional on the representation, an exact Volterra equation separates two response components: a forcing term that propagates unresolved target error and a memory kernel that propagates stochastic-error injections. We prove that either component follows a power law if and only if its cumulative weighted spectral mass has the corresponding low-spectrum scaling; individual eigenvalues and target coefficients need not obey coordinatewise power laws. Under a joint schedule, intrinsic time controls optimization progress, while controls noise injection. Their interaction yields sharp conditions under which a schedule preserves, changes, or destroys the clean power law, together with a memory ceiling on noise reduction. The power-law random-feature model realizes this mechanism in propagation regimes with phase-dependent compute rates. Controlled nanoGPT experiments show that (1) learning-rate and batch-size schedules with matched paths are nearly equivalent in intrinsic time, (2) a forcing-memory surrogate accurately predicts loss across schedules, and (3) its fitted exponents across real-world datasets identify the regime of LLMs in map.

Primary area:
learning theory
Keywords:
scaling laws, stochastic gradient descent, spectral analysis, learning-rate and batch-size schedules, random features, LLM Pre-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_henbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This is a largely theoretical paper that studies SGD optimization of linear random feature models with power law spectral/convergence properties. The paper uses a standard representation of the risk evolution in terms of Volterra iterations with suitable kernels. The first stated result is an equivalence between asymptotic power laws in the spectral data and the kernels (Theorem 3.1). Then, the paper presents a (largely known) phase diagram of different convergence regimes corresponding to constant batch/learning rate schedules. After that, the paper analyzes non-constant, power law schedules and finds the deviation of the noisy risk trajectories from the noiseless one (Theorem 5.1). Next, results on schedule optimization and applications to LLM training are presented.

Critical strengths

The paper is an advanced study of SGD in linear noisy models. It addresses several potentially important topics related to power-law SGD convergence. Some elements of the study are new and interesting. However, the material in the present submission is poorly organized, and significance of the specific contributions is not convincing (see below).

Critical weaknesses

Poor exposition

The paper is very hard to read. It is extremely dense and tries to cover numerous topics each of which is sufficiently complex to deserve a separate extended discussion: power-law spectral and convergence laws, a phase diagram with a large number of learning regimes, non-stationary schedules and associated convergence rates, deterministic equivalents and surrogates, schedule optimization, applications to LLMs,.. None of these topics seems to me sufficiently clearly explained.

Importantly, only two results of the paper are stated as proper theorems (Theorems 3.1 and 5.1), and these theorems rely on hidden assumptions not found in the main text.

The paper includes a large 80-page appendix, and its correspondence to the main text is complicated and confusing. In particular, I struggled to find the proof of one of the two main theorems, Theorem 5.1. Its assumptions and proof seem to be scattered across multiple sections, and some genuine research of the document is needed to identify the relevant pieces.

Marginal contributions

While the paper claims multiple new contributions, I struggle to find one that would be genuinely novel, interesting, convincing and well-described.

The first claimed result is the equivalence between power laws in the cumulative distribution and kernel asymptotics (Theorem 3.1). This essentially boils down to a classical asymptotic property of the Laplace transform and is literally a century-old topic in analysis. The distribution to Laplace transform direction is known as Abelian theorems, and the opposite as Tauberian (e.g., Karamata's Tauberian theorem). Though the paper has a special setting (a joint width-time limit, particular distributions, etc), the essense of Theorem 3.1 is exactly this Abelian/Tauberian power law duality. This duality is well-known and is not even mentioned.

Several versions of the phase diagram very similar to the 3+3(+2) diagram of the paper have already appeared in the literature (in particular the cited 4+3 diagram in Paquette et al. (2024)). There are some natural variations in the setting details, in the parameterization and in what to call a distinct regime, leading to different versions. I don't quite see a major contribution in the new version.

Theorem 5.1 on schedule transformation is the second of the two theorems presented in the main text. This theorem actually looks like a reasonable new result. However, as mentioned, the exposition is too dense; the theorem itself relies on multiple hidden assumptions scattered across the appendix, and it takes much effort to even locate the proof.

The application to LLMs in section 5.4 look very superficial to me. The paper essentially just fits some artificial parametric family of risk curves to the experimental curves. It is stated that these experiments test the theory, but I don't quite see how. The experiments do not directly reveal or estimate the spectral distributions, and it is not even clear how relevant is the linear random feature model studied in the paper to the LLMs in question.

Questions for authors

N/A

4 comments

We thank for Reviewer henb's feedback especially on the presentation. We have thoroughly polished this paper, with major revisions highlighted in red.

Q1: The presentation is overly dense. Only Theorems 3.1 and 5.1 are stated in the main text, yet their assumptions are unclear. The appendix is difficult to navigate.

A1: We sincerely thank for the reviewer's writing suggestions. We have polished the main text, reorganized the appendices to follow the main text progression: spectral criteria in Appendix B, schedule analysis in Appendix C, PLRF risk bounds in Appendix D, and resource optimization in Appendix E.

Besides, we add two proof roadmaps in the appendix for better readability, see Figure 6 for Theorem 3.1, and Figure 11 for Theorem 5.1, respectively.

In our main text, we highlight Theorems 3.1 and 5.1 that answer the paper’s two central questions: when forcing and memory follow power laws, and how joint schedules change risk decay. The required assumptions, formal versions, and the proof are deferred to Appendices B.1 and C.1 for Theorems 3.1 and 5.1, respectively.


Q2: Theorem 3.1 largely reflects classical Abelian/Tauberian duality. This connection is insufficiently acknowledged. The technical novelty arising from the joint width–time setting is unclear.

A2: We acknowledge Abelian–Tauberian duality and credit the Karamata theorem in the main text.

For the joint width–time setting, fixed-width positive modes eventually decay exponentially, while larger models can contain slower modes and allow longer power-law windows. The technical difficulty is obtaining estimates that remain valid as the spectrum changes. We establish this joint-limit criterion under explicit uniform assumptions in Appendix B.1, supported by the Tauberian result in Appendix B.3.


Q3: Similar phase diagrams appear in [1]. The new classification’s contribution is unclear.

A3: The 4+3 map of [1] describes compute-optimal model size and risk with learning rate and batch size fixed, in a static sense. However, our 3+3(+2) map describes how target error decays and stochastic errors persist under different training schedules requirements, in a dynamic sense.

The SGD error comes from two parts: target error and SGD noise. We separate the phases by how fast each part decays and which part decays more slowly. This is now explained in Section 4. Time-varying learning rate and batch size schedules affect how fast SGD noise accumulate. So once we distinguish the 3+3(+2) phases, we show how to set the schedule in each phase, which is now explained in Section 5.


Q4: The relevance of the linear random-feature to LLMs is unclear. The LLM experiments do not support the theory.

A4: The proxy model, linear random features, cannot match the structure and formulation of LLMs. However, some mechanisms of the proxy model and LLMs trained by SGD can match each other, e.g., the stochastic injections, joint LR–batch scheduling effects.

Our theory is

+\int_0^T \frac{F(u)+\sigma^2}{r(u)}\\,k(T-u)\\,du, \qquad r(u)=\frac{B(u)}{\eta(u)}.$$ Here, $F(T)$ is the unresolved target error, $1/r(u)$ controls stochastic injections, and $k(T-u)$ describes how much of an earlier injection survives until $T$. We formulate the LLM modeling by assuming $F(t) \approx (1+t)^{-q_F}$ and $k(T-u) \approx (T-u)^{-q_K}$ $$L(T)=L_\infty+A_F(1+T)^{-q_F} +\int_0^T \frac{A_1(1+u)^{-q_F}+A_0}{r(u)} \bigl(1+c_K(T-u)\bigr)^{-q_K}\\,du,$$ that recovers FSL’s loss formula [2], reparameterized by the forcing and memory exponents $q_F$ and $q_K$. The terms in the LLM loss model correspond to those in the theoretical risk as follows: $A_F(1+T)^{-q_F}$ models the target error $F(T)$; $A_1(1+u)^{-q_F}+A_0$ models the injection $F(u)+\sigma^2$; and $(1+c_K(T-u))^{-q_K}$ models the memory $k(T-u)$. We retain the schedule factor $1/r(u)$ and add $L_\infty$ as the loss floor. This model is used for cross-schedule prediction: we fit its parameters on one 8-1-1 trajectory, then freeze them and change only the input ratio path $r(T)$ to predict loss under other schedules. --- In general, our LLM experiments test if this LLM model predicts loss: i) Different LR–batch factorizations of the same $r(T)$ path produce coincident loss, consistent with the model’s dependence on $T$ and $r(T)$. ii) The model fits an 8-1-1 trajectory well and predicts other LR and batch-size schedules without refitting, testing whether the same parameters remain predictive after the schedule changes. --- We thank for the reviewer's time again and hope our response addressed the reviewer's concerns. >Reference [1] Elliot Paquette, et al. "4+3 Phases of Compute-Optimal Neural Scaling Laws." NeurIPS'24. [2] Binghui Li, et al. "Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules." NeurIPS'25.

Dear Reviewer henb,

Thank you again for your review. As the discussion period draws to a close, we would appreciate any feedback on whether the revised presentation makes the main results and their assumptions easier to follow. We would also be happy to clarify any remaining questions about the theory or the experiments.

Best regards,
The Authors

I thank the authors for their clarifications and for addressing my concerns. I acknowledge that the revised paper has improved in several respects. Most notably, it is now significantly easier to navigate between the main theorem statements and their assumptions and proofs in the appendix.

Nevertheless, my overall assessment has not fundamentally changed. I still find the paper overly dense, and I remain unconvinced that its contributions offer sufficiently novel or significant insights relative to the existing literature. In recognition of the improvements, I am raising my score from 1 to 2.

Dear Reviewer henb,

Thank you for taking the time to revisit our paper and response. We appreciate your recognition of the improvements and your updated assessment. Your feedback helped us improve the paper’s presentation and organization.

Best regards,
The Authors

Reviewer_FciZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates two questions: when and why power laws emerge from learning dynamics, and how joint learning-rate/batch-size schedules affect these laws. First, it establishes conditions under which the forcing and memory components exhibit power-law decay, characterized by the low-spectrum behavior of their weighted spectral measures. Second, it shows how training schedules affect stochastic-noise accumulation and thereby determine whether the observed risk retains or changes its clean power-law exponent. Building on the forcing decay exponent qFq_F and the memory decay exponent qKq_K abstracted from the power law origin, the paper establishes a theoretical scaling-law framework, leading to a 3+3(+2) classification of scaling regimes.

Critical strengths
  1. The paper addresses the fundamental question of why and when power-law learning curves emerge, clarifying that power-law eigenvalue decay alone is neither necessary nor sufficient. This insight is important for understanding scaling laws beyond idealized spectral assumptions.
  2. The paper provides a unified framework connecting spectral properties, forcing–memory dynamics, and joint learning-rate/batch-size schedules. This offers a systematic perspective for understanding the origin and transformation of power-law learning curves.
Critical weaknesses

1. Limited novelty of the schedule-transfer insights.

The accumulation of stochastic noise through a memory kernel under varying training schedules has been studied in Li et al. (2025). The preserve/change/destroy classification and memory ceiling also appear to follow from standard asymptotic arguments.

2. Applicability of the response-based framework.

In the canonical power-law random-feature model, the proposed scaling-law framework based on the forcing decay exponent qFq_F and memory decay exponent qKq_K essentially reparameterizes the original spectral exponents. Beyond explaining the origin of power laws and accommodating irregular spectra, it remains unclear what theoretical or empirical applications this framework enables.

Questions for authors
  1. Existing scaling-law frameworks have already been extended to numerical precision and data mixing [1,2]. Does the proposed forcing–memory framework offer any advantages for studying these additional scaling dimensions or deriving new theoretical predictions? These works should also be discussed in the related work section.

References:

[1] Scaling Laws for Precision in High-Dimensional Linear Regression.

[2] Explaining Data Mixing Scaling Laws.

3 comments

We thank Reviewer FciZ for the comments and suggestions. We have addressed by adding comparisons with FSL, advantages of (qF,qK)(q_{\mathcal F},q_{\mathcal K}) as well as discussions on numerical precision and data mixing.

Q1: Comparison with FSL [1] on Section 5 how schedules change scaling laws.

A1: Different from FSL [1] using SDE, we study discrete SGD under joint learning-rate and batch-size schedules via finite-step control and different noise-suppression requirements across memory regimes.

The convolution calculations use standard asymptotic tools, but the additional work is needed to control time-varying update products and noise feedback under the stated spectral and stability conditions. Feedback changes the noisy–clean gap by at most a bounded factor, preserving its decay exponent.

Besides, FSL’s spectral assumptions place its model in IM; we also analyze LM and FB. Under our theorem’s conditions, for B(T)/η(T)≍TθB(T)/\eta(T)\asymp T^\theta, the noisy–clean gap decays when θ>0\theta>0 in IM, but requires θ>1−qK\theta>1-q_{\mathcal K} in LM because earlier noise persists longer. Thus, the same schedule can suppress noise in IM but remain insufficient in LM. FB accumulation additionally depends on model width.

We have expanded the FSL comparison in Related Work in the revised manuscript and explained these regime differences and schedule thresholds in Sections 4–5. Appendix C.1 collects the assumptions, formal statement, and proof of Theorem 5.1.


Q2: Unclear response framework (qF,qK)(q_{\mathcal F},q_{\mathcal K}) vs. (α,β)(\alpha,\beta)

A2: A spectrum may have different power-law exponents across different ranges that cannot be charecterized by (α,β)(\alpha,\beta).

For example, broken power law has two different exponents on head and tails, supported by [2] (Section 4.2.1, Figure 3) and [3] (Figure 3).

\begin{cases} j^{-2\alpha_1}, & j\le k,\\\\ k^{2(\alpha_2-\alpha_1)}j^{-2\alpha_2}, & j>k, \end{cases} \qquad \alpha_1\ne\alpha_2.$$ No single $\alpha$ describes both spectral ranges, but such case does not affect our results based on $(q_{\mathcal F},q_{\mathcal K})$. --- > **Q3:** Discussion on numerical precision and data mixing **A3:** We thank the reviewer for suggesting these connections. We have discussed both papers in Related Work. The potential benefit of the forcing–memory perspective is to distinguish changes in the learnable signal, the strength of stochastic injections, and the persistence of their effects. This provides a basis for studying how precision and data mixing interact with training schedules. **Numerical precision.** In [4], the authors study how quantization affects effective model and data sizes in sketched linear regression. Our framework could help determine whether quantization changes only the strength of stochastic errors or also their persistence and the resulting loss-decay exponent. **Data mixing.** In [5], the authors study how mixture weights affect capacity allocation and domain losses. Our spectral criteria could help analyze how mixture weights change each domain’s forcing and memory, and whether different domains require different LR–batch schedules to suppress noise. >Reference [1] Binghui Li, et al. "Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules." NeurIPS'25. [2] Elliot Paquette. "Power-Law Covariance." Lecture notes, 2026. [3] Dean A. Pospisil and Jonathan W. Pillow. "Revisiting the high-dimensional geometry of population responses in the visual cortex." PNAS'25. [4] Dechen Zhang, et al. "Scaling Laws for Precision in High-Dimensional Linear Regression." ICML'26. [5] Rui Dai and Shuran Zheng. "Explaining Data Mixing Scaling Laws." ICML'26.

Thanks for the clarification. My concerns are addressed. I keep my positive score.

Dear Reviewer FciZ,

Thank you for taking the time to read our response and for letting us know it addressed your concerns. We appreciated your suggestions on the comparison with FSL and the connections to numerical precision and data mixing.

Best regards,
The Authors

Reviewer_wkzVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how spectral properties and joint learning-rate/batch-size schedules shape learning curves. Using an online-SGD random-feature model, it characterizes power-law forcing and memory responses and develops a classification into 3+3(+2) regimes. It then examines whether the resulting schedule coordinates and response model can describe and predict language-model training dynamics, through matched-schedule experiments and cross-schedule prediction.

Critical strengths
  1. The controlled experiment in Figure 1 is particularly convincing. Different learning-rate and batch-size schedules following the same ratio path r(T)=B(T)/η(T)r(T)=B(T)/\eta(T) produce nearly overlapping loss curves when plotted against intrinsic time. The shared checkpoint and matched data consumption make this an informative test of the proposed mechanism. This is the aspect of the paper that I find most compelling.

  2. The cross-schedule predictions provide useful complementary evidence. In Figure 5 and Appendix F.3, parameters fitted on an 8-1-1 trajectory are frozen and used to predict other training trajectories. Evaluating predictions beyond the fitted trajectory strengthens the empirical case for the response model.

Critical weaknesses
  1. The relevance of the tested learning-rate range to low-loss training remains somewhat unclear. The learning rates reported for the paired SGD experiments in Appendix F.2, approximately 0.0005–0.005, appear relatively small compared with those considered in recent vanilla-SGD language-model studies [1, 2]. In particular, Glentis et al. [2] report a learning rate of 0.1. Although the settings and training stages differ, this raises the possibility that the tested schedules may be away from the region achieving the lowest loss.

  2. The connection between the broader regime classification and the LLM experiments could be clearer. The fitted LLM responses concentrate near the LM1/IM1 boundary, with qF<1q_F<1 and qK≈1q_K\approx1. I appreciate the authors' caveat that these are effective exponents over a finite training window. However, the practical interpretation of the other theoretical regimes is less apparent from the current experiments.

[1] Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful. NeurIPS, 2025.

[2] Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. Memory-Efficient LLM Pretraining via Minimalist Optimizer Design. ICML, 2026.

Questions for authors
  1. Could the authors provide a more systematic comparison with FSL [1], covering the theoretical assumptions, results, and experimental protocols? Appendix F.3 already explains some connections, and a consolidated comparison would help readers understand the relationship between the two papers.

  2. In the authors' setting, can substantially larger learning rates—for example, around 0.1—still achieve good training performance? If so, do the matched-ratio curve collapse and cross-schedule predictions continue to hold? An additional experiment addressing this would help clarify the applicability of these findings to the low-loss training region.

  3. What observable training behaviors would distinguish the theoretical regimes in language models? Beyond locating fitted responses near the LM1/IM1 boundary, can behaviors associated with other regimes also be observed experimentally?

[1] Binghui Li, Fengling Chen, Zixun Huang, Lean Wang, and Lei Wu. Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS, 2025.

4 comments

We thank Reviewer wkzV for the constructive comments and address by adding the large LR experiment and LM/IM regimes for image tasks, as well as detailed comparisons with FSL.

Q1: Can conclusion hold for a larger LR, e.g., 0.1?

A1: Yes. We have added experiments with a peak LR of 0.10.1 for both 124M and 300M models trained by SGD in Appendix G.3. Both reach substantially lower validation loss, but are still higher than advanced optimizers, e.g., Adam or Muon.

A large LR setting doesn't change our conclusion: Matched-ratio trajectories still nearly coincide at equal intrinsic time, and a surrogate fitted on one schedule still predicts the others without refitting.


Q2: Practical meaning of other regimes and classification.

A2: The regimes distinguish how long stochastic errors persist and therefore how strongly training schedules must suppress noise. In LM, earlier noise persists longer, requiring faster growth of B/ηB/\eta than in IM.

Besides, the regimes can be also split by the line qK=qFq_{\mathcal K} = q_{\mathcal F}. Beyond this line, e.g., IM1, IM2, LM1, the speed of noise forgetting is faster than that of signal learning. That means, the decay rate of the loss is dominated by signal learning.

Our LLM runs fit lie near the LM/IM boundary. We also run image-classification experiments (ResNet18 on ImageNet-32), which provides an example inside IM1 (qF=0.533q_{\mathcal F}=0.533 and qK=1.44q_{\mathcal K}=1.44), see Appendix G.4, Figure 27. This is because, image tasks in general have relatively lower noise than web data, so signal learning dominates.


Q3: A systematic comparison with FSL [1] is needed, covering theoretical assumptions, results, and experimental protocols.

A3: We have added a systematic comparison with FSL [1] in Related Work, see line 146 - 156 in the updated version. Different from FSL using SDE analysis, we directly focus on discrete SGD analysis, provide the iff condition of scaling laws, give a precise description of various learning regimes. All of them are not studied in FSL. Besides, the derived forcing-memory formula extends learning rate schedules in FSL to both learning-rate and batch-size schedules, leading to a reparameterized FSL fit-then-transfer protocol in LLM experiments.

See detailed comparisons as below.

DimensionFSLOur additional results
Dynamics and derivationCharacterizes loss evolution through a time-changed SDE.Starts from an exact loss recursion for discrete SGD and extends the Volterra/deterministic-equivalent approach of [2] to jointly varying learning rates and batch sizes.
Spectral characterizationCharacterizes loss evolution under prescribed power-law capacity and source assumptions.Gives iff criteria for power-law forcing and memory through cumulative weighted spectral masses (Theorem 3.1) without power-law capacity and source assumptions. Such iff condition of scaling laws is not studied in FSL.
Memory regimes and schedule responseIts power-law spectral setting corresponds to the IM regime in our classification.Derives distinct schedule effects across LM, IM, and FB: A slow increase in Bt/ηtB_t/\eta_t can make the noise contribution to loss decay in IM. In LM, the same increase may be insufficient because noise from earlier updates persists longer. FB exhibits a distinct, width-dependent accumulation of noise.
LLM experimental testFits an 8-1-1 trajectory and predicts other learning-rate schedules at fixed batch size.Additionally tests transfer across different learning-rate and batch-size factorizations of the same ηt/Bt\eta_t/B_t ratio path, using controlled trajectories aligned in intrinsic time.

Section 5.2 now explicitly credits FSL for the surrogate formulation and fit-then-transfer protocol, and [2] for the Volterra/deterministic-equivalent framework.

Reference

[1] Binghui Li, et al. "Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules." NeurIPS'25.

[2] Elliot Paquette, et al. "4+3 Phases of Compute-Optimal Neural Scaling Laws" NeurIPS'24.

Dear Reviewer wkzV,

Thank you again for your review. As the discussion period draws to a close, we would welcome any feedback on whether the new higher-learning-rate experiments and our comparison with FSL address your concerns. We would be happy to clarify any remaining points.

Best regards,
The Authors

Thanks for the clarifications and extra experiments. The experiments with a larger learning rate and the clearer comparison with FSL address my main concerns. I'll raise my score.

Dear Reviewer wkzV,

Thank you for taking the time to read our response and for updating your score. We appreciated your suggestions to test a higher learning rate and clarify the comparison with FSL. We’re glad these additions addressed your main concerns.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.