- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Checkpoint averaging along a single training trajectory is increasingly used in LLM pretraining to approximate terminal learning-rate decay, estimate annealed performance during training, and support horizon-free schedules. Classical averaging theory primarily characterizes final convergence rates. It does not fully explain when merging improves performance, how its effect depends on the learning-rate schedule, or how to choose among averaging profiles with the same asymptotic rate.
To address these challenges, this paper proposes Model-Merging Functional Scaling Laws (MM-FSL), extending the Functional Scaling Law framework (FSL; Li et al., 2025a; Li et al., 2026) to checkpoint averaging in power-law kernel regression. Specifically, the authors:
- derive exact transformations of the signal-learning function and noise-forgetting kernel under fractional averaging;
- show that merging increases signal error, retains more early noise, and suppresses late noise;
- characterize risk-optimal tail and power-averaging windows after jointly tuning the learning rate;
- interpret power averaging as a dynamic EMA with a growing window;
- prove that checkpoints suffice to preserve the noise-reduction rate of continuous averaging.
The authors evaluate their approach using controlled SGD experiments on Gaussian power-law kernel regression and 124M-parameter LLaMA-style models trained on FineWeb-Edu. The comparisons include the last iterate, fixed EMA, uniform merging, tail averaging, dynamic EMA, and constant, cosine, and WSD learning-rate schedules.
Critical strengths
S1. Important and timely research problem
Checkpoint averaging is increasingly used in LLM pretraining, as illustrated by methods such as LAWA, WSM, WSD-based training, and the use of EMA in the DeepSeek-V3 Technical Report. However, the interaction between averaging and learning-rate scheduling remains insufficiently understood. The paper addresses a meaningful gap between classical averaging theory and current LLM training practice.
S2. Clean theoretical characterization
The key technical insight is that the merged noise kernel depends separately on evaluation time and noise-injection time. This distinction enables the paper to identify three different effects of merging: increased signal error, increased retention of early noise, and suppression of late noise.
The resulting criterion—that merging helps when net noise reduction exceeds signal cost—is intuitive and useful. The risk-optimal window analysis also goes beyond convergence rates by resolving profile-dependent constants within averaging families that share the same asymptotic rate.
S3. Strong controlled validation
The kernel-regression experiments test several distinct predictions, including data-scaling exponents, optimal-window trends, and the checkpoint-density threshold. The results closely match the theory, including the predicted distinction between hard and easy tasks.
The use of 512 trajectories per configuration provides substantially stronger evidence for the theoretical results than a small number of individual optimization runs would provide.
S4. Useful analysis and practical implications
The paper connects power averaging to dynamic EMA and explains the growing-window retention rule used by the Anytime Pretraining method (Meterez et al., 2026). It also clarifies why fixed-rate EMA can retain a positive noise floor while growing-window averaging continues reducing noise.
The checkpoint-count theorem is particularly practical: it suggests that averaging every update is unnecessary and gives a sublinear checkpoint-count requirement for preserving the continuous-averaging noise rate.
Critical weaknesses
W1. Need to better justify novelty relative to existing theory
The framework closely follows the Functional Scaling Law framework (FSL; Li et al., 2025a; Li et al., 2026) in its setting, SDE approximation, intrinsic-time formulation, and analytical tools.
The removal of saturation through tail averaging was already established in Beating SGD Saturation with Tail-Averaging and Minibatching (Mücke et al., 2019). Minimax properties of averaging under horizon-free schedules were also studied by the Anytime Pretraining method (Meterez et al., 2026).
The main new elements appear to be:
- the two-time structure of the merged noise kernel;
- profile-dependent constant optimization within rate-equivalent families;
- the checkpoint-count theorem.
W2. Gap between the theoretical assumptions and the LLM experiments
The theory assumes power-law kernel regression, one-pass SGD, and a label-noise-dominated approximation to gradient covariance. The LLM experiments instead use MuonW, combining Muon for hidden-layer matrices with AdamW for the remaining parameters.
Moreover, the LLM decomposition fit reaches , outside the theoretical range . The fitting procedure also uses conventions that do not satisfy all assumptions of the kernel-regression theory. Therefore, MM-FSL functions partly as a phenomenological loss model in the LLM experiments rather than as a direct application of the theory.
This issue does not invalidate the kernel-regression results, but it limits how strongly the fitted signal–noise decomposition can support the proposed explanation of LLM behavior.
W3. Limited LLM evaluation
The LLM evaluation uses only one 124M-parameter model scale. The cross-schedule prediction study covers three learning-rate schedules and lacks a comparison with empirical annealing-aware loss-curve models using a similar number of fitted parameters.
The signal and noise components are inferred rather than measured independently. Consequently, agreement with total validation loss does not uniquely identify the decomposition. The fitted noise exponent reaching its optimization boundary further highlights this concern.
The fixed-EMA comparison also uses a single , while dynamic EMA uses a fixed . Neither parameter is independently optimized across budgets. The reported budget curves are pointwise envelopes of intermediate evaluations from 20B-token runs rather than separately trained runs at every budget. Multi-seed uncertainty is also not reported.
W4. Limited generalizability and operational guidance
The optimal rules depend on spectral quantities such as , and , but the paper does not provide a reliable procedure for estimating these quantities for an LLM. It is therefore unclear how a practitioner should choose the tail fraction or power exponent without performing a substantial hyperparameter sweep.
The checkpoint-count theorem is practically promising, but it is validated only in kernel regression. The paper does not test whether its checkpoint-density criterion transfers to LLM pretraining or quantify checkpoint storage, communication, and online averaging overhead at realistic model scales.
Questions for authors
Q1. Could the authors provide a result-by-result comparison with the Functional Scaling Law framework (FSL; Li et al., 2025a; Li et al., 2026), Beating SGD Saturation with Tail-Averaging and Minibatching (Mücke et al., 2019), and the Anytime Pretraining method (Meterez et al., 2026)? In particular, which parts of Theorem 2 require fundamentally new arguments?
Q2. Do the two-time kernel structure and the qualitative signal–noise conclusions remain valid under diagonal preconditioning, AdamW, or Muon? How should readers interpret the LLM decomposition when the fitted value lies outside the theoretical range?
Q3. How does MM-FSL compare with empirical cross-schedule loss predictors using a similar number of fitted parameters? Could the authors report multi-seed results, a tuned fixed-EMA baseline, and results for at least one larger model?
Q4. How should practitioners estimate and , or directly select and , from a short pilot run? Does the checkpoint-count criterion hold empirically in LLM pretraining?
4 comments
Response to Reviewer qGmD (2/3)
[W2-2&Q2-2] How should readers interpret the fitted noise exponent , outside the theoretical range?
The fitted should not be interpreted as an independently identified forgetting exponent in LLMs. In Figure 4, the signal, noise, and irreducible-loss terms are jointly inferred from total-loss curves over a limited training budget. Their parameters can compensate for one another, making the decomposition difficult to identify uniquely. Our additional cutoff-sensitivity analysis in Appendix J.7 of the revised manuscript illustrates this: changing from to , without constraining , yields with lower fitting error. Nevertheless, across all six tested cutoffs, dynamic EMA consistently increases the fitted signal contribution and reduces the fitted noise contribution relative to the unmerged constant-LR trajectory. Thus, the qualitative comparison is robust across these settings, but the fit does not independently establish the underlying mechanism or guarantee reliable extrapolation.
Moreover, is not a fundamental restriction of FSL. It follows from the specific noise–curvature assumption in PLK regression with SGD. Under a more general noise covariance , we obtain for (Appendix G). This demonstrates that larger exponents are mathematically possible, without claiming that this covariance structure has been established for LLMs.
[W3-2] The signal and noise components are inferred rather than independently measured.
We add controlled constant-LR linear-regression experiments for original and merged models (Figure 8). Population-gradient descent from the original initialization measures , while SGD initialized at , averaged over multiple runs, measures . Separately, fitting FSL/MM-FSL solely to total-loss curves yields and . We find , , and . This independently supports the fitted decomposition in the controlled setting; the LLM decomposition remains a theory-inspired effective interpretation.
[W3-1&Q3-1] How does MM-FSL compare with empirical cross-schedule loss predictors using a similar number of fitted parameters?
MM-FSL addresses a distinct prediction problem. Existing cross-schedule predictors, including the LR-annealing law [6], Multi-Power Law [7], and original FSL [1], model unmerged loss trajectories without explicitly accounting for model averaging. Importantly, merged loss depends on cross-checkpoint correlations, which cannot be recovered from raw loss curves alone, even if the latter are predicted accurately. MM-FSL captures these correlations through its two-time forgetting kernel, enabling profile-aware prediction of merged-loss trajectories across LR schedules. To our knowledge, this capability has not been established by existing cross-schedule predictors.
[W3-3&Q3-2] Tuned fixed-EMA baselines and larger-model experiments.
We add budget-tuned fixed-EMA comparisons on both 124M and 275M LLaMA-style models (Figure 7), searching over . Smaller budgets favor , whereas larger budgets favor . Dynamic EMA with one fixed averaging configuration per model remains competitive across the evaluated budgets, supporting its practical advantage of accommodating different budgets without retuning the averaging parameter.
[W4-1&Q4-1] How should practitioners estimate and , or directly select and , from a short pilot run?
Practical parameter selection does not necessarily require estimating and . For example, one can evaluate a small grid of power-averaging parameters along a single training trajectory, updating the corresponding averaged models online and selecting the best candidate by validation loss. Tail-averaging ratios can similarly be compared using saved checkpoints.
However, selecting the best parameter at the current budget is different from predicting its optimum at a larger budget. As illustrated in Figure 6, the preferred averaging parameter can change with training budget. Reliably predicting this dependence from short pilot runs remains an important direction for future work.
Response to Reviewer qGmD (3/3)
[W4-2&Q4-2] Does the checkpoint-count criterion hold empirically in LLM pretraining?
We add a checkpoint-density experiment on a 124M LLaMA-style model, reported in Figure 10 of the revised manuscript. Using the same constant-LR trajectory and training endpoint, we vary the checkpoint count within tail windows of steps. We define the empirical threshold as the minimum tested count needed to remain within validation loss of a dense reference sampled every 10 steps.
The resulting thresholds are , with an empirical fit . This provides qualitative support for the predicted sublinear checkpoint requirement: substantially fewer checkpoints suffice than maintaining a fixed sampling density. The empirical criterion concerns total validation loss, whereas the theorem characterizes noise-risk scaling.
References
[1] Li et al. (2025). Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS, 2025. arXiv:2509.19189.
[2] Li et al. (2026). Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay. COLT, 2026. arXiv:2602.06797.
[3] Mücke, N., Neu, G., and Rosasco, L. (2019). Beating SGD Saturation with Tail-Averaging and Minibatching. NeurIPS, 2019. arXiv:1902.08668.
[4] Meterez et al. (2026). Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging. arXiv:2602.03702.
[5] Wang et al. (2026). When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective. NeurIPS, 2026. arXiv:2611.16560.
[6] Tissue, H., Wang, V., and Wang, L. (2024). Scaling Law with Learning Rate Annealing. arXiv:2408.11029.
[7] Luo et al. (2025). A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules. arXiv:2503.12811.
Looking Forward to the Reviewer's Reply
Dear reviewer qGmD,
As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your suggestions to ensure we have fully resolved any outstanding issues. Thank you for your time and consideration.
Response to Reviewer qGmD (1/3)
We thank the reviewer for their thoughtful and detailed assessment, particularly for recognizing the significance of our theoretical contributions and the strength of our controlled validation. We have carefully addressed the concerns raised in the review and revised the manuscript accordingly, with theoretical extensions in Appendices G–I and additional experiments in Appendix J.
We thank the reviewer for requesting this clarification. Our contributions go beyond known averaging rates.
Compared with FSL [1, 2]. We derive the merged signal filter hϕ(z)=(∫01ϕ(u)e−zudu)2. A power-law universality argument yields Sϕ(t)≍t−mins,2κ for ϕ(u)≍uκ−1 near zero, with a logarithmic correction at s=2κ, identifying profile-dependent signal saturation. We also derive a two-time forgetting kernel that captures shared noise across checkpoints. Together, these characterize merged-loss dynamics under general admissible LR schedules.
Compared with Mücke et al. [3]. Beyond rate-optimal tail averaging, Theorem 2 selects risk-optimal parameters within tail and power families after joint LR tuning. In the hard regime, rN⋆≍(κN⋆)−1≍N−(sc−s)/(1+sc), where sc=1−1/q. This requires signal-penalty and noise expansions valid for budget-dependent profiles, optimizing the correction above gradient flow. In the easy regime, we optimize the leading-risk constant. These results distinguish rate-equivalent parameter choices.
Compared with Anytime Pretraining [4]. Their work establishes risk bounds and optimal-rate guarantees for horizon-free schedules. We provide an explicit merged-loss functional for general admissible LR schedules and fractional merging profiles, supporting full-trajectory prediction and profile optimization.
Finally, our checkpoint-count theorem establishes m≳L1−1/q for preserving the noise-risk order of continuous averaging under constant LR.
The MM-FSL construction extends beyond SGD when an effective propagation and noise-covariance description is available. We provide this extension in Appendix I, including signal learning, the two-time noise kernel, and connections to preconditioned and sign-based optimizers.
Signal learning and two-time noise. For constant effective matrices, consider dδt=−Aδtdt+n(t)GdWt, where δt=θt−θ⋆ and E(θ)=21δ⊤Hδ. For deterministic initialization, fractional merging gives
EE(θˉt)=Sϕ(t)+∫0tKϕ(t;τ)n(τ)dτ,
with
Sϕ(t)=21δ0⊤Mϕ(t)⊤HMϕ(t)δ0,Mϕ(t)=∫01ϕ(u)e−Atudu,
Kϕ(t;τ)=21Tr(HRϕ(t,τ)GG⊤Rϕ(t,τ)⊤),Rϕ(t,τ)=∫τ/t1ϕ(u)e−A(tu−τ)du.
Thus, the two-time structure follows from averaging shared stochastic trajectories and does not require A=H. For a fixed symmetric positive-definite preconditioner P, we have A=PH and GG⊤=PΣP⊤, where Σ is the gradient-noise covariance. Appendix I.2 shows that the signal penalty Sϕ(t)≥S(t) persists without requiring P and H to commute. The noise benefit depends on the corresponding propagation, covariance, and LR schedule.
Sign-based optimizers, AdamW, and Muon. Recent work [5] establishes a SignSGD FSL with S(t)≂t−2s and K(u)≂u−(2−2/q), arising from effective H1/2 preconditioning. Appendix I.4 applies our merging construction within this effective model using its optimizer-specific intrinsic clock and noise intensity. The same work develops heuristic extensions to RMSprop and Adam with numerical support. Muon uses a related matrix-sign mechanism, but a full Muon FSL is not established by that analysis. Accordingly, these results support extensions beyond SGD under the stated effective-model conditions; they do not transfer the SGD exponents or optimality guarantees unchanged to full AdamW or Muon dynamics.