acceptodds
Under review as a conference paper at ICLR 2027

Model Merging Reshapes Learning Dynamics: A Functional Scaling Law Perspective

Abstract

We study model merging in power-law kernel regression, a tractable setting for understanding how checkpoint averaging reshapes learning dynamics. Specifically, we establish a functional scaling law (FSL) for fractional averaging in intrinsic time under general learning rate schedules, decomposing merged loss dynamics into signal learning and noise forgetting. Compared with the last iterate, merging increases signal error and can saturate its decay exponent. Its effect on noise depends on the injection time: merging retains more early noise but suppresses late noise, making its overall benefit depend on the learning rate schedule. The key mechanism is that averaging retains less-trained checkpoints, while late noise affects only the checkpoints following its injection. This characterization yields implications for merging design: tail and power averaging mitigate signal saturation, power averaging corresponds to a dynamic exponential moving average with a growing window, and sufficiently dense checkpoints preserve the noise-reduction rate of continuous averaging. Controlled SGD experiments in kernel regression empirically support the predicted data-scaling rates, optimal-window trends, and checkpoint-count criterion. Large language model (LLM) pretraining experiments show cross-schedule prediction without refitting and illustrate how the inferred signal–noise tradeoff explains observed merging gains.

Primary area:
learning theory
Keywords:
model merging, functional scaling laws, scaling laws, learning dynamics, learning rate schedules, deep learning theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qGmDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Checkpoint averaging along a single training trajectory is increasingly used in LLM pretraining to approximate terminal learning-rate decay, estimate annealed performance during training, and support horizon-free schedules. Classical averaging theory primarily characterizes final convergence rates. It does not fully explain when merging improves performance, how its effect depends on the learning-rate schedule, or how to choose among averaging profiles with the same asymptotic rate.

To address these challenges, this paper proposes Model-Merging Functional Scaling Laws (MM-FSL), extending the Functional Scaling Law framework (FSL; Li et al., 2025a; Li et al., 2026) to checkpoint averaging in power-law kernel regression. Specifically, the authors:

  • derive exact transformations of the signal-learning function and noise-forgetting kernel under fractional averaging;
  • show that merging increases signal error, retains more early noise, and suppresses late noise;
  • characterize risk-optimal tail and power-averaging windows after jointly tuning the learning rate;
  • interpret power averaging as a dynamic EMA with a growing window;
  • prove that m≳L1−1/qm \gtrsim L^{1-1/q} checkpoints suffice to preserve the noise-reduction rate of continuous averaging.

The authors evaluate their approach using controlled SGD experiments on Gaussian power-law kernel regression and 124M-parameter LLaMA-style models trained on FineWeb-Edu. The comparisons include the last iterate, fixed EMA, uniform merging, tail averaging, dynamic EMA, and constant, cosine, and WSD learning-rate schedules.

Critical strengths

S1. Important and timely research problem

Checkpoint averaging is increasingly used in LLM pretraining, as illustrated by methods such as LAWA, WSM, WSD-based training, and the use of EMA in the DeepSeek-V3 Technical Report. However, the interaction between averaging and learning-rate scheduling remains insufficiently understood. The paper addresses a meaningful gap between classical averaging theory and current LLM training practice.

S2. Clean theoretical characterization

The key technical insight is that the merged noise kernel depends separately on evaluation time and noise-injection time. This distinction enables the paper to identify three different effects of merging: increased signal error, increased retention of early noise, and suppression of late noise.

The resulting criterion—that merging helps when net noise reduction exceeds signal cost—is intuitive and useful. The risk-optimal window analysis also goes beyond convergence rates by resolving profile-dependent constants within averaging families that share the same asymptotic rate.

S3. Strong controlled validation

The kernel-regression experiments test several distinct predictions, including data-scaling exponents, optimal-window trends, and the checkpoint-density threshold. The results closely match the theory, including the predicted distinction between hard and easy tasks.

The use of 512 trajectories per configuration provides substantially stronger evidence for the theoretical results than a small number of individual optimization runs would provide.

S4. Useful analysis and practical implications

The paper connects power averaging to dynamic EMA and explains the growing-window retention rule used by the Anytime Pretraining method (Meterez et al., 2026). It also clarifies why fixed-rate EMA can retain a positive noise floor while growing-window averaging continues reducing noise.

The checkpoint-count theorem is particularly practical: it suggests that averaging every update is unnecessary and gives a sublinear checkpoint-count requirement for preserving the continuous-averaging noise rate.

Critical weaknesses

W1. Need to better justify novelty relative to existing theory

The framework closely follows the Functional Scaling Law framework (FSL; Li et al., 2025a; Li et al., 2026) in its setting, SDE approximation, intrinsic-time formulation, and analytical tools.

The removal of saturation through tail averaging was already established in Beating SGD Saturation with Tail-Averaging and Minibatching (Mücke et al., 2019). Minimax properties of averaging under horizon-free schedules were also studied by the Anytime Pretraining method (Meterez et al., 2026).

The main new elements appear to be:

  1. the two-time structure of the merged noise kernel;
  2. profile-dependent constant optimization within rate-equivalent families;
  3. the checkpoint-count theorem.

W2. Gap between the theoretical assumptions and the LLM experiments

The theory assumes power-law kernel regression, one-pass SGD, and a label-noise-dominated approximation to gradient covariance. The LLM experiments instead use MuonW, combining Muon for hidden-layer matrices with AdamW for the remaining parameters.

Moreover, the LLM decomposition fit reaches ν=3\nu=3, outside the theoretical range ν∈(1,2)\nu\in(1,2). The fitting procedure also uses conventions that do not satisfy all assumptions of the kernel-regression theory. Therefore, MM-FSL functions partly as a phenomenological loss model in the LLM experiments rather than as a direct application of the theory.

This issue does not invalidate the kernel-regression results, but it limits how strongly the fitted signal–noise decomposition can support the proposed explanation of LLM behavior.

W3. Limited LLM evaluation

The LLM evaluation uses only one 124M-parameter model scale. The cross-schedule prediction study covers three learning-rate schedules and lacks a comparison with empirical annealing-aware loss-curve models using a similar number of fitted parameters.

The signal and noise components are inferred rather than measured independently. Consequently, agreement with total validation loss does not uniquely identify the decomposition. The fitted noise exponent reaching its optimization boundary further highlights this concern.

The fixed-EMA comparison also uses a single β=0.99\beta=0.99, while dynamic EMA uses a fixed κ=50\kappa=50. Neither parameter is independently optimized across budgets. The reported budget curves are pointwise envelopes of intermediate evaluations from 20B-token runs rather than separately trained runs at every budget. Multi-seed uncertainty is also not reported.

W4. Limited generalizability and operational guidance

The optimal rules depend on spectral quantities such as s,qs, q, and scs_c, but the paper does not provide a reliable procedure for estimating these quantities for an LLM. It is therefore unclear how a practitioner should choose the tail fraction rr or power exponent κ\kappa without performing a substantial hyperparameter sweep.

The checkpoint-count theorem is practically promising, but it is validated only in kernel regression. The paper does not test whether its checkpoint-density criterion transfers to LLM pretraining or quantify checkpoint storage, communication, and online averaging overhead at realistic model scales.

Questions for authors

Q1. Could the authors provide a result-by-result comparison with the Functional Scaling Law framework (FSL; Li et al., 2025a; Li et al., 2026), Beating SGD Saturation with Tail-Averaging and Minibatching (Mücke et al., 2019), and the Anytime Pretraining method (Meterez et al., 2026)? In particular, which parts of Theorem 2 require fundamentally new arguments?

Q2. Do the two-time kernel structure and the qualitative signal–noise conclusions remain valid under diagonal preconditioning, AdamW, or Muon? How should readers interpret the LLM decomposition when the fitted value ν=3\nu=3 lies outside the theoretical range?

Q3. How does MM-FSL compare with empirical cross-schedule loss predictors using a similar number of fitted parameters? Could the authors report multi-seed results, a tuned fixed-EMA baseline, and results for at least one larger model?

Q4. How should practitioners estimate ss and qq, or directly select rr and κ\kappa, from a short pilot run? Does the checkpoint-count criterion m≳L1−1/qm\gtrsim L^{1-1/q} hold empirically in LLM pretraining?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qGmD (1/3)

We thank the reviewer for their thoughtful and detailed assessment, particularly for recognizing the significance of our theoretical contributions and the strength of our controlled validation. We have carefully addressed the concerns raised in the review and revised the manuscript accordingly, with theoretical extensions in Appendices G–I and additional experiments in Appendix J.


[W1&Q1] Need to better justify novelty relative to existing theory.

We thank the reviewer for requesting this clarification. Our contributions go beyond known averaging rates.

Compared with FSL [1, 2]. We derive the merged signal filter hϕ(z)=(∫01ϕ(u)e−zu du)2h_\phi(z)=(\int_0^1\phi(u)e^{-zu}\,du)^2. A power-law universality argument yields Sϕ(t)≍t−min⁡s,2κS_\phi(t)\asymp t^{-\min\\{s,2\kappa\\}} for ϕ(u)≍uκ−1\phi(u)\asymp u^{\kappa-1} near zero, with a logarithmic correction at s=2κs=2\kappa, identifying profile-dependent signal saturation. We also derive a two-time forgetting kernel that captures shared noise across checkpoints. Together, these characterize merged-loss dynamics under general admissible LR schedules.

Compared with Mücke et al. [3]. Beyond rate-optimal tail averaging, Theorem 2 selects risk-optimal parameters within tail and power families after joint LR tuning. In the hard regime, rN⋆≍(κN⋆)−1≍N−(sc−s)/(1+sc)r_N^\star\asymp(\kappa_N^\star)^{-1}\asymp N^{-(s_c-s)/(1+s_c)}, where sc=1−1/qs_c=1-1/q. This requires signal-penalty and noise expansions valid for budget-dependent profiles, optimizing the correction above gradient flow. In the easy regime, we optimize the leading-risk constant. These results distinguish rate-equivalent parameter choices.

Compared with Anytime Pretraining [4]. Their work establishes risk bounds and optimal-rate guarantees for horizon-free schedules. We provide an explicit merged-loss functional for general admissible LR schedules and fractional merging profiles, supporting full-trajectory prediction and profile optimization.

Finally, our checkpoint-count theorem establishes m≳L1−1/qm\gtrsim L^{1-1/q} for preserving the noise-risk order of continuous averaging under constant LR.


[W2-1&Q2-1] Do the two-time kernel structure and the qualitative signal–noise conclusions remain valid under diagonal preconditioning, AdamW, or Muon?

The MM-FSL construction extends beyond SGD when an effective propagation and noise-covariance description is available. We provide this extension in Appendix I, including signal learning, the two-time noise kernel, and connections to preconditioned and sign-based optimizers.

Signal learning and two-time noise. For constant effective matrices, consider dδt=−Aδt dt+n(t)G dWtd\delta_t=-A\delta_t\,dt+\sqrt{n(t)}G\,dW_t, where δt=θt−θ⋆\delta_t=\theta_t-\theta^\star and E(θ)=12δ⊤Hδ\mathcal E(\theta)=\frac12\delta^\top H\delta. For deterministic initialization, fractional merging gives

EE(θˉt)=Sϕ(t)+∫0tKϕ(t;τ)n(τ) dτ,\mathbb E\mathcal E(\bar\theta_t)=S_\phi(t)+\int_0^tK_\phi(t;\tau)n(\tau)\,d\tau,

with

Sϕ(t)=12δ0⊤Mϕ(t)⊤HMϕ(t)δ0,Mϕ(t)=∫01ϕ(u)e−Atu du,S_\phi(t)=\frac12\delta_0^\top M_\phi(t)^\top HM_\phi(t)\delta_0,\qquad M_\phi(t)=\int_0^1\phi(u)e^{-Atu}\,du,

Kϕ(t;τ)=12Tr⁡ ⁣(HRϕ(t,τ)GG⊤Rϕ(t,τ)⊤),Rϕ(t,τ)=∫τ/t1ϕ(u)e−A(tu−τ) du.K_\phi(t;\tau)=\frac12\operatorname{Tr}\!\left(HR_\phi(t,\tau)GG^\top R_\phi(t,\tau)^\top\right),\qquad R_\phi(t,\tau)=\int_{\tau/t}^1\phi(u)e^{-A(tu-\tau)}\,du.

Thus, the two-time structure follows from averaging shared stochastic trajectories and does not require A=HA=H. For a fixed symmetric positive-definite preconditioner PP, we have A=PHA=PH and GG⊤=PΣP⊤GG^\top=P\Sigma P^\top, where Σ\Sigma is the gradient-noise covariance. Appendix I.2 shows that the signal penalty Sϕ(t)≥S(t)S_\phi(t)\ge S(t) persists without requiring PP and HH to commute. The noise benefit depends on the corresponding propagation, covariance, and LR schedule.

Sign-based optimizers, AdamW, and Muon. Recent work [5] establishes a SignSGD FSL with S(t)≂t−2sS(t)\eqsim t^{-2s} and K(u)≂u−(2−2/q)K(u)\eqsim u^{-(2-2/q)}, arising from effective H1/2H^{1/2} preconditioning. Appendix I.4 applies our merging construction within this effective model using its optimizer-specific intrinsic clock and noise intensity. The same work develops heuristic extensions to RMSprop and Adam with numerical support. Muon uses a related matrix-sign mechanism, but a full Muon FSL is not established by that analysis. Accordingly, these results support extensions beyond SGD under the stated effective-model conditions; they do not transfer the SGD exponents or optimality guarantees unchanged to full AdamW or Muon dynamics.

AuthorsOpenReview ↗

Response to Reviewer qGmD (2/3)

[W2-2&Q2-2] How should readers interpret the fitted noise exponent ν=3\nu=3, outside the theoretical range?

The fitted ν=3\nu=3 should not be interpreted as an independently identified forgetting exponent in LLMs. In Figure 4, the signal, noise, and irreducible-loss terms are jointly inferred from total-loss curves over a limited training budget. Their parameters can compensate for one another, making the decomposition difficult to identify uniquely. Our additional cutoff-sensitivity analysis in Appendix J.7 of the revised manuscript illustrates this: changing δK\delta_K from 0.50.5 to 0.050.05, without constraining ν\nu, yields ν=1.387\nu=1.387 with lower fitting error. Nevertheless, across all six tested cutoffs, dynamic EMA consistently increases the fitted signal contribution and reduces the fitted noise contribution relative to the unmerged constant-LR trajectory. Thus, the qualitative comparison is robust across these settings, but the fit does not independently establish the underlying mechanism or guarantee reliable extrapolation.

Moreover, ν∈(1,2)\nu \in (1,2) is not a fundamental restriction of FSL. It follows from the specific noise–curvature assumption Σ∝H\Sigma\propto H in PLK regression with SGD. Under a more general noise covariance Σ∝H1+a\Sigma\propto H^{1+a}, we obtain ν=2+a−1/q>2\nu=2+a-1/q>2 for a>1/qa>1/q (Appendix G). This demonstrates that larger exponents are mathematically possible, without claiming that this covariance structure has been established for LLMs.


[W3-2] The signal and noise components are inferred rather than independently measured.

We add controlled constant-LR linear-regression experiments for original and merged models (Figure 8). Population-gradient descent from the original initialization measures StindS_t^{\mathrm{ind}}, while SGD initialized at θ⋆\theta^\star, averaged over multiple runs, measures NtindN_t^{\mathrm{ind}}. Separately, fitting FSL/MM-FSL solely to total-loss curves yields StfitS_t^{\mathrm{fit}} and NtfitN_t^{\mathrm{fit}}. We find Stfit≈StindS_t^{\mathrm{fit}}\approx S_t^{\mathrm{ind}}, Ntfit≈NtindN_t^{\mathrm{fit}}\approx N_t^{\mathrm{ind}}, and Rtobs≈Stfit+Ntfit≈Stind+NtindR_t^{\mathrm{obs}}\approx S_t^{\mathrm{fit}}+N_t^{\mathrm{fit}}\approx S_t^{\mathrm{ind}}+N_t^{\mathrm{ind}}. This independently supports the fitted decomposition in the controlled setting; the LLM decomposition remains a theory-inspired effective interpretation.


[W3-1&Q3-1] How does MM-FSL compare with empirical cross-schedule loss predictors using a similar number of fitted parameters?

MM-FSL addresses a distinct prediction problem. Existing cross-schedule predictors, including the LR-annealing law [6], Multi-Power Law [7], and original FSL [1], model unmerged loss trajectories without explicitly accounting for model averaging. Importantly, merged loss depends on cross-checkpoint correlations, which cannot be recovered from raw loss curves alone, even if the latter are predicted accurately. MM-FSL captures these correlations through its two-time forgetting kernel, enabling profile-aware prediction of merged-loss trajectories across LR schedules. To our knowledge, this capability has not been established by existing cross-schedule predictors.


[W3-3&Q3-2] Tuned fixed-EMA baselines and larger-model experiments.

We add budget-tuned fixed-EMA comparisons on both 124M and 275M LLaMA-style models (Figure 7), searching over β∈0.9,0.99,0.999,0.9999\beta\in\\{0.9,0.99,0.999,0.9999\\}. Smaller budgets favor β=0.99\beta=0.99, whereas larger budgets favor β=0.999\beta=0.999. Dynamic EMA with one fixed averaging configuration per model remains competitive across the evaluated budgets, supporting its practical advantage of accommodating different budgets without retuning the averaging parameter.


[W4-1&Q4-1] How should practitioners estimate ss and qq, or directly select rr and κ\kappa, from a short pilot run?

Practical parameter selection does not necessarily require estimating ss and qq. For example, one can evaluate a small grid of power-averaging parameters κ\kappa along a single training trajectory, updating the corresponding averaged models online and selecting the best candidate by validation loss. Tail-averaging ratios rr can similarly be compared using saved checkpoints.

However, selecting the best parameter at the current budget is different from predicting its optimum at a larger budget. As illustrated in Figure 6, the preferred averaging parameter can change with training budget. Reliably predicting this dependence from short pilot runs remains an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer qGmD (3/3)

[W4-2&Q4-2] Does the checkpoint-count criterion m≳L1−1/qm \gtrsim L^{1-1/q} hold empirically in LLM pretraining?

We add a checkpoint-density experiment on a 124M LLaMA-style model, reported in Figure 10 of the revised manuscript. Using the same constant-LR trajectory and training endpoint, we vary the checkpoint count within tail windows of L∈80,160,320,640,1280,2560L\in\\{80,160,320,640,1280,2560\\} steps. We define the empirical threshold as the minimum tested count needed to remain within 0.0010.001 validation loss of a dense reference sampled every 10 steps.

The resulting thresholds are 6,7,12,15,22,29\\{6,7,12,15,22,29\\}, with an empirical fit mcrit≈0.71L0.48m_{\mathrm{crit}}\approx0.71L^{0.48}. This provides qualitative support for the predicted sublinear checkpoint requirement: substantially fewer checkpoints suffice than maintaining a fixed sampling density. The empirical criterion concerns total validation loss, whereas the theorem characterizes noise-risk scaling.


References

[1] Li et al. (2025). Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS, 2025. arXiv:2509.19189.

[2] Li et al. (2026). Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay. COLT, 2026. arXiv:2602.06797.

[3] Mücke, N., Neu, G., and Rosasco, L. (2019). Beating SGD Saturation with Tail-Averaging and Minibatching. NeurIPS, 2019. arXiv:1902.08668.

[4] Meterez et al. (2026). Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging. arXiv:2602.03702.

[5] Wang et al. (2026). When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective. NeurIPS, 2026. arXiv:2611.16560.

[6] Tissue, H., Wang, V., and Wang, L. (2024). Scaling Law with Learning Rate Annealing. arXiv:2408.11029.

[7] Luo et al. (2025). A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules. arXiv:2503.12811.

AuthorsOpenReview ↗

Looking Forward to the Reviewer's Reply

Dear reviewer qGmD,

As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your suggestions to ensure we have fully resolved any outstanding issues. Thank you for your time and consideration.

Reviewer_z3ebOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends functional scaling laws to study model merging in kernel regression. It characterizes how averaging affects signal learning and noise forgetting, and how this depends on the learning rate schedule. The paper also provides results on optimal averaging windows, dynamic EMA, and checkpoint counts. The experiments include kernel regression and language modeling at 124M scale.

Critical strengths
  1. The theoretical analysis of how merging affects early and late noise is interesting and helps explain its dependence on the learning rate schedule.

  2. The results on optimal averaging windows and checkpoint counts are useful.

  3. The kernel regression experiments are consistent with the theoretical predictions.

Critical weaknesses
  1. Some of the rate-level results are already known from previous averaging literature. It is unclear how much the additional theoretical results change the practical choice of averaging strategies.

  2. The LLM experiments are not very convincing. The cross-schedule predictions are not compared against existing scaling laws, and some of the experimental settings do not directly match the theoretical assumptions.

  3. The paper does not validate the theoretically optimal averaging parameters in LLM training. The fixed EMA baseline in Figure 5 also does not appear to be sufficiently tuned.

Questions for authors
  1. How does the cross-schedule prediction compare with existing loss prediction methods?

  2. Would the conclusions of Figure 4 change if intrinsic-time EMA is used with the theoretically assumed parameter range?

  3. Can the authors verify whether the predicted optimal power exponent also holds for language modeling?

3 comments
AuthorsOpenReview ↗

Response to Reviewer z3eb (1/2)

We sincerely thank the reviewer for the encouraging and insightful feedback. We have revised the manuscript PDF to include the additional experiments in Appendix J and the theoretical extensions in Appendices G–I. Below are our responses to the questions and suggestions raised by the reviewer.


[W1] Some of the rate-level results are already known from previous averaging literature.

Beyond rate optimality. While existing averaging methods can achieve optimal convergence rates, rate optimality alone cannot distinguish averaging strategies with substantially different risks. For example, different fixed power exponents can attain the same optimal rate but yield different merged losses. By characterizing the signal–noise trade-off, MM-FSL enables risk-level optimization beyond convergence rates: minimizing the leading-risk constant in the easy regime and the correction above the gradient-flow baseline in the hard regime. This yields quantitative principles for selecting optimal averaging profiles.

Practical evidence from LLM pretraining. Figure 9 of the revised manuscript compares multiple fixed power exponents along the same constant-LR training trajectory for each model. The preferred exponent shifts from 2525 to 5050 to 100100 for 124M, and from 2525 to 5050 to 100100 to 200200 for 275M as the budget increases. The loss curves exhibit corresponding crossovers: smaller exponents perform better earlier, whereas larger exponents become preferable later. These results illustrate the practical importance of averaging-parameter selection beyond rate-level guarantees: a fixed exponent can be rate-optimal without minimizing loss at every finite budget. Experimental details are provided in Appendix J.5.


[W2-1&Q1] The cross-schedule predictions are not compared against existing scaling laws.

We clarify that all existing cross-schedule loss predictors—including Scaling Law with Learning Rate Annealing [1], Multi-Power Law [2], and the original FSL [3]—model raw, unmerged loss trajectories. They do not account for checkpoint averaging or its dependence on the merging profile. In contrast, our target is the loss curve of checkpoint-merged models, which cannot generally be recovered by averaging raw checkpoint losses. To our knowledge, our work is the first to predict checkpoint-merged loss curves across LR schedules while explicitly accounting for the merging profile.


[W2-2&Q2] Some of the experimental settings do not directly match the theoretical assumptions.

We clarify three points.

1. Time coordinate. Under a constant LR η0\eta_0, intrinsic time satisfies τ=η0t\tau=\eta_0t, so τ′/τ=t′/t\tau'/\tau=t'/t. Power averaging therefore assigns identical weights in intrinsic and physical time, yielding the same merged model.

2. Noise exponent and fitting sensitivity. The range ν∈(1,2)\nu\in(1,2) follows from specific noise–curvature assumptions: Σ∝H\Sigma\propto H yields ν=2−1/q\nu=2-1/q, whereas Σ∝H1+a\Sigma\propto H^{1+a} yields ν=2+a−1/q\nu=2+a-1/q, which can exceed 22 (Appendix G). Empirically, changing the noise cutoff from 0.50.5 to 0.050.05, without constraining ν\nu, changes the fitted exponent from 3.3743.374 to 1.3871.387 with lower fitting error. Across all six tested cutoffs, merging consistently increases the fitted signal contribution and reduces the fitted noise contribution relative to the unmerged constant-LR trajectory. These components remain effective descriptions rather than independently measured quantities; we report the cutoff-sensitivity analysis in Appendix J.7 of the revised manuscript.

3. Independent validation in a controlled setting. We add constant-LR linear-regression experiments (Figure 8): population-gradient descent from the original initialization measures signal error, while SGD initialized at θ⋆\theta^\star, averaged over multiple runs, measures noise error. Applying the same merging weights to both confirms that merging increases signal error while reducing noise error, independently of loss-curve fitting.


[W3-1&Q3] Can the authors verify whether the predicted optimal power exponent also holds for language modeling?

We add constant-LR power-merging experiments on 124M and 275M LLaMA-style models, searching over the power exponent κ∈6.25,12.5,25,50,100,200\kappa\in\\{6.25,12.5,25,50,100,200\\} at each data budget (Figure 6). For the 124M model trained for 10B tokens, the best tested κ\kappa increases as 25→50→10025\rightarrow50\rightarrow100, with a descriptive fit κ⋆(D)≈17.16D0.63\kappa^\star(D)\approx17.16D^{0.63}. For the 275M model trained for 20B tokens, it increases as 25→50→100→20025\rightarrow50\rightarrow100\rightarrow200, with a descriptive fit κ⋆(D)≈12.35D0.86\kappa^\star(D)\approx12.35D^{0.86}. Here, DD is measured in billions of tokens. Both trends qualitatively support the budget-dependent optimal power exponent predicted by Theorem 2.

AuthorsOpenReview ↗

Reviewer z3eb (2/2)

[W3-2] The fixed EMA baseline in Figure 5 also does not appear to be sufficiently tuned.

We strengthen the fixed-EMA baseline by tuning β∈0.9,0.99,0.999,0.9999\beta\in\\{0.9,0.99,0.999,0.9999\\} at each data budget on both 124M and 275M LLaMA-style models (Figure 7). Smaller budgets favor β=0.99\beta=0.99, while larger budgets favor β=0.999\beta=0.999. Dynamic EMA remains competitive across budgets using a single averaging configuration per model, without retuning for each budget.


References

[1] Tissue, H., Wang, V., and Wang, L. (2024). Scaling Law with Learning Rate Annealing. arXiv:2408.11029.

[2] Luo et al. (2025). A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules. arXiv:2503.12811.

[3] Li et al. (2025). Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS, 2025. arXiv:2509.19189.

AuthorsOpenReview ↗

Looking Forward to the Reviewer's Reply

Dear reviewer z3eb,

As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your support if we have fully resolved all outstanding issues. Thank you for your time and consideration.

Reviewer_ZHGhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends Functional Scaling Laws (FSL) to checkpoint averaging within a single training trajectory. It derives how merging affects signal learning and noise forgetting, and predicts optimal merging windows, averaging weights, and checkpoint density. The theory is validated through controlled kernel regression experiments, with additional experiments on 124M language models showing cross-schedule loss prediction and benefits from dynamic EMA.

Critical strengths
  1. Meaningful theoretical contribution. The paper extends FSL to model merging while accounting for correlated noise across checkpoints. The resulting analysis provides a clearer understanding of how merging interacts with LR schedules.
  2. Concrete, testable predictions. The derived scaling laws for optimal merging windows and checkpoint density go beyond qualitative explanations. Controlled experiments generally support these predictions.
Critical weaknesses
  1. Limited validation on real LLM training. The theoretical results rely on kernel regression and SGD assumptions, while LLM experiments are limited to 124M dense models. It remains unclear whether the predicted scaling laws hold for larger models and longer training budgets.
  2. The proposed mechanism is not convincingly validated on LLMs. The signal–noise decomposition is inferred from fitted loss curves rather than independently measured. Moreover, the fitted noise exponent reaches 3.0, outside the theoretical range of (1, 2). The good loss fit therefore does not necessarily validate the proposed mechanism.
Questions for authors
  1. Does the predicted optimal merging window scaling law hold for larger LLMs?
  2. How can the signal–noise decomposition be justified when the fitted noise exponent falls outside the theoretical range?
  3. Does dynamic EMA still outperform fixed EMA when both are properly tuned?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZHGh (1/2)

We sincerely thank the reviewer for the positive support and valuable feedback. We have revised the manuscript to include the additional experiments in Appendix J and the theoretical extensions in Appendices G–I. Below are our responses to the questions and suggestions raised by the reviewer.


[W1&Q1] Does the predicted optimal merging window scaling law hold for larger LLMs?

We conduct additional constant-LR power-merging experiments on 124M and 275M LLaMA-style models, searching over κ∈6.25,12.5,25,50,100,200\kappa\in\\{6.25,12.5,25,50,100,200\\} at different training budgets (see Figure 6 in Appendix J).

  • 124M (10B tokens): The best tested κ\kappa increases from 25 to 50 to 100, with a fitted trend κ⋆(D)≈17.16D0.63\kappa^\star(D)\approx17.16D^{0.63}.
  • 275M (20B tokens): The best tested κ\kappa increases from 25 to 50 to 100 to 200, with a fitted trend κ⋆(D)≈12.35D0.86\kappa^\star(D)\approx12.35D^{0.86}.

Here, DD is measured in billions of tokens. Both models exhibit a growing preference for larger κ\kappa as the training budget increases, qualitatively consistent with the budget-dependent behavior predicted by Theorem 2, although the fitted exponents do not quantitatively validate the scaling law derived for kernel regression.


[W2&Q2] The good loss fit therefore does not necessarily validate the proposed mechanism.... How can the signal–noise decomposition be justified when the fitted noise exponent falls outside the theoretical range?

We agree that a good total-loss fit alone does not validate the underlying signal–noise decomposition. We clarify three points:

  • 1. The forgetting exponent ν\nu can exceed 22. The range ν∈(1,2)\nu\in(1,2) follows from the specific noise–curvature assumptions of kernel regression with SGD, rather than the FSL functional form itself. For quadratic stochastic dynamics, K(u)∝Tr⁡(He−HuΣe−Hu)K(u)\propto\operatorname{Tr}(He^{-Hu}\Sigma e^{-Hu}), where HH is the Hessian and Σ\Sigma is the noise covariance. Under λj≍j−q\lambda_j\asymp j^{-q}, taking Σ∝H\Sigma\propto H gives ν=2−1/q\nu=2-1/q, whereas Σ∝H1+a\Sigma\propto H^{1+a} yields ν=2+a−1/q>2\nu=2+a-1/q>2 when a>1/qa>1/q. Thus, larger forgetting exponents can arise from different noise–curvature relationships. We provide a rigorous derivation in Appendix G.

  • 2. Loss fitting alone cannot uniquely identify the decomposition. In our LLM experiments, the inferred forgetting exponent can be sensitive to fitting choices, partly due to compensation between the signal-learning term, noise-forgetting term, and irreducible loss. Our additional cutoff-sensitivity analysis illustrates this: changing δK\delta_K from 0.50.5 to 0.050.05, without constraining ν\nu, changes the fitted exponent from 3.3743.374 to 1.3871.387 and reduces RMSE from 0.0091460.009146 to 0.0078170.007817. Across all six tested cutoffs, however, dynamic EMA consistently yields a larger fitted signal contribution and a smaller fitted noise contribution than the unmerged constant-LR trajectory. These results, reported in Appendix J.7 of the revised manuscript, support the robustness of this qualitative comparison across the tested settings, but do not uniquely identify the underlying components. We therefore interpret the fitted LLM components as effective descriptions rather than independently established mechanisms. Robust identification in practical LLM training requires further investigation beyond the scope of this work, as discussed in our global response.

  • 3. Independent validation in controlled experiments. We add linear-regression experiments that measure signal and noise separately (Figure 8 in Appendix J). Population gradient descent measures StindS_t^{\mathrm{ind}}, while SGD initialized at θ⋆\theta^\star measures NtindN_t^{\mathrm{ind}}. Independently, fitting MM-FSL to total-loss trajectories yields StfitS_t^{\mathrm{fit}} and NtfitN_t^{\mathrm{fit}}. The fitted components closely match the independent measurements, and their sum accurately reconstructs held-out total-loss trajectories. This provides direct support for the decomposition in the controlled setting, while its mechanistic interpretation in LLMs remains preliminary.

AuthorsOpenReview ↗

Response to Reviewer ZHGh (2/2)

[Q3] Does dynamic EMA still outperform fixed EMA when both are properly tuned?

We thank the reviewer for raising this point. Dynamic EMA need not strictly outperform fixed EMA when the latter is retuned for every budget; its advantage is achieving the optimal rate without budget-specific retuning.

Theory in kernel regression. A budget-independent fixed-EMA coefficient β\beta is rate-suboptimal. Budget-dependent tuning can recover the optimal rate: for example, choosing 1−β(N)≍N−(1+s)/(2−1/q)1-\beta(N)\asymp N^{-(1+s)/(2-1/q)} matches the optimal window scaling when s<1−1/qs<1-1/q, where NN denotes the number of SGD updates. In contrast, dynamic EMA can achieve the optimal rate with a fixed admissible power exponent κ\kappa, without knowing the final training budget, enabling anytime training. We provide the analysis in Appendix H of the revised manuscript.

LLM pretraining experiments. We add comparisons on both 124M and 275M LLaMA-style models in Figure 7 of the revised manuscript, allowing fixed EMA to select β∈0.9,0.99,0.999,0.9999\beta \in \\{0.9,0.99,0.999,0.9999\\} at each budget. Smaller budgets favor β=0.99\beta=0.99, whereas larger budgets favor β=0.999\beta=0.999. Dynamic EMA with a single averaging configuration remains competitive across the evaluated budgets, supporting its practical advantage of adapting to different budgets without retuning the averaging parameter.

AuthorsOpenReview ↗

Looking Forward to the Reviewer's Reply

Dear reviewer ZHGh,

As the deadline approaches, we kindly ask if our responses have adequately addressed your concerns. We would greatly appreciate your support if we have fully resolved all outstanding issues. Thank you for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their constructive and valuable feedback. We would like to take this opportunity to clarify the central contribution and scope of our work.

1. Our primary contribution is theoretical. Model averaging is widely used in stochastic optimization and modern LLM pretraining, yet a quantitative understanding of how it reshapes learning dynamics remains limited. We develop MM-FSL in kernel regression, which provides an explicit characterization of how model averaging reshapes learning dynamics by disentangling its effects on signal learning and noise forgetting under general LR schedules. This characterization reveals the underlying mechanisms of averaging and yields quantitative design principles, including risk-optimal averaging profiles, dynamic EMA, and checkpoint requirements. These predictions are quantitatively tested and supported by controlled SGD experiments.

2. Our LLM experiments aim to demonstrate the practical relevance and explanatory power of MM-FSL. The results show how MM-FSL helps interpret the effects of averaging in LLM pretraining through the trade-off between signal learning and noise forgetting. They also demonstrate promising predictive capability for merged-loss trajectories across LR schedules. In response to the reviewers, we have substantially expanded the LLM experiments, including larger models, budget-dependent averaging strategies, tuned EMA comparisons, and checkpoint-density studies.

3. Interpreting model averaging in LLM pretraining is distinct from establishing its underlying mechanisms and quantitatively optimal strategies. Our LLM experiments demonstrate the potential of MM-FSL as an explanatory and predictive tool, but do not yet establish a quantitative theory for practical LLM averaging. Doing so requires addressing a broad range of challenges arising from the complexity of practical LLM training, including:

  • Effective learning rates (ELRs): The loss dynamics of LLM pretraining is primarily governed by ELRs, rather than nominal LRs (e.g., Liu et al., 2026).
  • Parameter identification: Reliably estimating the signal-learning and noise-forgetting exponents ss and ν\nu from small-scale training runs.
  • Scaling and design: Determining how optimal averaging strategies and their gains scale with model size, batch size, and training budget.

In summary, our contribution is to establish a rigorous FSL-based theory for understanding and designing model averaging, quantitatively validate its predictions in controlled SGD experiments, and demonstrate its relevance and potential for LLM pretraining. Developing a fully quantitative theory for practical LLM averaging is an important next step, but should not be a prerequisite for recognizing the contributions established in this paper.

Dear Reviewers,

The author-reviewer discussion period will be closed on Nov. 18th. If you have not yet done so, please read the authors' rebuttal and the other reviews, and respond to the authors to clarify whether your concerns are addressed or remain.

Thank you for your time and effort.

Best regards, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.