acceptodds
Under review as a conference paper at ICLR 2027

How Multi-Timescale Memory Cools Optimization: Theory and Evidence from AdEMAMix

Abstract

Augmenting AdamW with multiple momentum timescales improves large language model pretraining, yet the mechanism driving these gains remains poorly understood. We identify an interaction between multi-timescale optimizer memory and the geometry of normalized networks: memory changes parameter norms and thereby controls how fast parameters rotate. Starting from exact norm dynamics under decoupled weight decay, we show that the parameter norm depends on the temporal correlation of the updates, with weight decay controlling how long these correlations influence optimization. We show how an independently weighted slow momentum can substantially increase the norm of parameters, while contributing negligible power to the instantaneous update, therefore acting as an endogenous annealing schedule. We study this mechanism in AdEMAMix for language-model pretraining. Controlled interventions show that this scheduling mechanism has clear practical consequences and is an important factor behind AdEMAMix's advantage over AdamW.

Primary area:
optimization
Keywords:
optimization, LLM, deep learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BGwBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper explains why AdEMAMix, which adds an independently weighted slow momentum branch to AdamW, outperforms AdamW in LLM pretraining. Starting from the exact squared-norm recursion under decoupled weight decay (Lemma 4.1), the authors show that in stationarity the parameter norm is governed by the temporal correlation structure of the updates (Theorem 4.3), with the squared radius equal to eta^2 C_delta[u] over (1 minus delta^2), where C_delta[u] weights the correlation between updates h steps apart by delta^h. Writing an optimizer as an execution kernel over its input history separates instantaneous power P[e] from the contribution of correlated history to the norm, F_delta[e]. For the two-timescale AdEMAMix kernel the paper proves that the ratio P/F_delta decreases strictly in the slow amplitude alpha (Theorem 4.8), and that the benefit is non-monotone in the slow memory length with an interior optimum (Proposition C.7). The proposed mechanism is that the slow branch contributes only a few percent of instantaneous update power (3.0% and 4.1% under constant and cosine schedules respectively, Table 3) yet, through its long correlation tail, multiplies the norm by roughly 42, which inversely slows parameter rotation and thereby turns the nominal learning rate schedule into an optimizer-specific rotational schedule. The mechanism is tested in 124M Llama pretraining on FineWeb through four controlled interventions: norm transplantation, radial removal, HYPERBALL fixed-radius control, and mid-training switch-off of the slow channel. Table 3 gives a sharp diagnostic contrast, 3 percent of present power versus 84.6 percent of survived displacement for the slow branch. Sec. 4.5 and App. E additionally score scheduling effects with a convex spherical-cap last-iterate bound imported from an anonymous companion submission. I verified the key derivations step by step and found no mathematical errors. My concerns are about the strength of the empirical evidence and the causal identification, detailed below.

Critical strengths
  1. The mechanism theory is clean, verifiable, and provides a genuinely new decomposition. Separating current-step power from the norm contribution of correlated history via C_delta[u] and F_delta[e], together with the exact two-pole pairing formula (C.20), the monotonicity theorem (Theorem 4.8), and the non-monotonicity result (Proposition C.7), forms a complete and falsifiable account of the phenomenon. It also recovers known limiting cases, namely the beta-independent AdamW radius of Kosson et al. (2024) as delta approaches 1 and the (1+beta)/(1-beta) growth factor of Heo et al. (2021). It is uncommon in this literature for every appendix proof to survive line-by-line checking, and this substantially increases my trust in the mechanism.
  2. The intervention experiments and diagnostic measurements are well designed. The four interventions each isolate a different facet of the mechanism and are matched to theoretical predictions, and Table 3 is a textbook mechanism diagnostic. The interaction prediction of Expectation A, namely a larger memory benefit under constant rates (0.076) than under cosine (0.029), follows from the theory and is confirmed experimentally. The resulting practical recommendation to choose memory and learning rate schedules jointly, and to compare optimizers by their realized rotational trajectories, has value beyond this single paper.
Critical weaknesses
  1. All central LLM evidence comes from single runs with no variance reported. App. F.1 states that all compared configurations share the same random seed and data order, and Table 1 notes that each tuning sweep uses one seed. The random-walk testbed uses five replicas and the MNIST MLP uses five seeds, yet every main conclusion (the 0.076 and 0.029 gaps, the norm transplants, HYPERBALL, and the switch-off experiment) rests on single training runs. It is not possible to rule out that the 0.029 cosine gap lies within seed noise, and the central causal claim that the norm channel mediates the AdEMAMix advantage is built entirely on this evidence.
  2. The causal identification is incomplete, and the norm-mediation and effective-LR-reparameterization explanations are not separated. After norm transplantation the best constant learning rate moves from 1e-4 to 8e-4 (Table 2), which shows that the transplant mostly shifts the effective learning rate operating point of AdamW by about a factor of eight. This is exactly the authors' Expectation B, but it also means that "AdamW with the AdEMAMix norm recovers most of the gap" is observationally equivalent to "AdEMAMix simply realizes a better effective step size schedule endogenously." The HYPERBALL fixed-radius control in Fig. 1c is the one experiment that can distinguish the two, and it is summarized as the benefit shrinking to about one third, but no calculation or concrete numbers behind this factor appear anywhere in the paper, so a reader cannot verify this key quantification.
  3. A core assumption conflicts with the paper's own diagnostics. Assumption 4.4 (zero-mean, mutually uncorrelated inputs) is a noise-dominated regime, while Fig. 15a shows non-negligible radial alignment between gradients and weights. Equation (C.18) further shows that a nonzero input mean enters C_delta with weight (1+delta)/(1-delta), which is much larger than the delta^h weighting of the noise channel, so the dominant contribution to the equilibrium norm may come from the mean or radial channel that the main text barely discusses. In addition, the execution-kernel approximation for AdamW and AdEMAMix freezes the preconditioner at injection time and ignores the coupling of the shared divisor with the alpha_t schedule, and this error is never quantified. The model-to-LLM comparison in Fig. 7 is only qualitative, the input power is not fitted, and the authors themselves note differences in shape and timing.
  4. The scheduling quantification rests on an anonymous, unverifiable companion submission. Sec. 4.5 and App. E.1 to E.3 rely on the bound in (E.2) and on claims such as the optimal allocation approaching linear decay, all cited as "Anonymous authors, 2026, under review at ICLR 2027." Moreover, using the value of an upper bound as a score across policies, where M and C_T(omega) change with the policy, is a heuristic whose justification is never argued. Fortunately the main mechanism in Sec. 4.1 to 4.3 and Sec. 5.1 to 5.3 does not depend on this part, so this is a supporting rather than fatal issue.
Questions for authors
  1. Can the main experiments (the selected configurations in Tables 1 and 2, Figs. 1b, 1c, 4 and 5) be replicated over at least three seeds with means and variances reported? In particular, what is the seed standard deviation of the 0.0289 cosine gap?
  2. How exactly is the "one third" residual benefit under fixed radius quantified? Please provide the final loss gap between AdamH and AdEMAMIXH at their respective optimal alpha, its ratio to the free-radius AdamW versus AdEMAMix gap, and the calculation behind it.
  3. Does Theorem 4.8 survive once the input mean is included as in (C.18)? Does the sign of d(P/F_delta)/d_alpha and the magnitude of the radius gain remain, and how much of the measured norm effect comes from the mean or radial channel versus the noise correlation channel?
3 comments

Thank you for your detailed review and interesting observations. We are glad that you found the mechanism informative and the experiments well designed. Your questions prompted additional experiments and an extension of the theoretical analysis.


Empirical evidence

Can the main experiments [...] be replicated over at least three seeds with means and variances reported? In particular, what is the seed standard deviation of the 0.0289 cosine gap?

We repeated the native optimizer comparisons, both norm transfers, and the Hyperball comparison on two additional seeds, keeping the selected hyperparameters fixed. These runs confirm the original findings, with small variation across seeds. All loss differences below are paired mean ±\pm sample standard deviation over three seeds.

  • AdEMAMix improves over AdamW by 0.074±0.0020.074\pm0.002 under constant LR and 0.030±0.0010.030\pm0.001 under cosine (Table 2).
  • Giving AdamW the AdEMAMix norm trajectory improves its loss by 0.063±0.0010.063\pm0.001, recovering about 85% of the mean constant-rate gap. The reverse transplant raises AdEMAMix's loss by 0.064±0.0030.064\pm0.003, removing about 87% of its advantage (Figure 1b, Table 2).

How exactly is the "one third" residual benefit under fixed radius quantified?

The mean final losses are approximately 3.1403.140 for AdamH and 3.1313.131 for AdEMAMixH at the selected α=2\alpha=2. Their paired gap is about 0.0090.009, with a standard deviation of 4×10−54\times10^{-5}. Relative to the free-radius cosine gap, this gives 0.009/0.030≃0.30.009/0.030\simeq0.3, leaving about one third of the original benefit. We now provide the calculation in Appendix F.3, alongside the results in Figure 1c and Table 2. Appendix F.1 specifies the tuning protocol.


Interpretation of the norm interventions

[...] "AdamW with the AdEMAMix norm recovers most of the gap" is observationally equivalent to "AdEMAMix simply realizes a better effective step size schedule endogenously."

Your observation that AdEMAMix "realizes a better effective step size schedule endogenously" captures the central idea of our paper. Our analysis identifies how multi-timescale memory generates this effective schedule through temporal correlations and norm growth, while the interventions test its contribution to the observed performance.

  • Each transplant configuration receives its own LR tuning on the common constant-rate grid. Giving AdamW the AdEMAMix norms both changes its preferred LR and improves the best loss reached across the grid (Figure 9, Appendix F.1).

  • Hyperball is a recently proposed optimization method that has demonstrated consistent improvements for AdamW and Muon in pretraining benchmarks. Applying it to our setting improves AdamW's performance, reproducing this established benefit. For AdEMAMix, however, it worsens performance and substantially narrows the advantage of slow memory (Figure 1c). This contrasting response supports our interpretation that AdEMAMix benefits from the norm dynamics that Hyperball removes.

These interventions thus support our conclusion that norm-induced rotational scheduling accounts for a substantial part of AdEMAMix's benefit.


[1/2, continues below]

[...continues]


Assumptions and the preconditioner approximation

Does Theorem 4.8 survive once the input mean is included as in (C.18)? Does the sign [...] and the magnitude of the radius gain remain [...]?

Theorem 4.3 already allows a constant nonzero update mean and arbitrary temporal correlations under its stationarity assumptions. Your question prompted us to extend the two-channel result as well. The new Corollary C.7 considers inputs with constant mean μ\boldsymbol\mu, centered variance σa2\sigma_a^2, and zero temporal correlations between centered inputs. For nonzero input power and stationary modeled responses, it gives

\bar R^2(\alpha) &=\frac{\eta^2}{1-\delta^2} \left[\sigma_a^2F_\delta[e] +\frac{1+\delta}{1-\delta}\|\boldsymbol\mu\|^2(1+\alpha)^2\right],\\\\ q_{\bar R}^2(\alpha) &=(1-\delta^2) \frac{\sigma_a^2P[e]+\|\boldsymbol\mu\|^2(1+\alpha)^2} {\sigma_a^2F_\delta[e]+\frac{1+\delta}{1-\delta}\|\boldsymbol\mu\|^2(1+\alpha)^2}, \qquad e=e^{\mathrm{AMix}}. \end{aligned}$$ Holding the input statistics, LR, decay, and ordered fast and slow retentions fixed, increasing $\alpha$ strictly increases the radius and strictly decreases the relative-update scale whenever $\sigma_a^2>0$. For a pure mean input, $q_{\bar R}=1-\delta$ independently of $\alpha$. These formulas give the magnitude of the effect for any specified mean and centered variance (Eq. C.25). > [...] how much of the measured norm effect comes from the mean or radial channel versus the noise correlation channel? We have not separately quantified these contributions in the LLM runs. The new extension nevertheless establishes that, with a fixed nonzero input mean and positive centered variance, increasing the slow amplitude still increases the stationary radius and decreases the relative-update scale. The predicted mechanism therefore extends beyond the zero-mean setting. > Fig. 15a shows non-negligible radial alignment between gradients and weights. We added raw-gradient measurements across the 60 targeted matrices. After warmup, local loss sensitivity to rescaling is small and the gradients are predominantly tangential, with the radial fraction decreasing during training (Figure 19a–b). We also checked the immediate effect of the two norm transfers in 24 before-and-after comparisons. The corrections ranged from −0.04% to +0.07%, with a maximum absolute loss change of $2.5\times10^{-4}$ and maximum output KL divergence of $3.5\times10^{-4}$. These measurements support local approximate scale invariance along training (Appendix F.5). > [...] the execution-kernel approximation [...] freezes the preconditioner at injection time [...] and this error is never quantified. Thank you for this observation. We now compare the accumulated displacement after weight decay produced by the modeled and actual updates. Across the recorded blocks, the discrepancy in their squared magnitudes stays within about 4% under constant and cosine LR (Figure 20, Appendix F.6). This supports using the approximation to analyze accumulated update effects. --- ## Rotational scheduling and the bound > The scheduling quantification rests on an anonymous, unverifiable companion submission. The cited submission, *From Learning Rates to Rotational Schedules*, is concurrently under review at ICLR and available at [this link](https://openreview.net/forum?id=qS8NmjrgrL). We have now included more details about this reference in the manuscript. > [...] using the value of an upper bound as a score across policies, where $M$ and $C_T(\omega)$ change with the policy, is a heuristic whose justification is never argued. [Schaipp et al. (2025)](https://arxiv.org/abs/2501.18965) show that convex last-iterate bounds reproduce empirical scheduling behavior and guide learning-rate design in LLM pretraining. Following this approach, we use the bound to compare the rotational schedules generated by our memory model. In Figure 4, we hold the problem constants fixed and optimize the overall scale separately for each profile. This gives an explicit comparison based on $\sqrt{C_T(\omega)}$, accounting for the different overall scales of the schedules (Appendix E.3). We have clarified this interpretation and procedure in the revised manuscript. --- ## Conclusions *Thank you again for your questions. We hope these additions and clarifications answer your comments, and we remain at your disposal for further requests or suggestions.* --- **[2/2]**

I have no more questions

Reviewer_dC8GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work analyzes why the "slow momentum" in AdEMAMix shows interesting optimization dynamics, specifically through the lens of parameter norms and rotational dynamics. The main claim is that even if the slow momentum contributes only a small fraction to the instantaneous update power, it can significantly influence the long-term formation of the parameter norm via temporal correlations. By employing a "stationary norm identity," the authors disentangle the effects of instantaneous update power from those of lagged correlations. The authors interpret this phenomenon as an implicit cooling mechanism, where the increased parameter norm reduces the effective rotational stepsize. Also, this claim is supproted by experiments on a 124M-parameter language model including several ablation studies.

Critical strengths
  • The paper explains the formulation of norms by decoupling the temporal correlation between current update power and past updates.

  • The authors address the timescales of fast/slow EMAs and weight decay, demonstrating that improving "slow memory" is distinct from simply slowing down a single EMA.

  • The proposed claims are validated with various approaches such as norm transplant, removing radial updates, radius fixing, and disabling of the slow-channel. These experiments give some evidence the authors' claim.

Critical weaknesses

I have several concerns about the paper.

  1. Limited implications. The analysis suggesting that slow momentum increases the parameter norm while decreasing the rotational step size is impressive. However, it is not fully understandable why such dynamics lead to superior optimization, specifically, what advantages they offer in terms of convergence or generalization.

  2. Can AdamW reproduce the benefits through LR scheduling? If AdamW's performance improves significantly in the norm transplant experiments, could the authors replicate the same rotational dynamics observed in AdEMAMix simply by adjusting AdamW's LR scheduling? Such comparions are quite necessary to decide whether multi-timescale momentum itself provide any advantages.

  3. Scale-invariance. In practice, the projection matrices in LLM are not strictly scale-invariant generally, partly due to the presence of residual connections. Consequently, it remains unclear which component contribute to the performance gain.

  4. Gap between theory and practice. The main theory relies on assumptions such as temporally uncorrelated inputs and a linear filter approximation. Further validation should be needed to see whether these assumptions hold true during the training for real and how accurately the theory predicts the observed norm dynamics.

Questions for authors

Please refer to the weaknesses.

2 comments

Thank you for your thoughtful review and questions. We are glad that you appreciated the interventions supporting the mechanism we study. Your comments helped us clarify its optimization implications and strengthen the connection between the analysis and experiments.


Optimization Implications

[...] it is not fully understandable why such dynamics lead to superior optimization, specifically, what advantages they offer in terms of convergence or generalization.

We agree that the connection to optimization performance needed a clearer explanation. Slow memory gradually increases parameter norms and reduces rotational step sizes, supplying cooldown even at constant LR. It therefore acts partly as a scheduler. A useful schedule preserves enough motion early in training to make progress while reducing stochastic fluctuations near the endpoint. The revised Section 4.5 assesses this effect using a finite-horizon last-iterate bound (Anonymous et al.) as a surrogate criterion for the modeled rotational profiles, with details in Appendix E.

In Figure 4, AdEMAMix's profile continues cooling as AdamW's flattens and yields a smaller bound value after optimizing each profile's overall scale with common problem constants. This scheduling interpretation also agrees with the measured loss gaps. AdEMAMix's mean advantage is approximately 0.0740.074 under constant LR and 0.0300.030 under cosine, where the learning rate already supplies explicit cooldown (Table 2). The smaller advantage under cosine supports the contribution of endogenous scheduling to AdEMAMix's performance.


Learning-Rate Scheduling and Slow Memory

[...] could the authors replicate the same rotational dynamics observed in AdEMAMix simply by adjusting AdamW's LR scheduling?

Thank you for this question. Our Hyperball intervention addresses the role of LR scheduling by controlling angular step sizes directly. By fixing matrix radii and normalizing updates, it gives both optimizers the same turning-angle schedule on the constrained matrices when their updates are tangent and they share the same prescribed LR. Under this control, AdamH improves over AdamW, while AdEMAMixH worsens relative to AdEMAMix, reducing the slow-memory advantage from approximately 0.0300.030 to 0.0090.009 (Figure 1c, Table 2, Appendix F.3).

This reduction by roughly two thirds supports norm-mediated scheduling as a major driver of AdEMAMix's observed advantage. The remaining benefit suggests complementary effects of slow memory, including its influence on update directions under this common control. While we have not yet run the LR-only comparison, the Hyperball results suggest that explicit LR scheduling would recover part of AdEMAMix's advantage, but reproducing the full effect may require accounting for the directional dynamics of memory.


[1/2, continues below]

[...continues]


Modeling Assumptions and Empirical Validation

The main theory relies on assumptions such as temporally uncorrelated inputs and a linear filter approximation. Further validation should be needed to see whether these assumptions hold true during the training for real and how accurately the theory predicts the observed norm dynamics.

In practice, the projection matrices in LLM are not strictly scale-invariant generally, partly due to the presence of residual connections. Consequently, it remains unclear which component contribute to the performance gain.

Our analysis follows the scale-invariant optimization literature, which studies how parameter norms regulate rotational motion in normalized networks. To predict these norm dynamics, the literature uses simplified models of gradients and optimizer updates. Kosson et al. use a noise-dominated random walk, Hyperball uses independent Gaussian inputs and idealized preconditioning, and AdamC uses equilibrium reasoning with slowly varying preconditioning. These models have proved useful beyond their exact theoretical settings, as illustrated by AdamC's improvements. We build on this approach to isolate how memory generates update correlations and norm growth. The predictive success of these models despite their simplifications is striking. Understanding why they capture aspects of LLM training so well remains an interesting open question. Kosson et al. (2024), Wen et al. (2026), Defazio (2025)

The general stationary norm identity in Theorem 4.3 allows arbitrary temporal correlations in the applied updates. We introduce uncorrelated inputs to isolate the correlations generated by optimizer memory. The new Corollary C.7 extends the cooling result to a fixed nonzero input mean with temporally uncorrelated centered fluctuations of positive variance.

We assess this approach in our Llama experiments by checking the immediate functional effect of norm rescaling, the accuracy of the preconditioner approximation, and the predicted norm relaxation after removing slow memory.

We measured local loss sensitivity to rescaling and the radial fraction of raw gradients across all 60 targeted matrices. After warmup, sensitivity is small and gradients are predominantly tangential, supporting local approximate scale invariance along training (Figure 19a–b).

To check the effect of the actual interventions, we evaluated the same 32,768 held-out tokens immediately before and after rescaling at six points in each of four transplant runs. The individual norm corrections ranged from −0.04% to +0.07%. Across these 24 checks, the largest absolute loss change was 2.5×10−42.5\times10^{-4} and the largest output KL divergence was 3.5×10−43.5\times10^{-4} (Appendix F.5).

To predict how the norms evolve during optimization, our model expresses the optimizer updates through a linear-filter approximation. We checked this approximation against recorded LLM updates, finding discrepancies within about 4% in squared accumulated displacement after weight decay across the recorded blocks under constant and cosine LR (Figure 20, Appendix F.6).

The controlled experiments also test the predicted norm responses (Figures 3 and 6), while the slow-channel switch tests the predicted weight-decay relaxation of the excess squared norm (Figure 5d).

These checks support the geometric interpretation of our norm interventions and the approximation used to model accumulated updates.


Conclusions

We hope these clarifications and additional measurements address your concerns. Thank you again for your helpful questions, and we remain at your disposal for further comments or suggestions.


[2/2]

Reviewer_iue2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why adding a slow momentum component can improve optimization. It argues that temporal correlations in updates increase parameter norms, reducing the update size relative to the parameters. The authors support this explanation with theoretical analysis and controlled interventions.

Critical strengths
  1. The analysis provides a useful explanation of how momentum memory affects parameter norms and relative update sizes.

  2. Several interventions, including transferring parameter norms between optimizers, provide evidence for the proposed explanation.

Critical weaknesses
  1. The main language-model experiments use one model size and a single seed, which limits confidence in the quantitative conclusions.

  2. Changing parameter norms may also change model outputs and subsequent gradients. The interventions therefore do not fully isolate the proposed mechanism.

Questions for authors
  1. Are the main findings consistent across additional random seeds?

  2. How much do model outputs or validation loss change immediately after norm rescaling, before any further training?

1 comment

Thank you for the careful review and thoughtful questions. We are glad to read that you found our paper clear and that it helped you gain a deeper understanding of the role of momentum memory.

In response to your remarks, we have added experiments and clarifications to the paper, which we elaborate below. We hope that these additions address your concerns and increase confidence in our conclusions.


Experimental Setup

Are the main findings consistent across additional random seeds?

In light of your comment and question, we repeated the main optimizer comparisons, norm transfer experiments, and Hyperball comparison on two additional seeds, keeping the selected hyperparameters fixed. These runs confirm the original findings, with small variation across seeds. The validation-loss differences below are reported as paired mean ±\pm sample standard deviation over the three seeds.

  • AdEMAMix reached a lower final validation loss than AdamW in all three pairs under both constant and cosine learning rates. The gap was 0.074±0.0020.074 \pm 0.002 under constant rates and 0.030±0.0010.030 \pm 0.001 under cosine (Table 2).

  • Giving AdamW the AdEMAMix norm trajectory lowered its validation loss in every pair, by 0.063±0.0010.063 \pm 0.001, recovering about 85% of the mean constant-rate gap. Giving AdEMAMix the AdamW trajectory increased its loss in every pair, by 0.064±0.0030.064 \pm 0.003, removing about 87% of its mean advantage (Figure 1b and Table 2).

  • The Hyperball comparison also reproduced the original finding in every pair. Under cosine LR, fixing the radii and normalizing the updates reduced the slow-memory advantage from 0.030±0.0010.030 \pm 0.001 to about 0.0090.009 (paired standard deviation 4×10−54\times10^{-5}) at the selected settings, leaving about one third of the original gap (Figure 1c, Table 2, and Appendix F.3).


Scale Invariance and Norm Rescaling

Changing parameter norms may also change model outputs and subsequent gradients. The interventions therefore do not fully isolate the proposed mechanism.

How much do model outputs or validation loss change immediately after norm rescaling, before any further training?

Thank you for this interesting question. Scale invariance is an established starting point for analyzing optimization in normalized networks. For example, Kosson et al. (2024) use it to explain how weight decay controls parameter norms and rotational motion. For a scale-invariant parameter group, positive rescaling preserves the model outputs and loss. The raw gradient with respect to that group scales inversely with its norm, preserving its direction. This is the geometric property motivating our norm interventions.

Since in our Llama model, scale invariance is approximate, we directly measured the immediate effect of rescaling. We evaluated the same 32,768 held-out tokens immediately before and after rescaling at six points in each of four runs, covering both directions of norm transfer. The individual norm corrections ranged from −0.04% to +0.07%. Across the 24 checks, the largest absolute change in loss was 2.5×10−42.5 \times 10^{-4} and the largest KL divergence between the output distributions was 3.5×10−43.5 \times 10^{-4} (Appendix F.5). Thus, at the audited updates, the rescaling steps produced very small immediate changes in predictions and validation loss.

We also added measurements of local loss sensitivity to rescaling and the radial fraction of the raw gradient across the 60 targeted matrices. After warmup, the loss sensitivity is small and the gradients are predominantly tangential, with the radial fraction decreasing during training. These observations further support approximate scale invariance along training (Figure 19a–b, Appendix F.5).


Conclusions

We hope that these clarifications and additional experiments have addressed your concerns and strengthened confidence in our conclusions. Thank you again for your helpful suggestions. We remain at your disposal for any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their comments and constructive feedback. We are pleased that all three reviewers recommend acceptance and recognize the value of our explanation of how optimizer memory shapes parameter norms and rotational dynamics.

Your comments helped us strengthen the evidence and clarify the analysis. The main revisions address the following points.

  • Consistency across seeds. We repeated the main optimizer comparisons, both norm transfers, and the Hyperball comparison on two additional seeds. The findings hold across all three seeds, with small variation. We also quantify the remaining slow-memory benefit under Hyperball (Table 2, Appendix F.3).

  • Effects of norm rescaling. We added measurements of predictions and loss immediately before and after transplant steps, finding small changes at the audited updates. New loss-sensitivity and raw-gradient measurements further support local approximate scale invariance (Appendix F.5).

  • Modeling assumptions. We extend the cooling result to a fixed nonzero input mean with temporally uncorrelated centered fluctuations of positive variance (Corollary C.7). We also check the effects of the preconditioner approximation (Appendix F.6).

  • Optimization implications. We clarify the use of the scheduling bound and expand the explanation of the loss dip and rebound after removing slow memory (Sections 4.5 and 5.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.