- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper explains why AdEMAMix, which adds an independently weighted slow momentum branch to AdamW, outperforms AdamW in LLM pretraining. Starting from the exact squared-norm recursion under decoupled weight decay (Lemma 4.1), the authors show that in stationarity the parameter norm is governed by the temporal correlation structure of the updates (Theorem 4.3), with the squared radius equal to eta^2 C_delta[u] over (1 minus delta^2), where C_delta[u] weights the correlation between updates h steps apart by delta^h. Writing an optimizer as an execution kernel over its input history separates instantaneous power P[e] from the contribution of correlated history to the norm, F_delta[e]. For the two-timescale AdEMAMix kernel the paper proves that the ratio P/F_delta decreases strictly in the slow amplitude alpha (Theorem 4.8), and that the benefit is non-monotone in the slow memory length with an interior optimum (Proposition C.7). The proposed mechanism is that the slow branch contributes only a few percent of instantaneous update power (3.0% and 4.1% under constant and cosine schedules respectively, Table 3) yet, through its long correlation tail, multiplies the norm by roughly 42, which inversely slows parameter rotation and thereby turns the nominal learning rate schedule into an optimizer-specific rotational schedule. The mechanism is tested in 124M Llama pretraining on FineWeb through four controlled interventions: norm transplantation, radial removal, HYPERBALL fixed-radius control, and mid-training switch-off of the slow channel. Table 3 gives a sharp diagnostic contrast, 3 percent of present power versus 84.6 percent of survived displacement for the slow branch. Sec. 4.5 and App. E additionally score scheduling effects with a convex spherical-cap last-iterate bound imported from an anonymous companion submission. I verified the key derivations step by step and found no mathematical errors. My concerns are about the strength of the empirical evidence and the causal identification, detailed below.
Critical strengths
- The mechanism theory is clean, verifiable, and provides a genuinely new decomposition. Separating current-step power from the norm contribution of correlated history via C_delta[u] and F_delta[e], together with the exact two-pole pairing formula (C.20), the monotonicity theorem (Theorem 4.8), and the non-monotonicity result (Proposition C.7), forms a complete and falsifiable account of the phenomenon. It also recovers known limiting cases, namely the beta-independent AdamW radius of Kosson et al. (2024) as delta approaches 1 and the (1+beta)/(1-beta) growth factor of Heo et al. (2021). It is uncommon in this literature for every appendix proof to survive line-by-line checking, and this substantially increases my trust in the mechanism.
- The intervention experiments and diagnostic measurements are well designed. The four interventions each isolate a different facet of the mechanism and are matched to theoretical predictions, and Table 3 is a textbook mechanism diagnostic. The interaction prediction of Expectation A, namely a larger memory benefit under constant rates (0.076) than under cosine (0.029), follows from the theory and is confirmed experimentally. The resulting practical recommendation to choose memory and learning rate schedules jointly, and to compare optimizers by their realized rotational trajectories, has value beyond this single paper.
Critical weaknesses
- All central LLM evidence comes from single runs with no variance reported. App. F.1 states that all compared configurations share the same random seed and data order, and Table 1 notes that each tuning sweep uses one seed. The random-walk testbed uses five replicas and the MNIST MLP uses five seeds, yet every main conclusion (the 0.076 and 0.029 gaps, the norm transplants, HYPERBALL, and the switch-off experiment) rests on single training runs. It is not possible to rule out that the 0.029 cosine gap lies within seed noise, and the central causal claim that the norm channel mediates the AdEMAMix advantage is built entirely on this evidence.
- The causal identification is incomplete, and the norm-mediation and effective-LR-reparameterization explanations are not separated. After norm transplantation the best constant learning rate moves from 1e-4 to 8e-4 (Table 2), which shows that the transplant mostly shifts the effective learning rate operating point of AdamW by about a factor of eight. This is exactly the authors' Expectation B, but it also means that "AdamW with the AdEMAMix norm recovers most of the gap" is observationally equivalent to "AdEMAMix simply realizes a better effective step size schedule endogenously." The HYPERBALL fixed-radius control in Fig. 1c is the one experiment that can distinguish the two, and it is summarized as the benefit shrinking to about one third, but no calculation or concrete numbers behind this factor appear anywhere in the paper, so a reader cannot verify this key quantification.
- A core assumption conflicts with the paper's own diagnostics. Assumption 4.4 (zero-mean, mutually uncorrelated inputs) is a noise-dominated regime, while Fig. 15a shows non-negligible radial alignment between gradients and weights. Equation (C.18) further shows that a nonzero input mean enters C_delta with weight (1+delta)/(1-delta), which is much larger than the delta^h weighting of the noise channel, so the dominant contribution to the equilibrium norm may come from the mean or radial channel that the main text barely discusses. In addition, the execution-kernel approximation for AdamW and AdEMAMix freezes the preconditioner at injection time and ignores the coupling of the shared divisor with the alpha_t schedule, and this error is never quantified. The model-to-LLM comparison in Fig. 7 is only qualitative, the input power is not fitted, and the authors themselves note differences in shape and timing.
- The scheduling quantification rests on an anonymous, unverifiable companion submission. Sec. 4.5 and App. E.1 to E.3 rely on the bound in (E.2) and on claims such as the optimal allocation approaching linear decay, all cited as "Anonymous authors, 2026, under review at ICLR 2027." Moreover, using the value of an upper bound as a score across policies, where M and C_T(omega) change with the policy, is a heuristic whose justification is never argued. Fortunately the main mechanism in Sec. 4.1 to 4.3 and Sec. 5.1 to 5.3 does not depend on this part, so this is a supporting rather than fatal issue.
Questions for authors
- Can the main experiments (the selected configurations in Tables 1 and 2, Figs. 1b, 1c, 4 and 5) be replicated over at least three seeds with means and variances reported? In particular, what is the seed standard deviation of the 0.0289 cosine gap?
- How exactly is the "one third" residual benefit under fixed radius quantified? Please provide the final loss gap between AdamH and AdEMAMIXH at their respective optimal alpha, its ratio to the free-radius AdamW versus AdEMAMix gap, and the calculation behind it.
- Does Theorem 4.8 survive once the input mean is included as in (C.18)? Does the sign of d(P/F_delta)/d_alpha and the magnitude of the radius gain remain, and how much of the measured norm effect comes from the mean or radial channel versus the noise correlation channel?
3 comments
[...continues]
Assumptions and the preconditioner approximation
Does Theorem 4.8 survive once the input mean is included as in (C.18)? Does the sign [...] and the magnitude of the radius gain remain [...]?
Theorem 4.3 already allows a constant nonzero update mean and arbitrary temporal correlations under its stationarity assumptions. Your question prompted us to extend the two-channel result as well. The new Corollary C.7 considers inputs with constant mean , centered variance , and zero temporal correlations between centered inputs. For nonzero input power and stationary modeled responses, it gives
\bar R^2(\alpha) &=\frac{\eta^2}{1-\delta^2} \left[\sigma_a^2F_\delta[e] +\frac{1+\delta}{1-\delta}\|\boldsymbol\mu\|^2(1+\alpha)^2\right],\\\\ q_{\bar R}^2(\alpha) &=(1-\delta^2) \frac{\sigma_a^2P[e]+\|\boldsymbol\mu\|^2(1+\alpha)^2} {\sigma_a^2F_\delta[e]+\frac{1+\delta}{1-\delta}\|\boldsymbol\mu\|^2(1+\alpha)^2}, \qquad e=e^{\mathrm{AMix}}. \end{aligned}$$ Holding the input statistics, LR, decay, and ordered fast and slow retentions fixed, increasing $\alpha$ strictly increases the radius and strictly decreases the relative-update scale whenever $\sigma_a^2>0$. For a pure mean input, $q_{\bar R}=1-\delta$ independently of $\alpha$. These formulas give the magnitude of the effect for any specified mean and centered variance (Eq. C.25). > [...] how much of the measured norm effect comes from the mean or radial channel versus the noise correlation channel? We have not separately quantified these contributions in the LLM runs. The new extension nevertheless establishes that, with a fixed nonzero input mean and positive centered variance, increasing the slow amplitude still increases the stationary radius and decreases the relative-update scale. The predicted mechanism therefore extends beyond the zero-mean setting. > Fig. 15a shows non-negligible radial alignment between gradients and weights. We added raw-gradient measurements across the 60 targeted matrices. After warmup, local loss sensitivity to rescaling is small and the gradients are predominantly tangential, with the radial fraction decreasing during training (Figure 19a–b). We also checked the immediate effect of the two norm transfers in 24 before-and-after comparisons. The corrections ranged from −0.04% to +0.07%, with a maximum absolute loss change of $2.5\times10^{-4}$ and maximum output KL divergence of $3.5\times10^{-4}$. These measurements support local approximate scale invariance along training (Appendix F.5). > [...] the execution-kernel approximation [...] freezes the preconditioner at injection time [...] and this error is never quantified. Thank you for this observation. We now compare the accumulated displacement after weight decay produced by the modeled and actual updates. Across the recorded blocks, the discrepancy in their squared magnitudes stays within about 4% under constant and cosine LR (Figure 20, Appendix F.6). This supports using the approximation to analyze accumulated update effects. --- ## Rotational scheduling and the bound > The scheduling quantification rests on an anonymous, unverifiable companion submission. The cited submission, *From Learning Rates to Rotational Schedules*, is concurrently under review at ICLR and available at [this link](https://openreview.net/forum?id=qS8NmjrgrL). We have now included more details about this reference in the manuscript. > [...] using the value of an upper bound as a score across policies, where $M$ and $C_T(\omega)$ change with the policy, is a heuristic whose justification is never argued. [Schaipp et al. (2025)](https://arxiv.org/abs/2501.18965) show that convex last-iterate bounds reproduce empirical scheduling behavior and guide learning-rate design in LLM pretraining. Following this approach, we use the bound to compare the rotational schedules generated by our memory model. In Figure 4, we hold the problem constants fixed and optimize the overall scale separately for each profile. This gives an explicit comparison based on $\sqrt{C_T(\omega)}$, accounting for the different overall scales of the schedules (Appendix E.3). We have clarified this interpretation and procedure in the revised manuscript. --- ## Conclusions *Thank you again for your questions. We hope these additions and clarifications answer your comments, and we remain at your disposal for further requests or suggestions.* --- **[2/2]**I have no more questions
Thank you for your detailed review and interesting observations. We are glad that you found the mechanism informative and the experiments well designed. Your questions prompted additional experiments and an extension of the theoretical analysis.
Empirical evidence
We repeated the native optimizer comparisons, both norm transfers, and the Hyperball comparison on two additional seeds, keeping the selected hyperparameters fixed. These runs confirm the original findings, with small variation across seeds. All loss differences below are paired mean ± sample standard deviation over three seeds.
The mean final losses are approximately 3.140 for AdamH and 3.131 for AdEMAMixH at the selected α=2. Their paired gap is about 0.009, with a standard deviation of 4×10−5. Relative to the free-radius cosine gap, this gives 0.009/0.030≃0.3, leaving about one third of the original benefit. We now provide the calculation in Appendix F.3, alongside the results in Figure 1c and Table 2. Appendix F.1 specifies the tuning protocol.
Interpretation of the norm interventions
Your observation that AdEMAMix "realizes a better effective step size schedule endogenously" captures the central idea of our paper. Our analysis identifies how multi-timescale memory generates this effective schedule through temporal correlations and norm growth, while the interventions test its contribution to the observed performance.
Each transplant configuration receives its own LR tuning on the common constant-rate grid. Giving AdamW the AdEMAMix norms both changes its preferred LR and improves the best loss reached across the grid (Figure 9, Appendix F.1).
Hyperball is a recently proposed optimization method that has demonstrated consistent improvements for AdamW and Muon in pretraining benchmarks. Applying it to our setting improves AdamW's performance, reproducing this established benefit. For AdEMAMix, however, it worsens performance and substantially narrows the advantage of slow memory (Figure 1c). This contrasting response supports our interpretation that AdEMAMix benefits from the norm dynamics that Hyperball removes.
These interventions thus support our conclusion that norm-induced rotational scheduling accounts for a substantial part of AdEMAMix's benefit.
[1/2, continues below]