- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a new optimizer family Massive-Lion that unifies Lion, SS Adam. Experiments include Cifar10 on ResNet-18 and LLM pretraining on Fine-web.
Critical strengths
- The experiments are solid, LLMs though small, trained to (20x) chinchilla and with multiple seeds.
- I think the physical interperation is interesting, the direction is definitely worth more investiagtion and investment.
Critical weaknesses
- Given the formulation, the optimizer looks frankly more like Adam than Lion. System-wise it needs double momentum buffers. So naming it Lion is misleading. I acknowledge the mixed gradient part of Lion, but the core characteristic of Lion is the signed update + single momentum formualtion.
Questions for authors
Why SS AdamW as baseline? Also in SS AdamW paper, it explicitly states that it cannot beat a well-tuned AdamW baseline.
Theoretically, following Lion-K's Lyapunov analysis, for Adam family to be convergent,
Empirically, the practioner community does not use beta 1 = beta 2, it's rather odd to compare against this. if comparing against AdamW would have been much more convincing since we know the baseline is well-tuned on the SlimPajama dataset.
Learning rate scheduling, how does it interact with the training dynamic?
Maybe I missed it, but how does local stability translate to global convergence?
The AdaM-Lion name, the equal-β AdamW baseline, and the scope of the convergence guarantees
Thank you for the detailed review and for finding the direction worthwhile. One correction to the summary: the language models are trained on SlimPajama, not FineWeb. Changes in the revised PDF are marked in green.
The optimizer looks more like Adam than Lion
We agree, and we have renamed the adaptive optimizer AdaM-Lion (Adaptive-Mass Lion, which also reads as Adam + Lion). Like Adam, it stores two buffers, the momentum μt and the squared adaptive mass at2, and the revision says so. Its update direction is Lion's mixed direction dt, which a saturating map turns into sign(dt) in the massless limit. It contains Lion and SS-AdamW as exact special cases (Table 1), and the fixed-mass M-Lion and M-Signum keep Lion's single buffer.
Why SS-AdamW and β1=β2
A controlled baseline. SS-AdamW is the exact ζ=0 member of our family (Proposition 3.2), so comparing against it isolates curvature-aware dissipation.
Equal betas are near-optimal here. Our recipe follows the 160M SlimPajama benchmark of Orvieto & Gower (2025). After tuning Adam over 200 settings, they write:
Equal betas never cost more than 0.3 perplexity points in their batch-size study (Figure 3). They describe the standard (0.9,0.95) setting at 410M (Figure 5) as
They claim near-optimality, which is what a baseline needs. Several recent studies also default to β1=β2=0.95 (Zhao et al.; Shah et al.; Zhang et al., 2025).
Our results agree. At 162M, the best of 95 AdamW settings is only 0.3% below SS-AdamW, and at 411M and 1.06B tuned SS-AdamW beats tuned AdamW on fresh seeds. AdaM-Lion beats tuned AdamW at every size (global response).
The β condition and global convergence
Condition on β. This is the Hamiltonian-descent condition for Adam derived by the Lion-K authors (Liang et al., 2024, Example 2.3). With first- and second-moment rates a∝1−β1 and b∝1−β2, H=L+2a1∑imi2/(vi+ϵ) is non-increasing when a≥b/4. All our adaptive optimizers use equal rates (SS-AdamW: β1=β2; AdaM-Lion: β3=κ=βmom), so a=b and the condition holds with a factor-of-four margin. It also holds with our floor ρ02+vi in the denominator, so the ζ=0 members of our family inherit this Lyapunov function.
Global convergence. With fixed mass, our ODE is a Lion-K ODE (Chen et al., 2024) with K(p)=c∑iρ2+pi2, γ=1/τd and gradient enhancement ϵ=ζ. Their Theorem 3.1 needs ϵγ≤1, our assumption ζ<τd, and their discrete Theorem 4.1 needs βmix<βmom, i.e. ζ>0. Their global analysis therefore applies: with weight decay λ>0, λθt enters the box domK∗ and stays there, while the Lyapunov function decreases. Finite mass adds one property: for ρ>0, ∇K∗ is strictly monotone, so their stationarity result (Theorem 3.1, part 3) applies and every accumulation point is stationary for the regularized objective; for Lion (ρ=0) it does not.
Local stability. Theorem D.1 then describes behavior near a minimum: which step sizes and mixing coefficients settle and which oscillate. A global guarantee for AdaM-Lion, which combines ζ>0 with adaptive mass, needs a Lyapunov function merging the two constructions above; we list it as a limitation.
Learning-rate schedule
All compared optimizers share one cosine schedule. The learning rate is our per-step speed limit, η=cΔt, and the stability band of Theorem D.1 depends on the stiffness si=ηλi/ρ0 of each Hessian eigenvalue λi. Lowering η lowers every si, so sharp modes outside the stable band early in training move into it later. This holds in general; our analysis makes it explicit for finite mass. Lion and Signum move every coordinate by exactly η, so only the schedule can shrink their updates near a minimum; finite mass also shrinks them when ∣dt∣≪ρ (Figure 4a).