acceptodds
Under review as a conference paper at ICLR 2027

Sign-based optimizers are massless relativistic learners

Abstract

Sign-based optimizers are like photons: they travel at a constant speed and only change direction. This suggests a connection with special relativity. We derive Signum and Lion from relativistic learning dynamics with a *maximum speed of adaptation*, , for each parameter. The sign operation emerges as the massless (photonic) limit, while Lion's gradient mixing emerges from *curvature-aware dissipation* that damps motion more strongly along directions of high positive curvature. We further interpret equal- Adam as Signum with an adaptive mass determined by gradient variance. Combining this Adam-like adaptation with our finite-mass Lion derivation yields *Massive Lion*, which incorporates Lion's curvature-aware dissipation, variance-based mass adaptation, and a tunable finite rest mass. Its updates interpolate between an approximately linear, SGD-like regime and a saturated, sign-like regime. Experiments reveal the complementary roles of these mechanisms. On CIFAR-10 with ResNet-18, finite rest mass improves both sign-based and adaptive variants, bridging their gap with momentum SGD. Curvature-aware dissipation accelerates optimization on heterogeneous quadratic landscapes and improves language modeling. For a 12-layer transformer trained on 3.25 billion SlimPajama tokens, it consistently improves validation perplexity at high momentum across three learning rates. Our results show how physical reasoning explains relationships among existing optimizers, distinguishes the roles of their constituent mechanisms, and guides the design of new ones.

Primary area:
optimization
Keywords:
sign-based, physics, lion, signSGD, signum

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YsizOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new optimizer family Massive-Lion that unifies Lion, SS Adam. Experiments include Cifar10 on ResNet-18 and LLM pretraining on Fine-web.

Critical strengths
  1. The experiments are solid, LLMs though small, trained to (20x) chinchilla and with multiple seeds.
  2. I think the physical interperation is interesting, the direction is definitely worth more investiagtion and investment.
Critical weaknesses
  1. Given the formulation, the optimizer looks frankly more like Adam than Lion. System-wise it needs double momentum buffers. So naming it Lion is misleading. I acknowledge the mixed gradient part of Lion, but the core characteristic of Lion is the signed update + single momentum formualtion.
Questions for authors

Why SS AdamW as baseline? Also in SS AdamW paper, it explicitly states that it cannot beat a well-tuned AdamW baseline.

Theoretically, following Lion-K's Lyapunov analysis, for Adam family to be convergent, β1≥β24\beta_1 \geq \frac{\beta_2}{4}

Empirically, the practioner community does not use beta 1 = beta 2, it's rather odd to compare against this. if comparing against AdamW would have been much more convincing since we know the baseline is well-tuned on the SlimPajama dataset.

Learning rate scheduling, how does it interact with the training dynamic?

Maybe I missed it, but how does local stability translate to global convergence?

1 comment
AuthorsOpenReview ↗

The AdaM-Lion name, the equal-β AdamW baseline, and the scope of the convergence guarantees

Thank you for the detailed review and for finding the direction worthwhile. One correction to the summary: the language models are trained on SlimPajama, not FineWeb. Changes in the revised PDF are marked in green.

The optimizer looks more like Adam than Lion

W1: System-wise it needs double momentum buffers. So naming it Lion is misleading.

We agree, and we have renamed the adaptive optimizer AdaM-Lion (Adaptive-Mass Lion, which also reads as Adam + Lion). Like Adam, it stores two buffers, the momentum μt\mu_t and the squared adaptive mass at2a_t^2, and the revision says so. Its update direction is Lion's mixed direction dtd_t, which a saturating map turns into sign⁡(dt)\operatorname{sign}(d_t) in the massless limit. It contains Lion and SS-AdamW as exact special cases (Table 1), and the fixed-mass M-Lion and M-Signum keep Lion's single buffer.

Why SS-AdamW and β1=β2\beta_1 = \beta_2

Q1: Why SS AdamW as baseline?

Q3: The practioner community does not use β1=β2\beta_1 = \beta_2.

A controlled baseline. SS-AdamW is the exact ζ=0\zeta = 0 member of our family (Proposition 3.2), so comparing against it isolates curvature-aware dissipation.

Equal betas are near-optimal here. Our recipe follows the 160M SlimPajama benchmark of Orvieto & Gower (2025). After tuning Adam over 200 settings, they write:

"we recommend adopting β1=β2\beta_1 = \beta_2 as the default setting for Adam for training language models at similar data and parameter scales."

Equal betas never cost more than 0.3 perplexity points in their batch-size study (Figure 3). They describe the standard (0.9,0.95)(0.9, 0.95) setting at 410M (Figure 5) as

"quite suboptimal"

They claim near-optimality, which is what a baseline needs. Several recent studies also default to β1=β2=0.95\beta_1 = \beta_2 = 0.95 (Zhao et al.; Shah et al.; Zhang et al., 2025).

Our results agree. At 162M, the best of 95 AdamW settings is only 0.3% below SS-AdamW, and at 411M and 1.06B tuned SS-AdamW beats tuned AdamW on fresh seeds. AdaM-Lion beats tuned AdamW at every size (global response).

The β\beta condition and global convergence

Q2: For Adam family to be convergent, β1≥β2/4\beta_1 \geq \beta_2/4.

Q5: How does local stability translate to global convergence?

Condition on β\beta. This is the Hamiltonian-descent condition for Adam derived by the Lion-K\mathcal{K} authors (Liang et al., 2024, Example 2.3). With first- and second-moment rates a∝1−β1a \propto 1 - \beta_1 and b∝1−β2b \propto 1 - \beta_2, H=L+12a∑imi2/(vi+ϵ)H = L + \frac{1}{2a}\sum_i m_i^2/(\sqrt{v_i} + \epsilon) is non-increasing when a≥b/4a \geq b/4. All our adaptive optimizers use equal rates (SS-AdamW: β1=β2\beta_1 = \beta_2; AdaM-Lion: β3=κ=βmom\beta_3 = \kappa = \beta_{\mathrm{mom}}), so a=ba = b and the condition holds with a factor-of-four margin. It also holds with our floor ρ02+vi\sqrt{\rho_0^2 + v_i} in the denominator, so the ζ=0\zeta = 0 members of our family inherit this Lyapunov function.

Global convergence. With fixed mass, our ODE is a Lion-K\mathcal{K} ODE (Chen et al., 2024) with K(p)=c∑iρ2+pi2\mathcal{K}(p) = c\sum_i\sqrt{\rho^2 + p_i^2}, γ=1/τd\gamma = 1/\tau_d and gradient enhancement ϵ=ζ\epsilon = \zeta. Their Theorem 3.1 needs ϵγ≤1\epsilon\gamma \leq 1, our assumption ζ<τd\zeta < \tau_d, and their discrete Theorem 4.1 needs βmix<βmom\beta_{\mathrm{mix}} < \beta_{\mathrm{mom}}, i.e. ζ>0\zeta > 0. Their global analysis therefore applies: with weight decay λ>0\lambda > 0, λθt\lambda\theta_t enters the box dom⁡K∗\operatorname{dom}\mathcal{K}^\ast and stays there, while the Lyapunov function decreases. Finite mass adds one property: for ρ>0\rho > 0, ∇K∗\nabla\mathcal{K}^\ast is strictly monotone, so their stationarity result (Theorem 3.1, part 3) applies and every accumulation point is stationary for the regularized objective; for Lion (ρ=0\rho = 0) it does not.

Local stability. Theorem D.1 then describes behavior near a minimum: which step sizes and mixing coefficients settle and which oscillate. A global guarantee for AdaM-Lion, which combines ζ>0\zeta > 0 with adaptive mass, needs a Lyapunov function merging the two constructions above; we list it as a limitation.

Learning-rate schedule

Q4: Learning rate scheduling, how does it interact with the training dynamic?

All compared optimizers share one cosine schedule. The learning rate is our per-step speed limit, η=cΔt\eta = c\Delta t, and the stability band of Theorem D.1 depends on the stiffness si=ηλi/ρ0s_i = \eta\lambda_i/\rho_0 of each Hessian eigenvalue λi\lambda_i. Lowering η\eta lowers every sis_i, so sharp modes outside the stable band early in training move into it later. This holds in general; our analysis makes it explicit for finite mass. Lion and Signum move every coordinate by exactly η\eta, so only the schedule can shrink their updates near a minimum; finite mass also shrinks them when ∣dt∣≪ρ|d_t| \ll \rho (Figure 4a).

Reviewer_tzVBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a physics-inspired framework for understanding sign-based optimizers through relativistic dynamics. The authors show that Signum and Lion can be recovered as massless limits, while Lion's gradient mixing can be interpreted through curvature-aware dissipation. Building on these connections, they introduce Massive Lion (M-Lion), which combines smooth clipping, variance-based adaptation, and gradient mixing. The paper also provides a local stability analysis and experiments on quadratic problems, CIFAR-10, and language modeling. The results suggest that finite mass can improve optimization stability, while curvature-aware dissipation can improve performance in some settings.

Critical strengths

I find the connection between relativistic dynamics and sign-based optimization interesting. The paper provides a coherent way to understand the relationships between Signum, Lion, and equal-beta Adam.

The paper includes a reasonable set of ablations to study the roles of different components. I also appreciate the local stability analysis, which helps explain some of the observed optimization behavior.

Critical weaknesses

I am not fully convinced that the relativistic interpretation provides a unique advantage for optimizer design. In particular, the ablation in Table 4 shows no clear performance difference between the relativistic clipping function and alternatives such as tanh and arctan. This makes it difficult to tell how much of the improvement comes specifically from the proposed physical framework.

The experimental comparisons are somewhat limited. The language modeling experiments mainly compare against equal-beta AdamW, rather than standard AdamW or Lion with independently tuned hyperparameters. Also, the rest mass is set to zero in these experiments, so the benefit of the full M-Lion formulation remains unclear.

Questions for authors
  1. Given the results in Table 4, what is the main advantage of the relativistic formulation over a more general smooth clipping framework? Is there any theoretical or empirical property that specifically distinguishes the proposed update from other smooth clipping functions?

  2. Could the authors clarify how M-Lion compares with standard AdamW and Lion under comparable hyperparameter tuning budgets? In particular, does introducing a positive rest mass improve language modeling performance when combined with curvature-aware dissipation and adaptive mass?

1 comment
AuthorsOpenReview ↗

Clipping ablations, baseline tuning, and the roles of rest mass and adaptive mass in language modeling

Thank you for the positive assessment and the concrete questions. Changes in the revised PDF are marked in green.

What distinguishes the relativistic update from other smooth clipping maps

W1: I am not fully convinced that the relativistic interpretation provides a unique advantage for optimizer design and Q1: Given the results in Table 4, what is the main advantage of the relativistic formulation over a more general smooth clipping framework?

The distinguishing property is theoretical: the relativistic update is the one that combines with variance-based adaptation into Adam. Rest mass and adaptive mass add in quadrature, ρt2=ρ02+at2\rho_t^2 = \rho_0^2 + a_t^2, as rest energy and momentum combine in E2=(mc2)2+(pc)2E^2 = (mc^2)^2 + (pc)^2. Proposition 3.2 shows that μt2+at2=vt\mu_t^2 + a_t^2 = v_t, Adam's second moment, so at βmix=βmom\beta_{\mathrm{mix}} = \beta_{\mathrm{mom}} and ρ0=0\rho_0 = 0 the update μt/μt2+at2\mu_t/\sqrt{\mu_t^2 + a_t^2} is exactly SS-AdamW's μt/vt\mu_t/\sqrt{v_t}. With tanh and arctan, the same reduction gives tanh⁡(μt/at)\tanh(\mu_t/a_t) and (2/π)arctan⁡(πμt/(2at))(2/\pi)\arctan(\pi\mu_t/(2a_t)), which are not Adam. Only the relativistic map gives one optimizer that contains Signum, Lion and SS-AdamW as special cases. In Table 4, tanh and arctan receive the same adaptive mass, so only the shape changes.

The framework is meant to explain where the gains come from. It derives Lion's gradient mixing from curvature-aware dissipation, with the ordering βmix≤βmom\beta_{\mathrm{mix}} \leq \beta_{\mathrm{mom}}, and the language-model gains come from this mechanism (revised Table 2 and Figure 2c-e). Table 4 shows that once the mechanisms are fixed, the shape of the map matters little, which is the separation of mechanisms the paper argues for. We will discuss this in the main text.

Standard AdamW and Lion under comparable tuning budgets

W2: The language modeling experiments mainly compare against equal-beta AdamW... and Q2: Could the authors clarify how M-Lion compares with standard AdamW and Lion under comparable hyperparameter tuning budgets?

We added independently tuned conventional AdamW at 162M, 411M and 1.06B parameters (global response, revised Table 2). AdaM-Lion (our new name for the adaptive M-Lion) is 1.1%, 4.3% and 8.0% below AdamW at the three sizes, and at 411M it wins all six fresh paired seeds.

Tuning budgets. The budgets were not identical. At 411M and 1.06B, the selection pools held 44-45 AdaM-Lion, 19-20 SS-AdamW and 12-14 AdamW settings (Appendix E), and at 162M AdamW had 95 settings. The matched curves in Figure 2c-e do not depend on tuning: at each learning rate they change only ζ\zeta, and AdaM-Lion has lower mean perplexity at every one. Matching the budgets for all optimizers and sizes is the first item in our plan for the final version.

Lion and Signum. Only four Lion and three Signum settings on our final data pipeline finished at 162M, so we do not report them as a tuned comparison. The best reached 21.718 and 21.616 perplexity, against 20.277 for tuned AdamW and 20.045 for AdaM-Lion. This agrees with Orvieto & Gower (2025): in the same 160M benchmark, Signum tuned over 65 configurations closed 96% of the gap between SGD and Adam, yet Adam reached the same perplexity with 3/4 of the training budget. Full Lion and Signum grids will be in the final version.

Rest mass in language modeling

Q2: Does introducing a positive rest mass improve language modeling performance when combined with curvature-aware dissipation and adaptive mass?

In our language-modeling setting it does not, because rest mass and adaptive mass play different roles.

CIFAR-10. Training runs for about 455 epochs over 45,000 images, so overfitting matters. Positive rest mass improves test accuracy for Lion, Signum and SS-AdamW (Figure 4), and the validation-selected rest mass grows with training length, from about 10−310^{-3} at early cutoffs to between 0.10.1 and 1010 by the end (Appendix E.4, Figure 5). This is consistent with rest mass limiting overfitting.

Language models. Each token is seen once, with 3 to 20 training tokens per parameter, so the models stay in an underfitting regime. In runs we did not report, a positive rest mass did not improve perplexity. The adaptive mass already sets a finite mass scale for every coordinate: at the final checkpoint of a 411M and a 1.06B AdaM-Lion run, the median ratio of update direction to adaptive mass lies between 0.0650.065 and 0.0750.075 in all five parameter groups, although the directions and masses themselves differ between groups by factors of 56 to 69 (new Appendix G, Figure 6). A typical coordinate therefore moves in the near-linear regime. The ratio is consistent enough across layers and scales that we plan to investigate its theory for the final version.

Reviewer_QjaFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper interprets Signum and Lion through relativistic dynamics and combines smooth saturation, gradient mixing, and variance adaptation into Massive Lion.

Critical strengths
  • The main derivations and fixed-mass local stability analysis appear mathematically sound.
  • The framework provides an intuitive account of relationships among existing optimizers.
Critical weaknesses
  • The novelty needs sharper positioning. Similar work such as VRAdam(ICLR 2026) has not been cited.
  • It seems like the real-task comparisons do not include independently tuned conventional AdamW.
  • The clipping-map ablation finds no clear advantage for the relativistic response over tanh.
Questions for authors
  • What is new beyond the explicit relativistic Lion-K\mathcal K specialization, both algorithmically and theoretically?
  • Do the improvements persist against independently tuned conventional AdamW with a tuned denominator floor?
1 comment
AuthorsOpenReview ↗

Deriving Lion’s gradient mixing and extending Lion-𝒦 with adaptive mass

Thank you for the careful review and for confirming that the derivations and the fixed-mass stability analysis are sound. Changes in the revised PDF are marked in green.

Novelty beyond Lion-K\mathcal{K}, and VRAdam

W1: "Similar work such as VRAdam(ICLR 2026) has not been cited." Q1: "What is new beyond the explicit relativistic Lion-K\mathcal{K} specialization, both algorithmically and theoretically?"

VRAdam. The original submission already cites VRAdam in Section 2.2, alongside França et al. (2020). VRAdam adds a quartic kinetic term and implements it as one global gate on AdamW's learning rate, ηt=α0/(1+min⁡(β3∥vt∥2,α1))\eta_t = \alpha_0/(1 + \min(\beta_3\lVert v_t\rVert^2, \alpha_1)). Our relativistic Lagrangian instead sets a speed limit for each coordinate, and inverting it exactly gives the saturating response whose massless limit is the sign update. VRAdam does not recover Signum or Lion, and it has no gradient mixing or curvature-aware dissipation. We will add this contrast to the related work.

Beyond Lion-K\mathcal{K}. The Lion paragraph of our introduction names the two gaps we address.

  • Theory. Lion-K\mathcal{K} is defined by its dynamics, and its authors write that "we lack a 'physical intuition' or constructive derivation like the standard optimization algorithms" (Chen et al., 2024, Section 6). We derive the dynamics from a relativistic Lagrangian with curvature-aware dissipation. With fixed mass, our ODE is the Lion-K\mathcal{K} ODE with K(p)=c∑iρ2+pi2\mathcal{K}(p) = c\sum_i\sqrt{\rho^2 + p_i^2}, the relativistic energy (ρ=mc\rho = mc), and with Lion-K\mathcal{K}'s gradient-enhancement coefficient ϵ\epsilon equal to our curvature coefficient ζ\zeta. The derivation therefore explains both K\mathcal{K} and Lion's gradient mixing, including the ordering βmix≤βmom\beta_{\mathrm{mix}} \leq \beta_{\mathrm{mom}} (Theorem 3.1), and yields an exact local stability band for finite mass (Theorem D.1).
  • Algorithm. Lion-K\mathcal{K} has a fixed K\mathcal{K} and no adaptive counterpart of Lion analogous to SS-Adam for Signum. In AdaM-Lion the saturation scale ρt2=ρ02+at2\rho_t^2 = \rho_0^2 + a_t^2 adapts over time and across coordinates, which places it outside Lion-K\mathcal{K}, and its equal-β\beta reduction is exactly SS-AdamW (Proposition 3.2). It is a new optimizer that combines Adam's variance adaptation with Lion's gradient mixing, and it improves on tuned AdamW and tuned SS-AdamW from 162M to 1.06B parameters (global response).

Independently tuned AdamW with a tuned denominator floor

W2: "...independently tuned conventional AdamW" and Q2: "Do the improvements persist against independently tuned conventional AdamW with a tuned denominator floor?"

Yes. With (β1,β2,η)(\beta_1, \beta_2, \eta) tuned separately at each size, AdamW reaches 20.277, 18.072 and 17.221 perplexity at 162M, 411M and 1.06B, against 20.045, 17.288 and 15.847 for AdaM-Lion (revised Table 2). At 162M this is the best of 95 AdamW settings, with β1\beta_1 from 0.8 to 0.9875, β2\beta_2 from 0.6 to 0.996875 and η\eta from 0.001 to 0.016.

Denominator floor. In our family the floor is the rest mass, added in quadrature: at βmix=βmom\beta_{\mathrm{mix}} = \beta_{\mathrm{mom}} the update is μt/ρ02+vt\mu_t/\sqrt{\rho_0^2 + v_t}, which is SS-AdamW with floor ρ0\rho_0. On CIFAR-10 the tuned floor improves SS-AdamW (Figure 4b), while in our language-model runs a positive floor did not help (see tzVB-3). For conventional AdamW we kept PyTorch's ϵ=10−8\epsilon = 10^{-8} and did not tune it. In the same 160M benchmark, Orvieto & Gower (2025) report similar validation perplexity for every ϵ\epsilon from 10−1510^{-15} to 10−610^{-6} (their Table 2). We will add an ϵ\epsilon sweep at every size in the final version.

The clipping-map ablation shows no advantage over tanh

W3: "The clipping-map ablation finds no clear advantage for the relativistic response over tanh."

We agree with this reading of Table 4. The paper does not claim that the relativistic shape is a better saturating function. Table 4 holds the mechanisms fixed (rest mass, adaptive mass and curvature-aware dissipation), changes only the shape, and finds no difference, which places the gains in the mechanisms.

We prefer the relativistic map because it is the only one of the three whose adaptive form contains Adam. Rest mass and adaptive mass add in quadrature, ρt2=ρ02+at2\rho_t^2 = \rho_0^2 + a_t^2, and μt2+at2=vt\mu_t^2 + a_t^2 = v_t (Proposition 3.2), so the equal-β\beta, massless update μt/μt2+at2\mu_t/\sqrt{\mu_t^2 + a_t^2} equals SS-AdamW's μt/vt\mu_t/\sqrt{v_t}. With tanh or arctan, the same reduction gives tanh⁡(μt/at)\tanh(\mu_t/a_t) or (2/π)arctan⁡(πμt/(2at))(2/\pi)\arctan(\pi\mu_t/(2a_t)), which is not Adam. We will discuss this in the main text.

Comments on the paper

1 comment
AuthorsOpenReview ↗

AdaM-Lion outperforms tuned AdamW in language model pretraining (162M to 1.06B parameters)

We thank the reviewers for their careful reading. They found the derivations and the fixed-mass stability analysis sound (QjaF), the account of how Signum, Lion and equal-β\beta Adam relate coherent (QjaF, tzVB), and the language-model experiments solid, with several seeds at 20×20\times Chinchilla (Ysiz).

Main new result: Comparison with independently tuned AdamW (all reviewers)

All three reviewers asked for a comparison with independently tuned conventional AdamW. We ran it at the paper's 162M scale and at two larger scales, 411M and 1.06B parameters, with the same data, token budget and schedule. Our adaptive optimizer, now called Adaptive-Mass Lion (AdaM-Lion, see Ysiz-1), reaches lower perplexity than both tuned AdamW and tuned SS-AdamW at all three sizes. The table reproduces the perplexities in the revised Table 2: final validation perplexity after 3.25B SlimPajama tokens (mean ±\pm sample SD; lower is better).

Model sizeSeedsAdamWSS-AdamWAdaM-Lion
162M720.277†20.277^{\dagger}20.344±0.06920.344 \pm 0.06920.045±0.068\mathbf{20.045} \pm 0.068
411M6 fresh18.072±0.42518.072 \pm 0.42517.700±0.05317.700 \pm 0.05317.288±0.148\mathbf{17.288} \pm 0.148
1.06B2 fresh17.221±0.62317.221 \pm 0.62316.148±0.00916.148 \pm 0.00915.847±0.029\mathbf{15.847} \pm 0.029

†^{\dagger}Best of 95 (β1,β2,η)(\beta_1, \beta_2, \eta) settings, on one seed.

Each optimizer was tuned separately at each size. At 411M and 1.06B, the selected settings were frozen and rerun on fresh seeds with a random data order. AdaM-Lion is below AdamW by 1.1%, 4.3% and 8.0% at 162M, 411M and 1.06B, and below SS-AdamW by 1.5%, 2.3% and 1.9%. At 411M it wins all six fresh paired seeds against each baseline (Holm-adjusted p=0.014p = 0.014), and at 1.06B both seeds completed so far favor it. In matched comparisons that change only the curvature-aware dissipation ζ\zeta (revised Figure 2c-e), AdaM-Lion has lower mean perplexity than SS-AdamW at every tested learning rate and model size.

These results also support our choice of SS-AdamW as the main baseline. It is the exact ζ=0\zeta = 0 member of our family, so comparing against it isolates curvature-aware dissipation, and equal-β\beta AdamW is near-optimal in the benchmark our recipe follows (Orvieto & Gower, 2025). Independently tuned AdamW does not beat tuned SS-AdamW at 411M or 1.06B, and at 162M its best setting is only 0.3% lower.

Changes in the revised paper (in green)

  • AdamW and larger models. Table 2 now compares AdaM-Lion, SS-AdamW and conventional AdamW (PyTorch, with bias correction), each tuned separately at 162M, 411M and 1.06B parameters. Figure 2 now shows the matched learning-rate curves at all three sizes (panels c-e), and Appendix E reports the tuning grids, selection pools and seeds.
  • Name. The adaptive optimizer is now AdaM-Lion. Section 3 states that it stores two buffers, like Adam, while the fixed-mass M-Lion stores one, like Lion. The abstract, introduction and discussion are updated to match.
  • Adaptive mass. A new Appendix G (Figure 6) shows that at the final checkpoint of a 411M and a 1.06B run, the median ratio of update direction to adaptive mass lies between 0.0650.065 and 0.0750.075 in all five parameter groups, so a typical coordinate is in the near-linear regime.

Plans for the final version

  • Tune all baselines with budgets matched to AdaM-Lion at all three sizes, including AdamW's ϵ\epsilon and weight decay and full Lion and Signum grids, and complete the remaining 1.06B seeds.
  • Bring into the main text the points clarified in the individual responses: the contrast with VRAdam, the mapping onto Lion-K\mathcal{K} and its convergence results, and why the relativistic map is the one whose adaptive form contains Adam.
  • Develop the theory behind the adaptive-mass ratio, and a global convergence analysis of AdaM-Lion with both ζ>0\zeta > 0 and adaptive mass.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.