acceptodds
Under review as a conference paper at ICLR 2027

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

Abstract

Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum, update geometry, and acceleration remain only partially understood. We develop an DMM-nspired omentum (AIM) framework based on residual-penalty variable splitting, which interprets momentum as a multiplier-like correction driven by the splitting residual. AIM recovers the exponential moving average of gradients from an ADMM-style multiplier update and separates three mechanisms that are often intertwined in practical optimizers: the residual penalty determines the update geometry, the multiplier update determines how gradient information is accumulated into momentum, and the approximation of the objective-related -subproblem determines the acceleration form. Building on AIM, we propose elativistic daptive gradient escent with ccelerated esidual (RADAR), which combines relativistic adaptive geometry, decoupled residual correction, and gradient-difference momentum filtering to improve the update direction and momentum estimation. We establish stochastic convergence through a variance-perturbed Lyapunov drift analysis. Experiments on supervised vision learning, language modeling, and reinforcement learning show that RADAR achieves consistent improvements over strong adaptive optimizer baselines.

Primary area:
optimization
Keywords:
Deep Learning Optimization, Momentum Methods, Adaptive Optimization, ADMM, Nesterov Acceleration, Multiplier

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fbLzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Instead of studying the original problem min⁡θL(θ)\min_\theta \mathcal{L}(\theta), this paper to add a simple constraint min⁡θ,yL(θ),s.t.y=θ.\min_{\theta,y}\mathcal{L}(\theta),\quad \text{s.t.} y=\theta. This new formulation motivates the use of ADMM-type algorithms. The authors propose to study the augmented Lagrangian of the form Lˉρ(θ,y,m)=L(θ)+⟨m,y−θ⟩+ρ2ψ(y−θ),\bar{L}_\rho(\theta,y,m)=\mathcal{L}(\theta)+\langle m,y-\theta\rangle+\frac{\rho}{2}\psi(y-\theta), where ψ\psi is closed and convex with ψ(0)=min⁡rψ(r)=0\psi(0)=\min_r\psi(r)=0. This leads to the ADMM-inspired Momentum (AIM) framework summarized in Algorithm 1.

Assuming that the θ\theta-subproblem is solved exactly, the authors recover a standard Heavy-Ball-like momentum update for mkm_k. Moreover, using a duality argument, the authors demonstrate that the mismatch between yky_k and θk\theta_k is linked to the mismatch between ∇L(θk)\nabla \mathcal{L}(\theta_k) and momentum mkm_k, through the conjugate geometry induced by ψ\psi.

Later, the authors demonstrate that by choosing ψ\psi and the accuracy of solving the θ\theta-subproblem recovers many popular algorithms like Muon, Adam, and NAdam. Based on this unified view of algorithm design, the authors propose the RADAR algorithm, which can be seen as some sort of modification of Adam and NAdam, inspired by the AIM framework.

The authors later test this algorithm in practice and demonstrate competitive performance across several tasks, including vision, language, and RL tasks. In the appendix, the authors also provide a convergence analysis under standard assumptions + bounded precondition for RADAR.

Critical strengths
  • The storyline is very clear and easy to follow. I enjoyed the idea of splitting variables and how it leads to the algorithm design. I also liked that the framework is quite general and covers many existing optimization methods.

  • The proposed algorithm is competitive in practice. It performs well across several tasks, demonstrating a universal behaviour.

  • The implementation looks correct to me. The authors also include most of the tuning details for reproducibility.

  • The authors also provide a convergence analysis with a novel Lyapunov function. It looks correct to me. Most steps are clearly explained. It establishes convergence of the form O(1/T+σ2/B)\mathcal{O}(1/T + \sigma^2/B) for the expected gradient norm.

Critical weaknesses
  • Though the AIM framework is clearly described, some parts of the algorithm design do not follow it. For example, I didn't find an explanation of momentum filtering. Why does RADAR use (22)? How does it appear in the algorithm? This step is unclear and looks like an unjustified heuristic. The authors did not explain why the preconditioner is chosen as in (19), but not a standard Adam-like one.

  • Some experiments look weird to me. In particular, I do not understand why some results are so noisy. For example, the performance of Lion and Adan is very inconsistent in the Qwen experiment. I also do not understand why, in the CIFAR-10 and CIFAR100 setting the authors report the test loss, not the test accuracy, as is typically done in the literature. I also find it weird that the loss dynamics in Figure 1 (left) have a weird drop around 7k steps. Finally, in most cases, the algorithms are statistically indistinguishable. These issues in the experimental comparison raise concerns about the validity of the reported performance metrics: most likely some algorithms are tuned well enough, while some training implementations went wrong (again, see Figure 1 left).

  • The authors should also mention that bias correction is used in the experiments as well, which is not covered in the analysis.

  • The authors do not compare against the SODA framework by [Pethick et al., 2026] that also studies a similar approach to viewing different algorithms from a special choice of regularization. The paper should provide a detailed comparison against that work.

Questions for authors
  • Could you explain the choice of warm-up steps? Why is it so different in Table 6?

  • Does the convergence analysis cover the experimental setting (i.e., the choice of hyperparameters, etc)?

  • It would be good to see also tuning results, to see that the chosen hyperparameters do not lie on the boundary, as well as why some algorithms get so noisy.

  • The gradient norm convergence of SGD has the form O(1/(γT)+γσ2/B)\mathcal{O}(1/(\gamma T) + \gamma \sigma^2/B). By taking γ\gamma to minimize the bound we arrive to O(1/T+σ/TB)\mathcal{O}(1/T + \sigma/\sqrt{TB}). In this paper, the convergence is O(1/T+σ2/B)\mathcal{O}(1/T + \sigma^2/B), which is basically using a constant γ\gamma in the very first bound. Can we get the second bound, which is an optimal convergence method for SGD?

18 comments
AuthorsOpenReview ↗

W1. Rationale for gradient-difference filtering and preconditioning geometry

Thank you for pointing out where the relationship between the framework and the concrete algorithm design needs further explanation. AIM distinguishes three interconnected mechanisms: the residual penalty determines the update geometry; the multiplier update or filtering rule determines how gradient information is accumulated; and the approximation of the objective-related parameter subproblem determines the form of the parameter correction. RADAR's relativistic adaptive geometry, gradient-difference momentum filtering, and decoupled residual correction correspond to these three design choices, respectively.

We agree that the gradient-difference filter in Eq. (22) of the original manuscript is not uniquely derived from the basic AIM framework. Rather, it is a specific design choice within the momentum-filtering mechanism identified by the framework. Its motivation is to use changes between consecutive gradients to correct the lag of a standard EMA in tracking changing gradients. In the revised manuscript, we have clearly distinguished relationships derived directly from the framework from the additional design choices made in RADAR, so that readers do not mistakenly infer that all components are uniquely determined by the same set of subproblems.

Regarding the choice of preconditioner, we adopt RAD's relativistic adaptive geometry because it has an explicit dynamical-systems rationale. RAD constructs its updates through conformal symplectic discretization, combining a relativistic speed-limiting mechanism with preservation of the dynamical system's structure, which motivates its use for improving long-term training stability. This also motivates our introduction of gradient-difference filtering and decoupled residual correction on top of this geometry: we seek to combine its potential stability benefits with improved gradient tracking and parameter updates. AIM itself is not restricted to this geometry; standard Adam-type adaptive geometry can also be incorporated into the framework. We have added this rationale to the revised manuscript and clarified which geometry RADAR inherits and which correction mechanisms it introduces.

Section 4.1 distinguishes the geometry, filtering, and residual-correction choices used in RADAR. Their roles within AIM are discussed in Section 3.4, and the scope of their algebraic equivalence is explained in Appendix A.5.

AuthorsOpenReview ↗

W2. Reliability of experimental results, evaluation metrics, and training curves

Thank you for raising these issues. We have reexamined the hyperparameter search and experimental settings and provide the following clarifications.

First, the poor performance of Adan and Lion in the original Qwen experiments may partly reflect the limited 1B-token training budget. To investigate this possibility, we increased the full training budget from 1B to 5B tokens and the learning-rate search budget from 0.1B to 0.5B tokens per candidate for all methods. The final validation perplexities on Qwen2.5-0.5B are reported below.

TaskMetricRADARRADAdamAdamWNAdamAdanAdaBeliefLion
FineWeb, Qwen2.5-0.5BValid PPL ↓21.698621.824922.311821.744727.541227.804621.926829.8632

Under the expanded protocol, Adan and Lion improve substantially, narrowing the gaps observed in the original experiments, although they remain less competitive than the other baselines. These observations suggest that the limited original budget may have contributed to their poor performance.

Second, regarding the evaluation metrics for CIFAR-10 and CIFAR-100, we agree with the reviewer's suggestion. Cross-entropy loss reflects how well the model fits predictive probabilities and captures changes during optimization, whereas classification accuracy directly measures the proportion of correct predictions and is an important and widely used metric for classification tasks. Reporting test loss alone is insufficient for a comprehensive evaluation. We have therefore also reported test accuracy in the revised manuscript to provide a more complete performance comparison.

TaskMetricRADARRADAdamAdamWNAdamAdanAdaBeliefLion
CIFAR-10, ViT (6.3M)Test Accuracy (%) ↑87.2600 ± 0.374086.4760 ± 0.055986.5340 ± 0.382286.2880 ± 0.216386.5300 ± 0.366387.6400 ± 0.266086.3260 ± 0.377987.5040 ± 0.3112
CIFAR-100, ResNet-50Test Accuracy (%) ↑76.5320 ± 0.319575.0440 ± 0.305776.3200 ± 0.160875.0920 ± 0.664676.2840 ± 0.381676.4680 ± 0.266774.8360 ± 0.395674.5480 ± 0.3774

Results are reported as mean ± sample standard deviation over five random seeds (0–4). Higher accuracy is better. The best mean accuracy in each row is highlighted in bold.

Third, the sharp loss drop at approximately 7,000 steps in the GPT-2 training curves coincides with an epoch transition. This experiment trains on WikiText-103 for approximately three epochs, and the full training trajectories shown in the inset plots of Figure 1 in the original manuscript exhibit a similar drop around 3,500–4,000 steps, corresponding to the preceding epoch transition. Similar epoch-boundary drops in training loss have been documented and studied as the “epochal sawtooth phenomenon” (Liu and Ma, 2025). These observations suggest that the drops are associated with epoch transitions rather than indicating sudden acceleration unique to our optimizer. We have clarified this phenomenon and provided the training-loss curves in Appendix B.3 ("Language Modeling").

Finally, we appreciate the reviewer's reminder about the statistical reliability of differences between optimizers. The original experiments include results from multiple random seeds. In the revised manuscript, we have reported their means, standard deviations, and numbers of seeds more clearly, and distinguished them from the new 5B-token experiments, which were conducted using a single seed. We also acknowledge that, on some tasks, the performance differences are small relative to the variation across seeds. These results do not support a claim that RADAR has a statistically significant advantage on every task. We have therefore adjusted our wording to conclude that RADAR is competitive on the tested tasks and achieves better average performance in some experiments, without interpreting every numerical improvement as statistically significant.

Section 5.2 reports the updated Qwen perplexities and vision test accuracies. Seed settings are summarized at the beginning of Appendix B; Appendix B.1 describes the search protocol, Appendix B.2 lists the selected hyperparameters, and Appendix B.3 provides the language-modeling configurations and training-loss curves.

AuthorsOpenReview ↗

W3. Clarification of bias correction & W4. Comparison with the SODA framework & Q1. Warmup settings across tasks

W3. Clarification of bias correction

Thank you for pointing out this omission. Our experimental implementation uses bias correction, which was not clearly stated in the original manuscript. Section 4.1 explicitly notes that the algorithm presentation uses the core recursion without bias correction. The scope statement in Appendix A.6 also clarifies that the convergence analysis concerns this uncorrected recursion. Appendix B.2 states that bias correction is applied in the experimental implementation.

W4. Comparison with the SODA framework

Thank you for bringing SODA, an excellent and closely related work, to our attention. We recognize the important connections between SODA and AIM in using dual structures to provide a unified understanding of optimizers, and have added a citation and discussion in the revised manuscript.

Starting from optimistic dual averaging, SODA provides a unified treatment of gradient averaging and optimism, update geometries induced by regularization functions, and parameter averaging and extrapolation. AIM shares the idea of using convex conjugacy to connect gradient information with update geometry. Specifically, letting r=y−θkr=y-\theta_k, AIM's yy-subproblem can be expressed as

rk+1∈argmin⁡r(⟨mk,r⟩+ρ1,k2ψ(r)),r_{k+1}\in\underset{r}{\operatorname{argmin}}\left(\langle m_k,r\rangle+\frac{\rho_{1,k}}{2}\psi(r)\right),

which implies

rk+1∈∂!(ρ1,k2ψ)∗(−mk).r_{k+1}\in\partial\\!\left(\frac{\rho_{1,k}}{2}\psi\right)^{\ast}(-m_k).

This expression directly demonstrates the connection between the two frameworks in their construction of geometry.

AIM's specific contribution is to explain the origins of momentum accumulation and parameter correction through variable splitting and the associated augmented Lagrangian structure. In particular, momentum admits a Lagrange-multiplier interpretation, with its update driven by the primal-variable residual. Combining this update with the first-order optimality condition of the exact θ\theta-subproblem yields a gradient EMA representation. At the same time, this subproblem connects the gradient–momentum discrepancy to the primal-variable residual, providing a basis for constructing residual corrections in parameter updates through subproblem approximations.

The two frameworks also overlap at the level of concrete algorithms. For example, under matched initialization and fixed coefficients, RADAR's gradient-difference momentum and residual correction can be rewritten as a mixture of the current gradient and a standard EMA, directly connecting it to optimistic gradient mixing in SODA. We therefore position AIM as an interpretive and design framework complementary to SODA: its distinctive feature is to connect momentum, update geometry, and parameter correction to a single split optimization problem through residual-driven multiplier updates and approximations of the objective-related subproblem.

The comparison with SODA is included in Section 2.1, under "Dual averaging and unified optimizer frameworks." Section 3.1 explains AIM's residual-driven multiplier interpretation, while Appendix A.5 discusses the gradient–EMA reparameterization of RADAR.

Q1. Warmup settings across tasks

Thank you for your question. Our general rationale for warmup is to provide a transition period for the model parameters and optimizer states to adapt during early training. Gradually increasing the learning rate avoids immediately applying the full update magnitude at initialization. The required transition length depends on the model initialization, the amount of data per update, and the learning-rate settings. We therefore do not impose identical warmup step counts or proportions across different tasks.

Specifically, GPT-2 pretraining on WikiText-103 uses 500 warmup steps, corresponding to 5% of the full training budget. The updated Qwen experiments use 1% warmup in both the search and full-training stages, with 95 warmup steps in full training. In contrast, WikiText-2 fine-tuning starts from pretrained parameters and uses no warmup.

This rationale explains our use of different warmup configurations across tasks, but does not uniquely determine the specific numbers of steps or imply that these settings are optimal for every optimizer.

The independently scheduled search and full-training stages for Qwen, each with 1% warmup, are described in Appendix B.1. The task-specific warmup settings are summarized in Appendix B.3.

Reviewer_R4LTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes AIM, a framework based on ADMM-style variable splitting in which momentum plays the role of a multiplier. The authors then create a taxonomy of known optimizers organized by the choice of residual penalty and by how the parameter subproblem is approximated. Finally, they propose RADAR, which uses the preconditioner of RAD together with a gradient-difference momentum and a residual correction term, and test it on image classification, language modeling and reinforcement learning.

Critical strengths

1. A nice unifying perspective.

I am a fan of attempts to describe existing optimizers as instances of a single framework. When successful, such frameworks give a clearer theoretical understanding of why optimizers behave as they do, and they can point to new optimizers or to improvements of existing ones. Separating the update geometry, the momentum accumulation and the acceleration mechanism, as AIM tries to do, is a reasonable way to organize this, and Table 1 gives a clear overview of how known methods fit into it.

2. Clear and correct derivations.

The paper is easy to follow, and the derivations in Section 3 and the appendix are, as far as I checked, correct. The authors are also explicit about where the exact splitting identity differs from the explicit updates they use (Section 3.2 and Appendix A.2), which makes the scope of the framework easy to assess.

Critical weaknesses

1. Scope, positioning, and contribution of the AIM framework

The main claimed contribution of the paper is that AIM provides a unified framework for understanding modern optimizers, particularly by deriving the momentum iteration instead of assuming it. However, I have concerns about whether the proposed framework actually achieves this.

In Equations (9)–(10), recovering the EMA momentum recursion requires solving the θ\theta-subproblem exactly, with the gradient evaluated at the new iterate. The optimizers recovered in the paper do not follow this procedure and instead introduce the EMA momentum recursion separately. In particular, running AIM as described in Algorithm 1, including its multiplier update, does not appear to reproduce the correspondence reported in Table 1. with direct assignment (θk+1=yk+1\theta_{k+1}=y_{k+1}) the multiplier stays the same, so there is no momentum, and with the one-step refinement we get heavy-ball instead of Nesterov.

More generally, for the recovered optimizers, and especially RADAR, the main algorithmic components, including the momentum recursion and gradient-difference correction, are not derived directly from AIM. Instead, AIM appears to serve primarily as motivation for RADAR. The authors acknowledge that the explicit updates are not instances of Algorithm 1 (Section 3.2), yet still describe RADAR as "an AIM-derived extension of RAD" that "instantiates the two AIM design axes" (Sections 1 and 4). This would be a reasonable limitation if the framework provided something additional in return, such as a new design principle, an algorithmic component, a theoretical result, or an empirically justified modification supported by explicit ablations. However, I could not identify such a contribution. Once the momentum recursion is specified externally, it is unclear what the AIM viewpoint adds or how it explains why RADAR's additional components should improve optimization.

I consider this particularly important because the claimed unification is central to the paper's novelty. If the momentum recursion is treated as an external component, the proposed framework for quadratic penalties, appears to reduce largely to a proximal step under a linear model:

θk+1=arg⁡min⁡θ{⟨νmk+(1−ν)gk, θ−θk⟩+12η∥θ−θk∥Q2}\theta_{k+1} = \arg\min_{\theta} \left\lbrace \left\langle \nu m_k + (1-\nu) g_k,\ \theta - \theta_k \right\rangle + \frac{1}{2\eta} \left\Vert \theta - \theta_k \right\Vert_Q^2 \right\rbrace

This viewpoint is already well established in proximal and stochastic model-based optimization (Asi and Duchi, 2019; Davis and Drusvyatskiy, 2019), including extensions involving momentum (Deng and Gao, 2021; Kim et al., 2022; Ahn and Sra, 2022; Schaipp et al., 2024; Crawshaw et al., 2025). Moreover, deriving momentum recursions through the dual of a model-based or proximal formulations has been explored explicitly by Topollai and Choromanska (2026), while the related idea of aggregating historical information into a model has a long history in proximal bundle methods (Kiwiel, 1983). None of these connections are discussed in the paper, making it difficult to assess what AIM contributes beyond existing model-based interpretations.

Finally, the proposed unification does not appear to provide additional theoretical understanding. Theorem 2 concerns RADAR only and says nothing about the AIM framework itself. Given the relatively limited computational budget and small scale of the experiments, and especially since the paper is centered on AIM as a unifying framework, I would expect a stronger theoretical contribution. In particular, a comparison with established results in proximal and model-based optimization would help clarify whether AIM provides genuinely new insights or primarily offers an alternative interpretation of existing methods.

2. Limited algorithmic novelty of RADAR and insufficient experimental controls

My second major concern is that RADAR's algorithmic components are closely related to existing optimizers, and the experiments do not establish whether its gains come from a new mechanism or a different configuration of known updates.

First, the gradient-difference correction in Equation (22) has essentially the same form as MARS-approx (Yuan et al., 2025):

RADAR (Eq. 22):

mk+1=β1mk+(1−β1)gk+γ(gk−gk−1)m_{k+1}=\beta_1 m_k+(1-\beta_1)g_k+\gamma(g_k-g_{k-1})

MARS-approx:

mk+1=β1mk+(1−β1)gk+γMβ1(gk−gk−1)m_{k+1}=\beta_1 m_k+(1-\beta_1)g_k+\gamma_{\mathrm{M}}\beta_1(g_k-g_{k-1})

The two recursions are identical under the reparameterization γ=γMβ1\gamma=\gamma_{\mathrm{M}}\beta_1. Similarly, the residual correction in Equation (21) has the same form as the paper's own NAdam update in Equation (17), with η(1−β1)\eta(1-\beta_1) replaced by a free coefficient ℓ\ell.

More importantly, the combined RADAR update can be expressed as a standard mixture of momentum and the current gradient. For constant β1∈(0,1)\beta_1\in(0,1) and γ\gamma, with zero initialization and g−1=0g_{-1}=0, let mˉk+1=β1mˉk+(1−β1)gk\bar m_{k+1}=\beta_1\bar m_k+(1-\beta_1)g_k denote the standard EMA. Equation (22) then gives exactly

mk+1=(1−γβ1)mˉk+1+γβ1gkm_{k+1}=\left(1-\frac{\gamma}{\beta_1}\right)\bar m_{k+1}+\frac{\gamma}{\beta_1}g_k

Substituting this into Equation (21), and comparing with simplified AdEMAMix, which uses a single unnormalized momentum buffer, gives

RADAR:

θk+1=θk−ηkRk+1−1[νkmˉk+1+(1−νk)gk],νk=(1−ℓkηk)(1−γβ1)\theta_{k+1}=\theta_k-\eta_k R_{k+1}^{-1}\left[\nu_k\bar m_{k+1}+(1-\nu_k)g_k\right],\qquad \nu_k=\left(1-\frac{\ell_k}{\eta_k}\right)\left(1-\frac{\gamma}{\beta_1}\right)

Simplified AdEMAMix:

θk+1=θk−ηkSAmk+1SA+αgkv^k+1+ϵ,mk+1SA=β1mkSA+gk\theta_{k+1}=\theta_k-\eta_k^{\mathrm{SA}}\frac{m_{k+1}^{\mathrm{SA}}+\alpha g_k}{\sqrt{\hat v_{k+1}}+\epsilon},\qquad m_{k+1}^{\mathrm{SA}}=\beta_1 m_k^{\mathrm{SA}}+g_k

Thus, the two proposed corrections jointly determine the mixing coefficient νk\nu_k between the EMA and the current gradient. When νk=ν\nu_k=\nu is constant and nonzero, RADAR and simplified AdEMAMix produce the same update directions under

α=1−νν(1−β1)\alpha=\frac{1-\nu}{\nu(1-\beta_1)}

with an appropriate learning-rate rescaling, apart from differences in preconditioning, bias correction, and weight decay. This type of gradient-momentum mixing is well established, for example in Quasi-Hyperbolic Momentum and QHAdam (Ma and Yarats, 2019). NAdam is also a related special case with ν=β1\nu=\beta_1, while AdEMAMix and its simplified variant provide the most relevant comparisons (Pagliardini et al., 2025; Morwani et al., 2025).

I believe these connections should be discussed explicitly and shown in the experimental evaluation. The current experiments lack the ablations and controlled comparisons needed to establish the contribution of RADAR's individual components. In particular, there are no comparisons against QHAdam, simplified AdEMAMix, or AdEMAMix with a tuned mixing coefficient, and no ablations that independently remove the gradient-difference and residual corrections. Appendix B.5 varies ℓ\ell and γ\gamma only on GPT-2 fine-tuning and does not set either coefficient to zero, which would help isolate their respective contributions.

The comparison with NAdam, including the large reported gaps in pretraining, also does not isolate RADAR's choice of mixing, since the mixing coefficient and its schedule, preconditioning, bias correction, and potentially other implementation details differ between the methods.

Since mixing the current gradient with historical momentum is already a known algorithmic component, the contribution of RADAR appears to lie primarily in its particular choice (and possibly scheduling) of this mixture. This makes controlled comparisons against the existing optimizer family important. Under the current experimental setup, it is difficult to determine whether the reported improvements reflect a better update rule or simply a different parameterization and tuning of an established one.

References

  • Ahn, K., & Sra, S. (2022). Understanding Nesterov's acceleration via proximal point method. Symposium on Simplicity in Algorithms (SOSA), 117–130.
  • Asi, H., & Duchi, J. C. (2019). Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity. SIAM Journal on Optimization, 29(3), 2257–2290.
  • Crawshaw, M., Modi, C., Liu, M., & Gower, R. M. (2025). An exploration of non-Euclidean gradient descent: Muon and its many variants. arXiv:2510.09827.
  • Davis, D., & Drusvyatskiy, D. (2019). Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization, 29(1), 207–239.
  • Deng, Q., & Gao, W. (2021). Minibatch and momentum model-based methods for stochastic weakly convex optimization. Advances in Neural Information Processing Systems (NeurIPS), 34.
  • Kim, J. L., Toulis, P., & Kyrillidis, A. (2022). Convergence and stability of the stochastic proximal point algorithm with momentum. Learning for Dynamics and Control Conference (L4DC), PMLR 168, 1034–1047.
  • Kiwiel, K. C. (1983). An aggregate subgradient method for nonsmooth convex minimization. Mathematical Programming, 27, 320–341.
  • Ma, J., & Yarats, D. (2019). Quasi-hyperbolic momentum and Adam for deep learning. International Conference on Learning Representations (ICLR).
  • Morwani, D., Vyas, N., Zhang, H., & Kakade, S. (2025). Connections between schedule-free optimizers, AdEMAMix, and accelerated SGD variants. arXiv:2502.02431.
  • Pagliardini, M., Ablin, P., & Grangier, D. (2025). The AdEMAMix optimizer: Better, faster, older. International Conference on Learning Representations (ICLR).
  • Schaipp, F., Ohana, R., Eickenberg, M., Defazio, A., & Gower, R. M. (2024). MoMo: Momentum models for adaptive learning rates. International Conference on Machine Learning (ICML), PMLR 235.
  • Topollai, K., & Choromanska, A. (2026). Adaptive memory momentum via a model-based framework for deep learning optimization. International Conference on Artificial Intelligence and Statistics (AISTATS).
  • Yuan, H., Liu, Y., Wu, S., Zhou, X., & Gu, Q. (2025). MARS: Unleashing the power of variance reduction for training large models. International Conference on Machine Learning (ICML), PMLR 267.
Questions for authors
  1. What does AIM add, either as an explanation or as theory, compared to existing proximal-point, model-based and bundle formulations?
  2. Can the authors add ablations on the main tasks with ℓ=0\ell=0 and γ=0\gamma=0.
  3. In Table 7, RADAR has a lower total time than AdamW. Each RADAR step does the AdamW operations plus the gradient-difference and residual-correction terms, so I do not see where it would save computation. Are both optimizers implemented in the same way?
10 comments
AuthorsOpenReview ↗

W1. Correctness of deriving EMA momentum from exact subproblem solutions

Thank you for carefully pointing out this issue. We clarify the role of exact subproblem solutions by explicitly presenting the original framework and its equivalent EMA reformulation. Throughout this discussion, we assume that ff is continuously differentiable, ψ\psi is proper, closed, and convex, ρ1,k,ρ2,k>0\rho_{1,k},\rho_{2,k}>0, 0≤β<10\leq\beta<1, and the minimization subproblems admit solutions.

Original framework (A): subproblem minimization and residual-driven multiplier updates.

yk+1∈argmin⁡y(⟨mk,y−θk⟩+ρ1,k2ψ(y−θk)).y_{k+1}\in\underset{y}{\operatorname{argmin}}\left(\langle m_k,y-\theta_k\rangle+\frac{\rho_{1,k}}{2}\psi(y-\theta_k)\right).

θk+1∈argmin⁡θ(f(θ)+⟨mk,yk+1−θ⟩+ρ2,k2ψ(yk+1−θ)).\theta_{k+1}\in\underset{\theta}{\operatorname{argmin}}\left(f(\theta)+\langle m_k,y_{k+1}-\theta\rangle+\frac{\rho_{2,k}}{2}\psi(y_{k+1}-\theta)\right).

mk+1=mk+(1−β)ρ2,k2dk+1.m_{k+1}=m_k+(1-\beta)\frac{\rho_{2,k}}{2}d_{k+1}.

Here, dk+1∈∂ψ(yk+1−θk+1)d_{k+1}\in\partial\psi(y_{k+1}-\theta_{k+1}) is selected consistently with the first-order optimality condition of the parameter subproblem, denoted below as Eq. (1):

0=∇f(θk+1)−mk−ρ2,k2dk+1.0=\nabla f(\theta_{k+1})-m_k-\frac{\rho_{2,k}}{2}d_{k+1}.

When ψ\psi is differentiable, this subgradient is its gradient, and no additional selection is needed.

Using Eq. (1), the multiplier update becomes

mk+1=mk+(1−β)(∇f(θk+1)−mk)=βmk+(1−β)∇f(θk+1).m_{k+1}=m_k+(1-\beta)\bigl(\nabla f(\theta_{k+1})-m_k\bigr)=\beta m_k+(1-\beta)\nabla f(\theta_{k+1}).

This gives the following equivalent framework.

Reformulated framework (B): the same subproblem minimizations with an EMA multiplier update.

yk+1∈argmin⁡y(⟨mk,y−θk⟩+ρ1,k2ψ(y−θk)).y_{k+1}\in\underset{y}{\operatorname{argmin}}\left(\langle m_k,y-\theta_k\rangle+\frac{\rho_{1,k}}{2}\psi(y-\theta_k)\right).

θk+1∈argmin⁡θ(f(θ)+⟨mk,yk+1−θ⟩+ρ2,k2ψ(yk+1−θ)).\theta_{k+1}\in\underset{\theta}{\operatorname{argmin}}\left(f(\theta)+\langle m_k,y_{k+1}-\theta\rangle+\frac{\rho_{2,k}}{2}\psi(y_{k+1}-\theta)\right).

mk+1=βmk+(1−β)∇f(θk+1).m_{k+1}=\beta m_k+(1-\beta)\nabla f(\theta_{k+1}).

Under the stated assumptions, frameworks (A) and (B) are equivalent:

Framework (A)⟺Framework (B).\text{Framework (A)}\quad\Longleftrightarrow\quad\text{Framework (B)}.

The forward implication follows by substituting Eq. (1) into the residual-driven multiplier update. Conversely, framework (B) retains the same exact parameter-subproblem minimization, so its solution also satisfies Eq. (1) with a corresponding subgradient. Substituting this relation into the EMA update recovers the multiplier update in framework (A).

This equivalence does not require convexity of the parameter subproblem: both frameworks retain its minimization step, and the derivation uses only the necessary first-order optimality condition at a minimizer. We do not replace minimization by stationarity or assume that every stationary point is a minimizer.

We use framework (B) to recover existing optimizers within AIM by combining an explicit EMA recursion using the available gradient with different approximations of the parameter subproblem. This provides a unified interpretation of familiar momentum-based updates. More generally, either framework (A) or framework (B) can serve as a starting point for algorithm design.

Since solving the parameter subproblem exactly is typically computationally impractical in training, practical algorithms use approximate updates. Although frameworks (A) and (B) are equivalent under exact subproblem solutions, this equivalence need not persist after approximation. Starting from framework (A) retains the residual-driven multiplier update, whereas starting from framework (B) allows the EMA recursion to be maintained explicitly. These choices can produce different algorithms because an approximate subproblem solution need not preserve the exact residual–EMA identity.

This distinction suggests a further direction for improving existing optimizers: exploring explicit algorithms obtained from different approximations of the two equivalent implicit frameworks. Such improvements are a possibility to investigate, rather than a consequence of the implicit equivalence. Each resulting algorithm is analyzed through its own update equations.

The revised derivation appears in Section 3.1, with the equivalence between the two exact minimization frameworks established in Appendix A.2. Section 3.2 and Appendix A.3 explain how framework (B) recovers existing momentum-based updates and distinguish these explicit constructions from applying the residual-driven multiplier update after an approximate subproblem solve.

AuthorsOpenReview ↗

Connections to proximal and model-based optimization methods

Thank you for pointing out this connection. We agree that AIM should be discussed within the broader literature on proximal and model-based optimization. The aProx framework of Asi and Duchi (2019) and the model-based optimization framework for weakly convex problems developed by Davis and Drusvyatskiy (2019) both construct stochastic optimization methods using regularized local subproblems. Ahn and Sra (2022), Deng and Gao (2021), and Kim et al. (2022) further investigate connections between proximal updates and momentum. In addition, MoMo (Schaipp et al., 2024) and Adaptive Memory Momentum (Topollai and Choromanska, 2026) use local optimization models to derive adaptive learning rates and momentum coefficients, respectively. We therefore do not claim that AIM is the first to interpret momentum through optimization principles or proximal methods.

AIM shares the implicit-update perspective of proximal methods: the next parameter is defined through an optimization subproblem whose optimality condition involves the gradient at the unknown new iterate. Approximating this condition provides a route to practical explicit updates. Under a quadratic residual penalty, this connection also admits an exact proximal representation.

Specifically, for ψ(r)=∥r∥2\psi(r)=\lVert r\rVert^2, the exact auxiliary-variable update is yk+1=θk−ρ1,k−1mky_{k+1}=\theta_k-\rho_{1,k}^{-1}m_k. Completing the square in the parameter subproblem gives

θk+1∈prox⁡f/ρ2,k(θk−(ρ1,k−1−ρ2,k−1)mk),\theta_{k+1}\in\operatorname{prox}_{f/\rho_{2,k}}\left(\theta_k-(\rho_{1,k}^{-1}-\rho_{2,k}^{-1})m_k\right),

where

prox⁡τf(v):=argmin⁡θ(f(θ)+12τ∥θ−v∥2).\operatorname{prox}_{\tau f}(v):=\underset{\theta}{\operatorname{argmin}}\left(f(\theta)+\frac{1}{2\tau}\lVert\theta-v\rVert^2\right).

Thus, the exact parameter update is a proximal step centered at a point shifted by momentum. When ρ1,k=ρ2,k\rho_{1,k}=\rho_{2,k}, it reduces to the standard proximal point update. This representation makes the shared implicit-update structure concrete.

AIM's specific contribution lies in its residual-driven multiplier interpretation and the resulting connection between gradient memory and update geometry. When the parameter subproblem is solved exactly, its optimality condition gives

ρ2,k2dk+1=∇f(θk+1)−mk,dk+1∈∂ψ(yk+1−θk+1).\frac{\rho_{2,k}}{2}d_{k+1}=\nabla f(\theta_{k+1})-m_k,\qquad d_{k+1}\in\partial\psi(y_{k+1}-\theta_{k+1}).

Consequently, the relaxed multiplier correction

mk+1=mk+(1−β)ρ2,k2dk+1m_{k+1}=m_k+(1-\beta)\frac{\rho_{2,k}}{2}d_{k+1}

can be written as

mk+1=βmk+(1−β)∇f(θk+1).m_{k+1}=\beta m_k+(1-\beta)\nabla f(\theta_{k+1}).

This relation connects the splitting residual signal determined by the residual geometry to the discrepancy between the gradient and the multiplier, thereby explaining EMA accumulation through relaxed multiplier updates. Building on this connection, AIM organizes three interacting design choices within a single framework: the residual penalty, the multiplier or filtering rule, and the approximation of the objective-related parameter subproblem.

We have clarified these connections and distinctions in the revised manuscript. In particular, the implicit formulation serves as a starting point for constructing explicit algorithms; it does not require solving an exact proximal subproblem at every training step. Nor does this construction imply that the practical explicit algorithm is equivalent to an exact proximal method or automatically inherits its convergence or acceleration guarantees.

The literature discussion and the implicit-update connection are presented in Section 2.1 ("Related Work"). Section 3.2 explains how approximating the implicit parameter-subproblem optimality condition leads to explicit updates.

AuthorsOpenReview ↗

Q1. Core contributions of the AIM framework

The core contributions of AIM are twofold.

(1) Designing multiplier updates based on KKT stationarity to provide an optimization-based interpretation of momentum

The direction and scale of the multiplier correction are motivated by the parameter-subproblem optimality condition and its connection to multiplier updates in ADMM. The relaxation coefficient controls how rapidly the multiplier tracks the corresponding gradient. For the splitting constraint y=θy=\theta, the KKT stationarity condition of the Lagrangian with respect to θ\theta is

m=∇f(θ).m=\nabla f(\theta).

On the other hand, when AIM's parameter subproblem satisfies its first-order optimality condition, we have

∇f(θk+1)−mk−ρ2,k2dk+1=0,dk+1∈∂ψ(yk+1−θk+1).\nabla f(\theta_{k+1})-m_k-\frac{\rho_{2,k}}{2}d_{k+1}=0,\qquad d_{k+1}\in\partial\psi(y_{k+1}-\theta_{k+1}).

Thus, applying the full multiplier correction

mk+1=mk+ρ2,k2dk+1,m_{k+1}=m_k+\frac{\rho_{2,k}}{2}d_{k+1},

yields

mk+1=∇f(θk+1),m_{k+1}=\nabla f(\theta_{k+1}),

so that the updated multiplier and parameter satisfy the θ\theta-stationarity relation above. This enforces the parameter-stationarity relation only; it does not imply that primal feasibility or the full KKT conditions hold at each iteration. It provides an explicit optimality-based rationale for the direction and scale of the multiplier correction.

Introducing a relaxation coefficient 1−β1-\beta then gives

mk+1=mk+(1−β)ρ2,k2dk+1=βmk+(1−β)∇f(θk+1).m_{k+1}=m_k+(1-\beta)\frac{\rho_{2,k}}{2}d_{k+1}=\beta m_k+(1-\beta)\nabla f(\theta_{k+1}).

EMA momentum can therefore be interpreted as relaxed tracking of the multiplier value that satisfies the θ\theta-stationarity relation: it retains historical information while incorporating new gradient information through a residual-driven correction. AIM thus establishes a direct connection between KKT stationarity, multiplier correction, and momentum accumulation, providing a structured optimization-based interpretation of momentum updates.

(2) Distinguishing and connecting update geometry, gradient accumulation, and parameter correction to provide a unified perspective on optimizer design

AIM explicitly distinguishes three interacting design choices: the residual penalty determines the update geometry; the multiplier update determines how gradient information accumulates into momentum; and the approximation of the objective-related θ\theta-subproblem determines the parameter correction and its acceleration form.

This decomposition provides a route to constructing acceleration-oriented updates from the parameter subproblem: given the geometry and momentum mechanism, different low-cost approximations to the implicit parameter subproblem can yield different explicit correction forms. Optimizer design can therefore proceed not only through momentum estimation or gradient filtering, but also through the structure of the parameter-update subproblem, by studying how to correct the auxiliary update point produced by momentum.

These two perspectives are complementary. Under certain geometries and parameter conditions, a parameter correction can be reparameterized as a momentum or gradient mixture; this equivalence does not negate AIM's interpretive value. AIM's contribution is to reveal the common subproblem structure underlying these updates and to organize geometry selection, gradient memory, and parameter correction into an interconnected design space. Whether a specific algorithm admits a rigorous acceleration guarantee requires separate analysis of its actual update rule under the relevant assumptions.

The multiplier-update rationale is explained in Sections 2.2 and 3.1, with further derivations in Appendix A.1. The three interacting design choices are summarized in Section 3.4 and instantiated in RADAR in Section 4.1.

Reviewer_SJ4kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AIM, an ADMM-inspired framework that views momentum as a relaxed multiplier-like correction driven by a splitting residual. With exact subproblem solves, the multiplier update recovers an EMA of gradients. The practical optimizer variants instead maintain the EMA explicitly and approximate the parameter subproblem. Building on this framework, the authors propose RADAR, which combines RAD's adaptive geometry with gradient-difference momentum filtering and a residual correction. The paper provides a Lyapunov-based stationarity analysis and experiments on vision, language modeling, and reinforcement learning. At fixed batch size, the theoretical result gives a variance-controlled stationarity neighborhood rather than convergence to zero gradient.

Critical strengths

I find the splitting interpretation interesting. It gives a useful way to think about the tentative descent step and the correction applied afterward, rather than treating the optimizer as a collection of update equations. I also appreciate the clarification in Section 3.2 that the exact splitting identity and the practical EMA implementation are not the same algorithm.

The experiments cover a reasonably broad range of settings. I particularly appreciate the use of separate tuning and final-evaluation seeds in the supervised experiments, as this makes the reported gains easier to interpret. The discussion of a memory-efficient implementation and the runtime measurements is also useful, since an optimizer's practical cost matters alongside its final performance.

Critical weaknesses

I am not yet convinced that the two correction mechanisms provide independent algorithmic contributions. Taking the reparameterization in Eq. (73) as stated and combining it with Eq. (21), I obtain, for fixed coefficients,

θk+1=θk−ηRk+1−1[qmˉk+(1−q)gk],\theta_{k+1} = \theta_k - \eta R_{k+1}^{-1}\left[q\bar m_k + (1-q)g_k\right],

where mˉk\bar m_k is the ordinary gradient EMA and q=(1−ℓη)(β1−γ)q = \left(1-\frac{\ell}{\eta}\right)(\beta_1-\gamma). Under this representation, different choices of ℓ\ell and γ\gamma with the same qq would give the same parameter updates, provided the preconditioner and stored EMA are unchanged. This leaves me unsure what is gained by treating residual correction and gradient-difference filtering as separate mechanisms in RADAR. A comparison with directly tuning the resulting gradient–EMA mixture would help clarify the algorithmic contribution.

I found the Qwen2.5-0.5B results difficult to interpret. Adan and Lion have perplexities of roughly 133 and 332, while RADAR is around 30. These are large enough gaps that I would want to understand what went wrong with those baselines before drawing a broad conclusion. The learning rates are selected using only 0.1B tokens, compared with the final 1B-token budget. I appreciate that the tuning budget is shared across methods, but I am not sure that this short run selects competitive configurations for all of them.

Questions for authors
  1. Am I interpreting Eqs. (21) and (73) correctly in the reduction above? How is ℓ\ell handled when the learning rate decays, and is ℓ/η\ell/\eta kept constant? I would like to understand what distinguishes the implemented RADAR update from a directly parameterized gradient–EMA mixture with the same preconditioner.

  2. Could you explain the poor Adan and Lion results on Qwen2.5-0.5B? Were these runs unstable, or did they train stably but converge to much worse solutions? Have you checked whether selecting learning rates over a longer training horizon changes the comparison?

9 comments
AuthorsOpenReview ↗

W1 & Q1: Reparameterization, correction mechanisms, and the learning-rate schedule

Thank you for identifying this reparameterization. Your interpretation of Eqs. (21) and (73) in the original manuscript is correct: under matched initialization and fixed momentum coefficients, RADAR's gradient-difference filtering and residual correction can be combined into an equivalent gradient–EMA mixture.

Specifically, define the standard gradient EMA by

\qquad \bar{m}_0 = 0.$$ With $m_0=0$ and $g_{-1}=0$, the filtered momentum satisfies $$m_{k+1} = (\beta_1-\gamma)\bar{m}_k + (1-\beta_1+\gamma)g_k.$$ Here, $\bar{m}_k$ contains gradient information up to iteration $k-1$, before incorporating $g_k$. Substituting this identity into the parameter update gives, for $\eta_k>0$, $$\theta_{k+1} = \theta_k - \eta_k R_{k+1}^{-1} \left[q_k\bar{m}_k + (1-q_k)g_k\right],$$ where $$q_k = \left(1-\frac{\ell}{\eta_k}\right)(\beta_1-\gamma).$$ Thus, the filtering and correction coefficients jointly determine the effective mixture. These identities apply to the updates as written, without bias correction. Equivalence between complete implementations additionally requires matching the preconditioner, initialization, full coefficient schedule, and any bias-correction and weight-decay conventions. Under these matched conditions, choices of $\ell$ and $\gamma$ that produce the same $q_k$ sequence yield the same parameter updates. The reparameterization also allows RADAR to maintain a standard EMA state without explicitly storing the previous gradient $g_{k-1}$. We have expanded the appendix discussion to explain both its implementation benefit and its algorithmic implications. **What remains useful about the AIM decomposition?** AIM distinguishes three interacting design choices: the residual penalty specifies the update geometry, the multiplier or filtering rule determines gradient accumulation, and the approximation of the objective-related $\theta$-subproblem determines parameter correction. Within this design space, gradient-difference filtering is a specific choice intended to improve the responsiveness of momentum to gradient changes; it is not uniquely derived from the basic AIM subproblems. The two components therefore have distinct constructional roles: filtering modifies gradient accumulation, while residual correction adjusts the tentative parameter update using the gradient–momentum discrepancy. For RADAR, they can be combined because $u\mapsto R_{k+1}^{-1}u$ is linear for a given preconditioner. For nonlinear direction maps, separately mapping directions and mapping their mixture are generally different operations, so the same linear reduction does not automatically extend to nonlinear geometries. This motivates distinguishing the two roles within AIM, without claiming that they remain independent algorithmic degrees of freedom within RADAR.
AuthorsOpenReview ↗

How is handled as the learning rate decays?

In our implementation, ℓ\ell remains fixed while ηk\eta_k follows the learning-rate schedule. Consequently, ℓ/ηk\ell/\eta_k varies, inducing a time-dependent EMA coefficient qkq_k. For 0≤γ≤β10\leq\gamma\leq\beta_1 and 0≤ℓ/ηk≤10\leq\ell/\eta_k\leq1,

0≤qk≤β1−γ.0\leq q_k\leq\beta_1-\gamma.

Thus, γ\gamma determines the upper bound on the EMA coefficient, while ℓ/ηk\ell/\eta_k controls its reduction from that bound. With fixed ℓ>0\ell>0 and β1>γ\beta_1>\gamma, decreasing ηk\eta_k reduces qkq_k, shifting weight from the EMA toward the current gradient. This describes the relative mixture weights, rather than implying that the absolute current-gradient step increases.

When ηk=ℓ>0\eta_k=\ell>0, qk=0q_k=0 and the update uses only the current gradient. If 0<ηk<ℓ0<\eta_k<\ell and β1>γ\beta_1>\gamma, then qk<0q_k<0, so the representation becomes an affine rather than a convex combination. This can occur during either warmup or learning-rate decay.

The update can also be expressed without dividing by ηk\eta_k. Writing a=β1−γa=\beta_1-\gamma, we obtain

θk+1=θk−Rk+1−1[a(ηk−ℓ)mˉk+((1−a)ηk+aℓ)gk].\theta_{k+1}=\theta_k-R_{k+1}^{-1}\left[a(\eta_k-\ell)\bar{m}_k+\bigl((1-a)\eta_k+a\ell\bigr)g_k\right].

Accordingly, with fixed nonzero ℓ\ell, setting ηk\eta_k to zero does not necessarily make the full update vanish. We have added these derivations and clarified the roles of the filtering and correction coefficients, including their interaction with the learning-rate schedule, in Appendix A.5, alongside the additional baseline comparisons. The correction-ablation setup is presented separately in Section 5.3.

To assess the practical value of the resulting update rule, we compare RADAR with QHAdam, Simplified AdEMAMix (Sim-AdEMAMix), and AdEMAMix on GPT-2-small pretraining using FineWeb-Edu. All methods share the same data, training budget, and learning-rate schedule. We normalize the nominal update scale across parameterizations and apply a common decoupled weight-decay rule.

We use a two-stage hyperparameter search, with 0.2480.248B tokens per trial and a dedicated tuning seed of 42. First, we search the effective peak learning rate over 3×10−4,5×10−4,1.5×10−3\\{3\times10^{-4},5\times10^{-4},1.5\times10^{-3}\\}, fixing the mixing parameter to γ=0.1\gamma=0.1 for RADAR, ν1=0.9\nu_1=0.9 for QHAdam, q=0.9q=0.9 for Sim-AdEMAMix, and αfinal=5\alpha_{\mathrm{final}}=5 for AdEMAMix. Second, at each method's selected learning rate, we search its mixing parameter over 0.01,0.1,0.2\\{0.01,0.1,0.2\\}, 0.7,0.9,1.0\\{0.7,0.9,1.0\\}, 0.7,0.9,1.0\\{0.7,0.9,1.0\\}, and 1,5,10\\{1,5,10\\}, respectively. Reusing the initial mixing configuration gives five short trials per optimizer. Candidates are ranked by their mean validation loss at 90%, 95%, and 100% of the short training horizon.

The selected effective peak learning rate is 5×10−45\times10^{-4} for all four methods, with γ=0.01\gamma=0.01, ν1=1\nu_1=1, q=0.9q=0.9, and αfinal=1\alpha_{\mathrm{final}}=1, respectively. We then train each method from scratch for 2.482.48B tokens using seed 0. Both stages use 1% linear warmup followed by cosine decay, ending at zero in the short trials and at 1% of the peak learning rate in full training.

MetricRADARQHAdamSim-AdEMAMixAdEMAMix
Validation PPL ↓31.962232.087843.448632.2085

RADAR achieves the lowest validation perplexity in this comparison. These are single-seed results, so the small differences relative to QHAdam and AdEMAMix should not be interpreted as statistically established improvements. We also clarify that the selected QHAdam configuration, ν1=ν2=1\nu_1=\nu_2=1, reduces to the AdamW update.

The design roles are clarified in Section 3.4 ("A Unified Optimizer Design Space") and Section 4.1 ("Algorithm Design"). Appendix A.5 ("Reparameterization and Additional Comparative Experiments") provides the equivalent update, its dependence on the learning-rate schedule, and the additional momentum-mixing baseline comparisons.

AuthorsOpenReview ↗

W2 & Q2: Qwen2.5-0.5B baselines and the learning-rate search horizon

Thank you for raising this concern, particularly regarding the poor performance of Adan and Lion in the original 1B-token experiments. We did not observe training instability in these runs: both optimizers trained stably but achieved relatively poor final performance, possibly reflecting lower optimization efficiency in this experimental setting.

To examine whether the limited training and tuning budgets contributed to these results, we repeated the Qwen2.5-0.5B experiments with expanded budgets. Specifically, we increased the full training budget from 1B to 5B tokens and the budget for each learning-rate candidate from 0.1B to 0.5B tokens. Thus, the per-candidate search budget remained 10% of the full training budget, while its absolute horizon increased fivefold. These changes were applied uniformly to all methods.

We pretrained Qwen2.5-0.5B from scratch on FineWeb, comparing RADAR with RAD, Adam, AdamW, NAdam, Adan, Lion, and AdaBelief. All methods used the same data, sequence length, and global batch size. Both the search and full-training stages used 1% linear warmup followed by 99% cosine decay.

For each optimizer, we searched five peak learning rates obtained by multiplying its reference learning rate by {0.1,0.5,1,5,10}\{0.1,0.5,1,5,10\}. The reference learning rate was 3×10−43\times10^{-4} for RADAR, RAD, Adam, AdamW, NAdam, and AdaBelief; 7.5×10−47.5\times10^{-4} for Adan; and 3×10−53\times10^{-5} for Lion. Each candidate was evaluated after 0.5B tokens, and the configuration with the lowest final validation loss was selected. Other optimizer coefficients were held fixed during the search. Each selected configuration was then trained from scratch for 5B tokens, without resuming from its search checkpoint. Both stages used seed 42.

The final validation perplexities are reported below.

TaskMetricRADARRADAdamAdamWNAdamAdanAdaBeliefLion
FineWeb, Qwen2.5-0.5B (5B tokens)Valid PPL ↓21.698621.824922.311821.744727.541227.804621.926829.8632

Under the expanded protocol, Adan and Lion achieved substantially lower perplexities, considerably narrowing the unusually large gaps observed in the original experiments. Nevertheless, they remained less competitive than the other baselines after 5B tokens. These results suggest that the original training and tuning budgets constrained their performance, while the remaining gaps may reflect lower optimization efficiency under the tested configuration. Because both budgets were increased together, these experiments do not isolate their individual contributions.

We have replaced the original Qwen2.5-0.5B results in the revised manuscript with these updated results and reported the search ranges, selected learning rates, and training budgets. We have also added the training-loss curves, which show that RADAR achieved a lower final training loss than the compared baselines under the updated protocol. Since these additional experiments used a single seed, we interpret them as results under the tested configuration and do not claim statistically significant differences across seeds.

The updated Qwen results are reported in the main-results table and discussed in Section 5.2. Appendix B.1 specifies the expanded search budget, candidate ranges, selection criterion, and fresh full-training protocol. Appendix B.2 reports the selected learning rates and optimizer hyperparameters, while Appendix B.3 provides the training configuration and training-loss curves.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses to Reviewers

We thank all three reviewers for their constructive feedback, which has helped us substantially improve the quality of our manuscript. Below we summarize the main revisions to our framework, theory, and experiments.

Clarifying AIM’s constructive and interpretive contributions (SJ4k, R4LT, and fbLz).

We have clarified AIM’s central contribution as a constructive bridge between ADMM-inspired variable splitting and neural network optimizer design. Within a common optimization formulation, the residual penalty determines update geometry, the multiplier update determines gradient accumulation, and the parameter-subproblem solution strategy determines parameter correction. In particular, we strengthened the explanation of two constructive mechanisms:

  • Constructing momentum through residual-driven multiplier corrections. The parameter-subproblem optimality condition connects the splitting residual to the gradient–multiplier discrepancy. Relaxing the corresponding multiplier correction yields EMA momentum, while alternative relaxation and residual-processing rules provide a starting point for developing new momentum recursions.

  • Constructing parameter updates through subproblem solvers. Different approximate solutions of the parameter subproblem generate different explicit update structures, making the solver itself a component of optimizer design. Retaining the residual-driven multiplier update also allows these solver choices to shape momentum dynamics.

To further substantiate AIM’s constructive and interpretive contributions, we added concrete derivations and an empirical comparison. Under Euclidean and weighted quadratic geometries, we derived the SODA framework from AIM through parameter-subproblem linearization, two inner Halpern steps, and multiplier reparameterization. This establishes SODA with iterate-centered quadratic regularizers and corresponding coefficient schedules as a special case of AIM, while also providing a Halpern-type interpretation of its anchoring mechanism. Under nonlinear spectral geometry, AIM produces separately transformed momentum and residual directions, distinct from mixing them before a single transformation. A matched nanoGPT comparison with SODA(Muon) illustrates this construction, with the AIM-style correction achieving lower validation loss and perplexity.

Clarifying the EMA derivation (R4LT).

We clarified the equivalence between the residual-driven and EMA formulations under exact subproblem solutions, and how practical algorithms can be constructed from either formulation through approximate solvers.

Strengthening convergence analysis (fbLz).

We strengthened the convergence analysis with an auxiliary-sequence proof yielding an SGD-type stationarity rate of O(1/T+σ/BT)O(1/T+\sigma/\sqrt{BT}) for RADAR and an O(1/T)O(1/T) bound in the deterministic case. The proof addresses the dependence between the stochastic gradient and the adaptive preconditioner.

Expanding experimental evaluation (SJ4k, R4LT, and fbLz).

We added baseline comparisons on GPT-2-small, and longer training budgets, multi-seed comparisons, and explicit correction ablations on Qwen2.5-0.5B. These additions strengthen the empirical support for RADAR and clarify the contributions of its correction mechanisms. We also added vision evaluation metrics, training curves, and resource measurements, and clarified implementation details.

We thank SJ4k and R4LT for increasing their scores, and fbLz for their consistently positive assessment. We hope these revisions and results assist the reviewers and Area Chair in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.