- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Instead of studying the original problem , this paper to add a simple constraint This new formulation motivates the use of ADMM-type algorithms. The authors propose to study the augmented Lagrangian of the form where is closed and convex with . This leads to the ADMM-inspired Momentum (AIM) framework summarized in Algorithm 1.
Assuming that the -subproblem is solved exactly, the authors recover a standard Heavy-Ball-like momentum update for . Moreover, using a duality argument, the authors demonstrate that the mismatch between and is linked to the mismatch between and momentum , through the conjugate geometry induced by .
Later, the authors demonstrate that by choosing and the accuracy of solving the -subproblem recovers many popular algorithms like Muon, Adam, and NAdam. Based on this unified view of algorithm design, the authors propose the RADAR algorithm, which can be seen as some sort of modification of Adam and NAdam, inspired by the AIM framework.
The authors later test this algorithm in practice and demonstrate competitive performance across several tasks, including vision, language, and RL tasks. In the appendix, the authors also provide a convergence analysis under standard assumptions + bounded precondition for RADAR.
Critical strengths
-
The storyline is very clear and easy to follow. I enjoyed the idea of splitting variables and how it leads to the algorithm design. I also liked that the framework is quite general and covers many existing optimization methods.
-
The proposed algorithm is competitive in practice. It performs well across several tasks, demonstrating a universal behaviour.
-
The implementation looks correct to me. The authors also include most of the tuning details for reproducibility.
-
The authors also provide a convergence analysis with a novel Lyapunov function. It looks correct to me. Most steps are clearly explained. It establishes convergence of the form for the expected gradient norm.
Critical weaknesses
-
Though the AIM framework is clearly described, some parts of the algorithm design do not follow it. For example, I didn't find an explanation of momentum filtering. Why does RADAR use (22)? How does it appear in the algorithm? This step is unclear and looks like an unjustified heuristic. The authors did not explain why the preconditioner is chosen as in (19), but not a standard Adam-like one.
-
Some experiments look weird to me. In particular, I do not understand why some results are so noisy. For example, the performance of Lion and Adan is very inconsistent in the Qwen experiment. I also do not understand why, in the CIFAR-10 and CIFAR100 setting the authors report the test loss, not the test accuracy, as is typically done in the literature. I also find it weird that the loss dynamics in Figure 1 (left) have a weird drop around 7k steps. Finally, in most cases, the algorithms are statistically indistinguishable. These issues in the experimental comparison raise concerns about the validity of the reported performance metrics: most likely some algorithms are tuned well enough, while some training implementations went wrong (again, see Figure 1 left).
-
The authors should also mention that bias correction is used in the experiments as well, which is not covered in the analysis.
-
The authors do not compare against the SODA framework by [Pethick et al., 2026] that also studies a similar approach to viewing different algorithms from a special choice of regularization. The paper should provide a detailed comparison against that work.
Questions for authors
-
Could you explain the choice of warm-up steps? Why is it so different in Table 6?
-
Does the convergence analysis cover the experimental setting (i.e., the choice of hyperparameters, etc)?
-
It would be good to see also tuning results, to see that the chosen hyperparameters do not lie on the boundary, as well as why some algorithms get so noisy.
-
The gradient norm convergence of SGD has the form . By taking to minimize the bound we arrive to . In this paper, the convergence is , which is basically using a constant in the very first bound. Can we get the second bound, which is an optimal convergence method for SGD?
18 comments
W2. Reliability of experimental results, evaluation metrics, and training curves
Thank you for raising these issues. We have reexamined the hyperparameter search and experimental settings and provide the following clarifications.
First, the poor performance of Adan and Lion in the original Qwen experiments may partly reflect the limited 1B-token training budget. To investigate this possibility, we increased the full training budget from 1B to 5B tokens and the learning-rate search budget from 0.1B to 0.5B tokens per candidate for all methods. The final validation perplexities on Qwen2.5-0.5B are reported below.
| Task | Metric | RADAR | RAD | Adam | AdamW | NAdam | Adan | AdaBelief | Lion |
|---|---|---|---|---|---|---|---|---|---|
| FineWeb, Qwen2.5-0.5B | Valid PPL ↓ | 21.6986 | 21.8249 | 22.3118 | 21.7447 | 27.5412 | 27.8046 | 21.9268 | 29.8632 |
Under the expanded protocol, Adan and Lion improve substantially, narrowing the gaps observed in the original experiments, although they remain less competitive than the other baselines. These observations suggest that the limited original budget may have contributed to their poor performance.
Second, regarding the evaluation metrics for CIFAR-10 and CIFAR-100, we agree with the reviewer's suggestion. Cross-entropy loss reflects how well the model fits predictive probabilities and captures changes during optimization, whereas classification accuracy directly measures the proportion of correct predictions and is an important and widely used metric for classification tasks. Reporting test loss alone is insufficient for a comprehensive evaluation. We have therefore also reported test accuracy in the revised manuscript to provide a more complete performance comparison.
| Task | Metric | RADAR | RAD | Adam | AdamW | NAdam | Adan | AdaBelief | Lion |
|---|---|---|---|---|---|---|---|---|---|
| CIFAR-10, ViT (6.3M) | Test Accuracy (%) ↑ | 87.2600 ± 0.3740 | 86.4760 ± 0.0559 | 86.5340 ± 0.3822 | 86.2880 ± 0.2163 | 86.5300 ± 0.3663 | 87.6400 ± 0.2660 | 86.3260 ± 0.3779 | 87.5040 ± 0.3112 |
| CIFAR-100, ResNet-50 | Test Accuracy (%) ↑ | 76.5320 ± 0.3195 | 75.0440 ± 0.3057 | 76.3200 ± 0.1608 | 75.0920 ± 0.6646 | 76.2840 ± 0.3816 | 76.4680 ± 0.2667 | 74.8360 ± 0.3956 | 74.5480 ± 0.3774 |
Results are reported as mean ± sample standard deviation over five random seeds (0–4). Higher accuracy is better. The best mean accuracy in each row is highlighted in bold.
Third, the sharp loss drop at approximately 7,000 steps in the GPT-2 training curves coincides with an epoch transition. This experiment trains on WikiText-103 for approximately three epochs, and the full training trajectories shown in the inset plots of Figure 1 in the original manuscript exhibit a similar drop around 3,500–4,000 steps, corresponding to the preceding epoch transition. Similar epoch-boundary drops in training loss have been documented and studied as the “epochal sawtooth phenomenon” (Liu and Ma, 2025). These observations suggest that the drops are associated with epoch transitions rather than indicating sudden acceleration unique to our optimizer. We have clarified this phenomenon and provided the training-loss curves in Appendix B.3 ("Language Modeling").
Finally, we appreciate the reviewer's reminder about the statistical reliability of differences between optimizers. The original experiments include results from multiple random seeds. In the revised manuscript, we have reported their means, standard deviations, and numbers of seeds more clearly, and distinguished them from the new 5B-token experiments, which were conducted using a single seed. We also acknowledge that, on some tasks, the performance differences are small relative to the variation across seeds. These results do not support a claim that RADAR has a statistically significant advantage on every task. We have therefore adjusted our wording to conclude that RADAR is competitive on the tested tasks and achieves better average performance in some experiments, without interpreting every numerical improvement as statistically significant.
Section 5.2 reports the updated Qwen perplexities and vision test accuracies. Seed settings are summarized at the beginning of Appendix B; Appendix B.1 describes the search protocol, Appendix B.2 lists the selected hyperparameters, and Appendix B.3 provides the language-modeling configurations and training-loss curves.
W3. Clarification of bias correction & W4. Comparison with the SODA framework & Q1. Warmup settings across tasks
W3. Clarification of bias correction
Thank you for pointing out this omission. Our experimental implementation uses bias correction, which was not clearly stated in the original manuscript. Section 4.1 explicitly notes that the algorithm presentation uses the core recursion without bias correction. The scope statement in Appendix A.6 also clarifies that the convergence analysis concerns this uncorrected recursion. Appendix B.2 states that bias correction is applied in the experimental implementation.
W4. Comparison with the SODA framework
Thank you for bringing SODA, an excellent and closely related work, to our attention. We recognize the important connections between SODA and AIM in using dual structures to provide a unified understanding of optimizers, and have added a citation and discussion in the revised manuscript.
Starting from optimistic dual averaging, SODA provides a unified treatment of gradient averaging and optimism, update geometries induced by regularization functions, and parameter averaging and extrapolation. AIM shares the idea of using convex conjugacy to connect gradient information with update geometry. Specifically, letting , AIM's -subproblem can be expressed as
which implies
This expression directly demonstrates the connection between the two frameworks in their construction of geometry.
AIM's specific contribution is to explain the origins of momentum accumulation and parameter correction through variable splitting and the associated augmented Lagrangian structure. In particular, momentum admits a Lagrange-multiplier interpretation, with its update driven by the primal-variable residual. Combining this update with the first-order optimality condition of the exact -subproblem yields a gradient EMA representation. At the same time, this subproblem connects the gradient–momentum discrepancy to the primal-variable residual, providing a basis for constructing residual corrections in parameter updates through subproblem approximations.
The two frameworks also overlap at the level of concrete algorithms. For example, under matched initialization and fixed coefficients, RADAR's gradient-difference momentum and residual correction can be rewritten as a mixture of the current gradient and a standard EMA, directly connecting it to optimistic gradient mixing in SODA. We therefore position AIM as an interpretive and design framework complementary to SODA: its distinctive feature is to connect momentum, update geometry, and parameter correction to a single split optimization problem through residual-driven multiplier updates and approximations of the objective-related subproblem.
The comparison with SODA is included in Section 2.1, under "Dual averaging and unified optimizer frameworks." Section 3.1 explains AIM's residual-driven multiplier interpretation, while Appendix A.5 discusses the gradient–EMA reparameterization of RADAR.
Q1. Warmup settings across tasks
Thank you for your question. Our general rationale for warmup is to provide a transition period for the model parameters and optimizer states to adapt during early training. Gradually increasing the learning rate avoids immediately applying the full update magnitude at initialization. The required transition length depends on the model initialization, the amount of data per update, and the learning-rate settings. We therefore do not impose identical warmup step counts or proportions across different tasks.
Specifically, GPT-2 pretraining on WikiText-103 uses 500 warmup steps, corresponding to 5% of the full training budget. The updated Qwen experiments use 1% warmup in both the search and full-training stages, with 95 warmup steps in full training. In contrast, WikiText-2 fine-tuning starts from pretrained parameters and uses no warmup.
This rationale explains our use of different warmup configurations across tasks, but does not uniquely determine the specific numbers of steps or imply that these settings are optimal for every optimizer.
The independently scheduled search and full-training stages for Qwen, each with 1% warmup, are described in Appendix B.1. The task-specific warmup settings are summarized in Appendix B.3.
W1. Rationale for gradient-difference filtering and preconditioning geometry
Thank you for pointing out where the relationship between the framework and the concrete algorithm design needs further explanation. AIM distinguishes three interconnected mechanisms: the residual penalty determines the update geometry; the multiplier update or filtering rule determines how gradient information is accumulated; and the approximation of the objective-related parameter subproblem determines the form of the parameter correction. RADAR's relativistic adaptive geometry, gradient-difference momentum filtering, and decoupled residual correction correspond to these three design choices, respectively.
We agree that the gradient-difference filter in Eq. (22) of the original manuscript is not uniquely derived from the basic AIM framework. Rather, it is a specific design choice within the momentum-filtering mechanism identified by the framework. Its motivation is to use changes between consecutive gradients to correct the lag of a standard EMA in tracking changing gradients. In the revised manuscript, we have clearly distinguished relationships derived directly from the framework from the additional design choices made in RADAR, so that readers do not mistakenly infer that all components are uniquely determined by the same set of subproblems.
Regarding the choice of preconditioner, we adopt RAD's relativistic adaptive geometry because it has an explicit dynamical-systems rationale. RAD constructs its updates through conformal symplectic discretization, combining a relativistic speed-limiting mechanism with preservation of the dynamical system's structure, which motivates its use for improving long-term training stability. This also motivates our introduction of gradient-difference filtering and decoupled residual correction on top of this geometry: we seek to combine its potential stability benefits with improved gradient tracking and parameter updates. AIM itself is not restricted to this geometry; standard Adam-type adaptive geometry can also be incorporated into the framework. We have added this rationale to the revised manuscript and clarified which geometry RADAR inherits and which correction mechanisms it introduces.
Section 4.1 distinguishes the geometry, filtering, and residual-correction choices used in RADAR. Their roles within AIM are discussed in Section 3.4, and the scope of their algebraic equivalence is explained in Appendix A.5.