- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The work argues that Adam's momentum hyper params beta_1 and beta_2 along with weight decay (lambda) should not be kept constant throughout training. The paper proposes that the momentum hyper params follow a logarithmic scaling, so the optimizers "memory" grows with the time horizon. With this, they propose an optimizer ADANA. On an Enoki ladder, the authors show that ADANA reaches up to 1.4x compute multiplier compared to AdamW with constant weight decay.
Critical strengths
- Extensive ablations! I appreciate the authors performing extensive ablations and a plethora of avenues. They explore damping, a(t) shape, weight decay form, k sensitivity, larger batches.
- Solid experiments at scale. The authors perform experiments at chinchilla token counts and large enough model sizes. They also redo experiments on the Qwen3 model which improves on their argument.
Critical weaknesses
- Batch size scale: at larger, more realistic batch sizes, ADANA no longer maintains its large gains. Batch size 256 is more representative of what is used in practice and on here, Fig. 24 vs Fig. 1, the gains do not seem to match. Moreover, the batch size used for Muon is too low, and that skews the baseline. Muon has more computational efficiency at larger batch sizes (1.4x at 256 vs 1.1x at 32; from the paper's own findings).
- The 40% compute savings is misleading! A 1.4x multiplier means ADANA needs about 1/1.4 = 71% of AdamW's compute. And this would mean a saving of 29%. In other words, to reach a certain loss value, if AdamW uses 100 compute units, ADANA would use at least 71. I ask the authors revise this language.
Questions for authors
- Please provide validation losses as a table. I, and most curious readers, would like to take a look at this. It would be an easier way of learning the gains.
- Two cents, but Figure 2 is a lot easier to read and understand compared to Figure 1. Perhaps switch them?
- "Constant beta_1 may not add any benefit over SGD" is a pretty tall claim which I do not think to be true. Can we test this empirically here? I urge the authors to rephrase the wording else.
We thank the reviewer for their detailed review and positive evaluation of our work. We are glad they appreciate our "extensive ablations" across damping, schedule shape, weight-decay form, κ sensitivity and larger batches, and find our experiments at scale "solid", with the Qwen3 results further supporting our argument. We address the points raised below.
Weaknesses
We agree with the reviewer that Muon's compute efficiency improves at larger batch sizes, while ADANA's gains decrease, and we made this explicit in our batch-size ablation. We note, however, that at fixed batch size, ADANA's efficiency still improves with scale, eventually outperforming Muon at around 2×102 PFH (Figure 5c).
Our terminology is defined in footnote 2 on page 1 and is consistent with prior work [1]. As suggested by the reviewer, we will add a remark clarifying this terminology and stating explicitly that the savings correspond to up to 29% of AdamW's compute.
[1] Qiu, Shikai, et al. "Hyperparameter transfer enables consistent gains of matrix-preconditioned optimizers across scales." Advances in Neural Information Processing Systems 38 (2026): 130867–130911.
Questions
We thank the reviewer for this suggestion. We provide the final validation losses below and will add this table to the appendix.
Final validation loss vs. number of heads H (lower is better; – = not run)
We thank the reviewer for this suggestion and will consider switching them in the revised manuscript. Our motivation for placing Figure 1 first is that it summarizes the compute multipliers of all algorithms relative to the AdamW baseline.
The sentence at line 207 reads: "constant β1 may not improve scaling behavior beyond stochastic gradient descent (SGD)". The claim concerns scaling behavior specifically, not performance in general. As explained at lines 216–217, this was proved in [2], where the authors show that constant-β1 momentum has the same scaling-law exponents as SGD, whereas logarithmic-time schedules with proper damping achieve better exponents. We agree the wording can be made more precise and will rephrase it to make clear that this refers to scaling exponents in the setting studied in [2].
[2] Ferbach, Damien, et al. "Dimension-adapted momentum outscales SGD." Advances in Neural Information Processing Systems 38 (2026) (Spotlight): 112780–112977.