acceptodds
Under review as a conference paper at ICLR 2027

Logarithmic-time Schedules for Scaling Language Models with Momentum

Abstract

In practice, the hyperparameters and weight-decay in AdamW are typically kept at fixed values. Is there any reason to do otherwise? We show that for large-scale language model training, the answer is yes: by exploiting the power-law structure of language data, one can design time-varying schedules for that deliver substantial performance gains. We study logarithmic-time scheduling, in which the optimizer’s gradient memory horizon grows with training time. Although naïve variants of this are unstable, we show that suitable damping mechanisms restore stability while preserving the benefits of longer memory. Based on this, we present ADana, an AdamW-like optimizer that couples log-time schedules with explicit damping to balance stability and performance. We empirically evaluate ADana across transformer scalings (45M to 4.4B parameters), comparing against AdamW, Muon, and Ademamix. When properly tuned, ADana achieves up to 40% compute efficiency relative to a tuned AdamW, with gains that persist—and even improve—as model scale increases. We further show that similar benefits arise when applying logarithmic-time scheduling to Ademamix, and that logarithmic-time weight-decay alone can yield significant improvements. Finally, we present variants of ADana that mitigate potential failure modes and improve robustness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
momentum, optimization, scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bCJwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work argues that Adam's momentum hyper params beta_1 and beta_2 along with weight decay (lambda) should not be kept constant throughout training. The paper proposes that the momentum hyper params follow a logarithmic scaling, so the optimizers "memory" grows with the time horizon. With this, they propose an optimizer ADANA. On an Enoki ladder, the authors show that ADANA reaches up to 1.4x compute multiplier compared to AdamW with constant weight decay.

Critical strengths
  1. Extensive ablations! I appreciate the authors performing extensive ablations and a plethora of avenues. They explore damping, a(t) shape, weight decay form, k sensitivity, larger batches.
  2. Solid experiments at scale. The authors perform experiments at chinchilla token counts and large enough model sizes. They also redo experiments on the Qwen3 model which improves on their argument.
Critical weaknesses
  1. Batch size scale: at larger, more realistic batch sizes, ADANA no longer maintains its large gains. Batch size 256 is more representative of what is used in practice and on here, Fig. 24 vs Fig. 1, the gains do not seem to match. Moreover, the batch size used for Muon is too low, and that skews the baseline. Muon has more computational efficiency at larger batch sizes (1.4x at 256 vs 1.1x at 32; from the paper's own findings).
  2. The 40% compute savings is misleading! A 1.4x multiplier means ADANA needs about 1/1.4 = 71% of AdamW's compute. And this would mean a saving of 29%. In other words, to reach a certain loss value, if AdamW uses 100 compute units, ADANA would use at least 71. I ask the authors revise this language.
Questions for authors
  1. Please provide validation losses as a table. I, and most curious readers, would like to take a look at this. It would be an easier way of learning the gains.
  2. Two cents, but Figure 2 is a lot easier to read and understand compared to Figure 1. Perhaps switch them?
  3. "Constant beta_1 may not add any benefit over SGD" is a pretty tall claim which I do not think to be true. Can we test this empirically here? I urge the authors to rephrase the wording else.
1 comment

We thank the reviewer for their detailed review and positive evaluation of our work. We are glad they appreciate our "extensive ablations" across damping, schedule shape, weight-decay form, κ\kappa sensitivity and larger batches, and find our experiments at scale "solid", with the Qwen3 results further supporting our argument. We address the points raised below.

Weaknesses

Batch size scale: at larger, more realistic batch sizes, ADANA no longer maintains its large gains. Batch size 256 is more representative of what is used in practice and on here, Fig. 24 vs Fig. 1, the gains do not seem to match. Moreover, the batch size used for Muon is too low, and that skews the baseline. Muon has more computational efficiency at larger batch sizes (1.4x at 256 vs 1.1x at 32; from the paper's own findings).

We agree with the reviewer that Muon's compute efficiency improves at larger batch sizes, while ADANA's gains decrease, and we made this explicit in our batch-size ablation. We note, however, that at fixed batch size, ADANA's efficiency still improves with scale, eventually outperforming Muon at around 2×1022 \times 10^2 PFH (Figure 5c).

The 40% compute savings is misleading! A 1.4x multiplier means ADANA needs about 1/1.4 = 71% of AdamW's compute. And this would mean a saving of 29%. In other words, to reach a certain loss value, if AdamW uses 100 compute units, ADANA would use at least 71. I ask the authors revise this language.

Our terminology is defined in footnote 2 on page 1 and is consistent with prior work [1]. As suggested by the reviewer, we will add a remark clarifying this terminology and stating explicitly that the savings correspond to up to 29% of AdamW's compute.

[1] Qiu, Shikai, et al. "Hyperparameter transfer enables consistent gains of matrix-preconditioned optimizers across scales." Advances in Neural Information Processing Systems 38 (2026): 130867–130911.

Questions

Please provide validation losses as a table. I, and most curious readers, would like to take a look at this. It would be an easier way of learning the gains.

We thank the reviewer for this suggestion. We provide the final validation losses below and will add this table to the appendix.

Final validation loss vs. number of heads H (lower is better; – = not run)

OptimizerH=6H=8H=10H=12H=14H=16H=18H=20H=22H=24H=26H=28H=30H=32H=34H=36H=39H=40H=41H=43H=44H=48
ADANA (κ=0.85\kappa=0.85)3.82073.57723.42263.27583.17093.06162.98312.90022.84382.7761–2.67822.64082.6017–2.53622.4940–2.4701–2.42832.3857
AdEMAMix (κ=0.85\kappa=0.85)3.81453.57623.42913.27383.17283.05922.98552.90432.84732.7820–2.68702.65042.60992.58132.54502.5051–2.4764–2.43742.3940
D-Muon3.82793.58733.43523.28683.18433.07683.00452.92602.86192.79372.75152.69792.66172.62732.59462.56132.5231–2.4945–2.4518–
AdamW3.85723.61683.46883.31543.21403.10093.02332.94142.88362.81482.77092.71642.67892.63492.60632.56892.52652.51002.49962.47302.45922.4143

Two cents, but Figure 2 is a lot easier to read and understand compared to Figure 1. Perhaps switch them?

We thank the reviewer for this suggestion and will consider switching them in the revised manuscript. Our motivation for placing Figure 1 first is that it summarizes the compute multipliers of all algorithms relative to the AdamW baseline.

"Constant beta_1 may not add any benefit over SGD" is a pretty tall claim which I do not think to be true. Can we test this empirically here? I urge the authors to rephrase the wording else.

The sentence at line 207 reads: "constant β1\beta_1 may not improve scaling behavior beyond stochastic gradient descent (SGD)". The claim concerns scaling behavior specifically, not performance in general. As explained at lines 216–217, this was proved in [2], where the authors show that constant-β1\beta_1 momentum has the same scaling-law exponents as SGD, whereas logarithmic-time schedules with proper damping achieve better exponents. We agree the wording can be made more precise and will rephrase it to make clear that this refers to scaling exponents in the setting studied in [2].

[2] Ferbach, Damien, et al. "Dimension-adapted momentum outscales SGD." Advances in Neural Information Processing Systems 38 (2026) (Spotlight): 112780–112977.

Reviewer_W7U8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ADANA, an AdamW-like optimizer that increases its gradient memory during training. It combines logarithmic-time schedules for the first and second moments with a damped Nesterov-style update and decreasing weight decay. Prior theory on power-law regression motivates these choices. The paper also proposes variants that use clipping and an estimate of gradient sparsity to improve stability.

The evaluation compares ADANA with AdamW, Muon, and AdEMAMix on FineWeb across model sizes from 45M to 4.4B parameters. Additional experiments test a modified Qwen3 architecture, larger batches, synthetic problems, and image classification. The authors report lower validation loss and compute multipliers of up to approximately 1.4 relative to AdamW. Experiment ablations examine the contribution of each component and show that decreasing weight decay can also improve existing optimizers.

Critical strengths

ADANA’s improvement over the tested AdamW baseline persists across a broad range of model sizes. Results on a modified Qwen3 architecture and with larger batches provide further support. At 2.47B parameters, the Qwen3 model reaches validation loss 2.496 with ADANA, compared with 2.536 with AdamW (Table 18). This consistency supports acceptance because it suggests that the benefit extends beyond the small models used for tuning.

The ablations help explain which changes contribute to the improvement. They test the current-gradient term, momentum damping, second-moment scheduling, and weight decay separately (Figures 3, 6, 7, and 10). The results show that longer memory needs suitable damping and update structure. Decreasing weight decay also improves AdamW and AdEMAMix, so part of the benefit is available without adopting the full optimizer. These findings make the paper useful for the design of other training methods.

Critical weaknesses

The comparison with AdEMAMix leaves uncertainty about the claimed advantage. Although learning rates are tuned separately, the main comparison uses AdEMAMix schedules chosen to mirror ADANA. Appendix J.5 provides joint hyperparameter sweeps, but only at 70M and 186M parameters. These checks do not establish that the selected schedules remain competitive at larger scales. The experiments also use one shared initialization and data-order seed; the confidence analysis covers learning-rate fits rather than variation across training runs. Since the main contribution is improved training efficiency, uncertainty about the baseline and the size of the gain materially limits the case for acceptance.

Questions for authors

How were the AdEMAMix schedules used in the main comparison selected? The main text says they were chosen to mirror ADANA, while Appendix J.5 reports joint sweeps at two smaller scales. What evidence supports transferring these settings to larger models, and does ADANA retain its advantage under other competitive AdEMAMix settings?

2 comments

We thank the reviewer for their review and valuable feedback. We are glad they find that ADANA's improvement over AdamW "persists across a broad range of model sizes", that our ablations "help explain which changes contribute to the improvement", and that these findings "make the paper useful for the design of other training methods". We address the points raised below.

Comparison with AdEMAMix

We first note that the functional form of AdEMAMix is more general than that of ADANA, so there are, by definition, AdEMAMix schedules that perform at least as well as ADANA. The key contribution of this paper is not to claim that ADANA beats every possible AdEMAMix configuration. Rather, it is to show that log-time schedules combined with power-damped Nesterov momentum work well in a Nesterov/AdamW combination (i.e. ADANA). We devoted Section J to explain precisely the correspondance and the motivation of our comparison: "In this section, we analyze the connection between ADANA Algorithm 1 and ADEMAMIX Algorithm 6. We show the existence of a scaling rule for ADEMAMIX hyperparameters β1\beta_1 and α\alpha such that ADEMAMIX bears strong connections with ADANA" (lines 3504-3506).

Regarding how the AdEMAMix schedules were selected: we tuned the AdEMAMix hyperparameters at 70M and 186M parameters (Appendix J.5, Figure 34), and these sweeps showed that our prescribed ADANA schedules were optimal at these scales. At larger scales, scaling rules for these hyperparameters must be fixed in advance, as tuning them directly is computationally infeasible. We chose rules that mirror ADANA for three reasons: they were optimal at small scale, they provided consistent compute gains over the AdamW baseline, and they make the connection between the two algorithms explicit.

More broadly, we expect ADANA-inspired rules for AdEMAMix (essentially, replacing the single current gradient in ADANA with a short-EMA Adam term) to improve AdEMAMix's performance to at least match ADANA, and possibly beat it. See in particular our ablation in Figure 10 where we compare both variants. We view this as a natural direction for future work that our findings enable.

Single seed

Regarding the use of a single initialization and data-order seed, running scaling ladders over multiple seeds is not feasible under our compute budget (22 H100-years were used for this project, see Table 23). For this reason, we note that using a single seed per configuration is very standard practice in the scaling literature (eg [1], [2], [3]). We note, however, that all optimizers share the same seed, so comparisons at each scale are paired. In addition, the consistency of the gains across the full scaling ladder, the modified Qwen3 architecture and the larger-batch experiments provides evidence that the improvement is not an artifact of a particular run. We report all learning-rate sweeps and extrapolations, along with a detailed description of our baselining procedure, in Appendix F.

We are happy to answer any further questions the reviewer may have.

[1]Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).

[2]Wen, Kaiyue, et al. "Fantastic pretraining optimizers and where to find them." International Conference on Learning Representations. Vol. 2026. 2026.

[3]Semenov, Andrei, Matteo Pagliardini, and Martin Jaggi. "Benchmarking optimizers for large language model pretraining." arXiv preprint arXiv:2509.01440 (2025).

My main concerns are resolved, and thus I raised my rating from 2 to 3.

Reviewer_NqZbOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ADANA, an AdamW-like optimizer combining logarithmic-time first- and second-moment schedules, damped Nesterov-style updates, and decaying independent weight decay. It studies how these schedules transfer across a transformer scaling ladder on FineWeb, with additional architecture, batch-size, and training-horizon experiments. Ablations examine the individual components, while sparse-gradient analysis motivates more robust DANA variants. The main empirical finding is improved validation loss and compute multipliers relative to the evaluated baselines, with gains that persist over the tested scales.

Critical strengths
  • The scaling ladder, small-scale LR searches, component ablations, and additional architecture/batch experiments make the scheduling results informative beyond a single model configuration. Comparisons against AdamW with decaying weight decay help separate the momentum contribution from the WD contribution.

  • The schedules require little structural change to AdamW, and the paper investigates both their benefits and failure modes. The damping and sparsity variants provide useful guidance on when long-memory updates may require additional safeguards.

Critical weaknesses
  • Its optimizer-specific parameters are mapped from ADANA rather than swept (p. 3, line 143). This is useful as a controlled schedule comparison, but does not establish superiority over a well-tuned AdEMAMix baseline. A modest independent search at representative scales, or narrower comparative claims, would strengthen the result.

  • A 1.4× multiplier corresponds to about 28.6% less compute relative to AdamW, rather than 40% savings. Moreover, the single-seed setup and endpoint extrapolation leave uncertainty beyond the LR-fit bootstrap in F.3.6. These issues affect the magnitude and reliability of the central practical claim, although they do not negate the observed loss improvements.

Questions for authors
  • Does ADANA retain its advantage when AdEMAMix's long-memory, mixing, and warmup parameters are tuned independently with a comparable budget?

  • Could you distinguish the 40% increase in compute efficiency from the approximately 28.6% reduction relative to baseline compute, and use consistent terminology?

  • Which compute multipliers require extrapolation beyond measured baseline losses? How sensitive are they to interpolation/fitting choices and training-seed variability, beyond the LR-fit uncertainty already reported?

  • Are the claims conditional on the common N = 20D training rule, or has the optimal model/token allocation been checked separately for the optimizers?

  • Stability scope: L.1/P.3 take p → 0 and a supremum over ε. What conclusions hold for fixed positive ε with the actual learning-rate and weight-decay schedules? Please distinguish worst-case non-uniformity from divergence in a fixed training run.

  • Practical overhead: What are the measured step-time and peak-memory costs of ADANA and DANA-Star-MK4 relative to AdamW? This would clarify how the FLOP-based gains translate into practical savings.

3 comments

We thank the reviewer for their positive evaluation of our empirical findings and ablations. We are also glad they find that our schedules "require little structural change to AdamW" and that our damping and sparsity variants "provide useful guidance on when long-memory updates may require additional safeguards". We address the points raised below.

Weaknesses

Its optimizer-specific parameters are mapped from ADANA rather than swept (p. 3, line 143). This is useful as a controlled schedule comparison, but does not establish superiority over a well-tuned AdEMAMix baseline. A modest independent search at representative scales, or narrower comparative claims, would strengthen the result.

We thank the reviewer for this suggestion. We have in fact already conducted such a hyperparameter sweep on AdEMAMix's parameters (Appendix J.5, Figure 34). Our results show that the prescribed parameters κ=0.75\kappa = 0.75 and β3=1−δ/T\beta_3 = 1 - \delta/T are optimal for 70M and 186M parameter transformers.

More generally, we agree that the AdEMAMix parameters are chosen to mirror the ADANA schedules. This choice is motivated by scaling constraints: AdEMAMix introduces two additional hyperparameters, α\alpha and β3\beta_3, which cannot be tuned at scale. We map them from ADANA to make the link between the two methods explicit, and our small-scale sweeps show that this choice is close to optimal.

A 1.4× multiplier corresponds to about 28.6% less compute relative to AdamW, rather than 40% savings. Moreover, the single-seed setup and endpoint extrapolation leave uncertainty beyond the LR-fit bootstrap in F.3.6. These issues affect the magnitude and reliability of the central practical claim, although they do not negate the observed loss improvements.

In accordance with the reviewer’s point, we will clarify the wording in the updated manuscript to highlight the difference between compute-multiplier and compute-savings. We remark that we defined the compute multiplier terminology in footnote 2 at page 1 which has already been used in previous work [1]. We additionally agree that our runs are done with a single seed due to the compute requirements of training at scale (already 22 H100-years were used for this project; see Table 23). This is standard practice in the scaling literature for that same reason (see, for example [1], [2], [3], [4])

[1] Qiu, Shikai, et al. "Hyperparameter transfer enables consistent gains of matrix-preconditioned optimizers across scales." Advances in Neural Information Processing Systems 38 (2026): 130867-130911.

[2]Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).

[3]Wen, Kaiyue, et al. "Fantastic pretraining optimizers and where to find them." International Conference on Learning Representations. Vol. 2026. 2026.

[4]Semenov, Andrei, Matteo Pagliardini, and Martin Jaggi. "Benchmarking optimizers for large language model pretraining." arXiv preprint arXiv:2509.01440 (2025).

Questions

Does ADANA retain its advantage when AdEMAMix's long-memory, mixing, and warmup parameters are tuned independently with a comparable budget?

In Appendix J.5 and Figure 34, we independently tuned the AdEMAMix hyperparameters and showed that our prescribed choice is optimal at these scales (70M and 186M parameters).

More broadly, the practicality of an optimizer depends largely on how many hyperparameters must be tuned to reach a given performance. Tuning more hyperparameters often yields better performance but becomes impractical at scale. This is why, for all algorithms in this work, we tuned only the learning rate and used predefined scaling rules for the other parameters (momentum EMA, weight decay, etc.).

Could you distinguish the 40% increase in compute efficiency from the approximately 28.6% reduction relative to baseline compute, and use consistent terminology?

The terms compute efficiency and compute multiplier are defined in footnote 2 on page 1, consistent with prior work [1]. As the reviewer suggests, we will additionally state explicitly that a 1.4× compute multiplier (40% compute efficiency gain) is equivalent to a 28.6% reduction in compute relative to the baseline.

Which compute multipliers require extrapolation beyond measured baseline losses? How sensitive are they to interpolation/fitting choices and training-seed variability, beyond the LR-fit uncertainty already reported?

In Figure 1, ADANA, DANA-Star-MK4 and AdEMAMix required extrapolation only at the largest scale (48 heads). Extrapolation is needed only when the lowest AdamW loss is higher than the loss reached by these optimizers. Since this concerns only a few points, the sensitivity of our conclusions is limited. Details on the scaling-law fits and the sensitivity of learning-rate extrapolation are reported in Appendices E and F.

Are the claims conditional on the common N = 20D training rule, or has the optimal model/token allocation been checked separately for the optimizers?

Most of our experiments are performed in the compute-optimal regime N=20DN = 20D. However, we do not believe our claims are conditional on this setting. The main inspiration for this work is [2], where the authors show an improved scaling exponent for DANA relative to SGD that is not tied to a particular allocation rule. Moreover, subsequent work [3] showed that the gains of ADANA are particularly strong in the overtraining regime, where N>20DN > 20D.

[2] Damien Ferbach, Katie Everett, Gauthier Gidel, Elliot Paquette, and Courtney Paquette. "Dimension-adapted Momentum Outscales SGD." NeurIPS 2025 (spotlight).

[3] Everett, Katie, and Shikai Qiu. "Optimizer Memory Schedules for Outscaling the Overtraining Axis." arXiv preprint arXiv:2609.04577 (2026).

Stability scope: L.1/P.3 take p → 0 and a supremum over ε. What conclusions hold for fixed positive ε with the actual learning-rate and weight-decay schedules? Please distinguish worst-case non-uniformity from divergence in a fixed training run.

We thank the reviewer for looking at these appendices. With fixed ϵ>0\epsilon > 0, the algorithms are indeed formally stable: nothing actually diverges. However, once pp is sufficiently small, the updates are scaled by 1/ϵ1/\epsilon and become SGD+momentum updates. On its own this is not a problem, and it can be handled with the usual LR/WD adjustments.

The issue is that our results are stated for oracles with a fixed sparsity, whereas language models have a whole spectrum of sparsity levels, so one can think of gluing together many of our sparse oracle problems with different values of pp. With scalar LR/WD choices, one must then balance how the dense and sparse subproblems respond, and this conflict cannot be fully resolved: either one is conservative on the dense parts, or the sparse parts get a huge LR. Choosing LRs that keep everything stable leads to very slow, ϵ\epsilon-dependent convergence.

Appendices L/P consider the setting where one does not make this conservative choice. This leads to a formal type of instability: there are choices of pp and ϵ\epsilon that drive a fixed LR/WD schedule to instability. This is a worst-case statement, not divergence in a given training run. In practice, the only places where we observed anything approximating this behaviour are the output head and the embedding, and neither led to an actual instability for ADANA, possibly due to ϵ>0\epsilon > 0 doing work.

Practical overhead: What are the measured step-time and peak-memory costs of ADANA and DANA-Star-MK4 relative to AdamW? This would clarify how the FLOP-based gains translate into practical savings.

At scale, the dominant cost of training is gradient computation; the elementwise operations an optimizer performs on the gradients are negligible in comparison. Similarly, peak memory is dominated by activations stored for backpropagation, so an additional optimizer buffer the size of the parameters has a small relative footprint. Please find below measurements of the overhead in training time and memory (on each H100 among a node of 44 of them) on batch 512 runs for ADANA, DANA-mk4 and DANA-star-mk4 against AdamW at various scales. Remarkably, the overhead in memory and steps/s is generally negligible.

n_headparamsAdamW steps/sADANADANA-mk4DANA-star-mk4AdamW peak GBADANADANA-mk4DANA-star-mk4
361.95B0.133−3%−2%−3%64.3≈−1%+2%
402.62B0.102−1%−1%−2%78.9≈≈+4%
443.42B0.072−1%≈−1%66.1≈≈+5%
484.39B0.058−1%−2%−1%80.0≈≈+5%

Most of my concerns have been addressed, so I have raised my rating from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.