acceptodds
Under review as a conference paper at ICLR 2027

Optimizer Memory Schedules for Outscaling the Overtraining Axis

Abstract

We show that optimizer memory schedules can produce an increasing token-efficiency advantage over AdamW as Transformer training horizons grow. We compare AdamW, Muon, SOAP, and ADANA across four model sizes from 51M to 458M parameters and overtraining (OT) factors from 1x to 256x, sweeping the base learning rate at every setting. We introduce a timescale-based momentum cooldown rule that shortens optimizer memory during terminal learning rate decay, producing large long-horizon gains for ADANA that compound with log-time weight decay. With these schedules, ADANA outscales AdamW with a scaling advantage close to that predicted by DANA theory on power-law random features. Longer horizons generally favor longer fixed memory, but ADANA's scaling advantage persists after tuning AdamW's fixed memory separately at each horizon. Muon's token-efficiency advantage over AdamW is roughly constant across OT factors, while SOAP gains further at high OT. ADANA begins behind both matrix-preconditioned optimizers but surpasses Muon and becomes competitive with SOAP at our highest OT factors. These results establish training horizon as an essential axis for optimizer evaluation and design.

Primary area:
optimization
Keywords:
Optimization, momentum, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ayPCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies optimizer accumulator decay schedules and their effect on the token-efficiency advantage of alternatives to AdamW (e.g., Muon, SOAP) over long training horizons (overtraining regime). The authors sweep multiples of the Chinchilla token budget and tune learning rates to argue that training horizon is an important axis for optimizer evaluation, finding ADANA to be surprisingly competitive in the overtraining regime. Their main contributions are (i) the framing of training horizon as an evaluation axis motivated by the overtraining of current open-weight models, and (ii) a momentum cooldown rule that caps each momentum timescale at the learning-rate decay timescale, which improves ADANA at longer horizons. They also show that a tuned fixed decay rates cannot match a dynamic one, indicating that the optimal decay rate is phase-dependent during training.

Critical strengths
  • The core idea and motivation are strong. Connecting token efficiency to data scarcity in the overtraining regime, and arguing that training horizon is an important axis for comparing optimizers, is a well-grounded framing.
  • The momentum cooldown rule (capping each momentum timescale at the learning-rate decay timescale) is a clean, well-motivated contribution that demonstrably helps ADANA at longer horizons.
  • The finding that a tuned fixed horizon cannot beat a dynamic one, showing the optimal horizon is phase-dependent, is an informative result. The paper is also supported by detailed appendices.
Critical weaknesses
  • Presentation severely undercuts the work. The paper discusses “optimizer memory” and “memory schedules” from the abstract onward but only defines what it means by memory around line 168 (four pages in). Readers should not have to guess the central concept for that long, and the term “memory” is itself confusing since it reads as system/hardware memory rather than a gradient horizon. I’d recommend adopting a clearer term such as “optimizer memory horizon” (cf. Ferbach et al.). The experiments section compounds this: lines 239–244 simply forward references figures and facts instead of walking through experiments one at a time, and passages like lines 250–252 (markers, inverted losses, dotted curves) appear without setup and are not understandable as written.
  • Experimental design raises fairness and scale concerns. Only the base learning rate is tuned; the betas (central to the gradient-horizon claims) are tuned only at 51M and not at other scales, and AdamW’s beta_1 does not appear to be tuned in Section 4, making the comparison unfair. SOAP is also missing for the 458M model at overtraining factors 4 and 8 without explanation.
  • The primary model size (51M) is too small for the weight of analysis placed on it. Only ~25M are transformer parameters, with nearly half going to the output projection, and such a tiny model also needs more seeds for the claims to be reliable. Code is not provided.
Questions for authors
  • Line 177: why is the effective memory window M = 1/(1−β)? Could you derive or explain this?
  • Line 181: what are Δ and δ? These symbols are not defined.
  • Why was SOAP not run for the 458M model at overtraining factors 4 and 8?
  • Why is AdamW’s beta_1 not tuned in Section 4, and why are the betas tuned only at 51M and not at other scales?
  • What happens if the ADANA schedule is applied to Muon and SOAP?
  • Figure 1 (top): what exactly is the “overtraining factor” used for that figure ?
1 comment

Thank you Reviewer ayPC for the detailed review. Please also see our general comment.

To the weaknesses you raise:

  • Presentation: we hope the rewritten abstract and contributions section improve the clarify of the core contributions. We defined what we mean by "memory" earlier in the paper.
  • We originally only tuned one momentum parameter per optimizer (beta2 for Adam and SOAP, and beta for Muon) due to compute constraints, because tuning momentum x LR x OT factor was expensive. We have added tuning experiments for beta1 also for Adam and SOAP, and we found that additional tuning of beta1 for Adam only mildly improved the performance and did not improve performance for SOAP. We hope this addresses the fairness concern.
  • We also have added the SOAP 458M points at OT4 and OT8. (These experiments hadn't finished their LR sweeps before the submission deadline, and we accidentally omitted an explanation for these missing points in the original submission).
  • To your point about the primary model size (51M) being too small: we recognize this is a weakness of the work. Under our compute constraints, this was the largest model where we could run the extensive tuning experiments for momentum x LR x OT and the additional appendix ablations for learning rate schedules and weight decay coefficients.
  • We do plan to release the code for the paper but have not completed this yet.

Questions:

  • Short derivation of effective memory window to explain why M = 1/(1-beta):
If we unroll m_t = beta m_{t-1} + (1-beta) g_t, we get:
m_t = (1-beta) * sum over a >= 0 of beta^a g_{t-a},
so a gradient from a updates ago has weight (1-beta) beta^a.

There are two different ways to think about a window length from this:
(i) For beta close to 1, beta^a is approximately exp(-a (1-beta)), so the weight falls by a factor of e every 1/(1-beta) updates.
(ii) The weights sum to one, and the newest gradient has weight (1-beta). A uniform average with the same weight on each gradient would therefore cover 1/(1-beta) gradients.

For example, beta = 0.9 averages over about 10 updates and beta = 0.99 over about 100. We wrote out this sum in Section 3.1 before defining M now.
  • Delta is a hyperparameter of ADANA that controls the momentum window: late in training, the momentum window will include roughly 1 / delta fraction of past training steps.
  • Figure 1 (top): what exactly is the “overtraining factor” used for that figure? The OT factor is tokens /(20 * parameters), defined in the introduction.
  • What happens if the ADANA schedule is applied to Muon and SOAP? This is a great question that we leave for future work. For Muon, some form of momentum schedule is likely beneficial. However, the exact ADANA definition with momentum schedules on both beta1 and beta2, along with the damping term, don't exactly map onto the Muon update rule. For SOAP, it may be challenging to reconcile the frequent basis re-estimations with the long momentum, since accumulating long momentum windows across different basis estimates might harm performance or cause instabilities.
Reviewer_nzPvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares AdamW, Muon, SOAP, and ADANA across different training lengths, using nominal model sizes from 51M to 458M. The smallest model is trained up to 256 times the Chinchilla reference budget. The authors tune the learning rate at each setting, study fixed memory windows and weight decay, and propose momentum cooldown, which shortens memory according to the rate of learning rate decay. ADANA gains an increasing advantage over AdamW as training gets longer. Cooldown and the log time schedule for weight decay improve it further, and its performance relative to Muon and SOAP also changes with training length.

Critical strengths
  1. A useful comparison across training lengths The study shows how the relative performance of optimizers changes as training continues. Tuning the learning rate separately at each model size and training length makes these comparisons more informative than using one setting throughout. The loss curves and efficiency estimates are useful results.

  2. Cooldown improves ADANA in the tested settings The experiments support cooldown as a practical improvement. In Figure 3a, the two runs agree before cooldown starts, and the run with cooldown continues improving afterward. Separate learning rate sweeps support the gain. The ablations also distinguish shortening the memory window from reducing the weight on the long memory component.

Critical weaknesses
  1. The main contribution needs a clearer focus

I had difficulty identifying the paper's central contribution. The optimizer comparison, weight decay experiments, and ADANA cooldown study each make sense, but I did not see a clear main result connecting them. Section 2 already reviews changes in optimizer rankings and the need to retune momentum for different training lengths. Section 5 reports an exponent for uniform weight decay consistent with prior AdamW work. What new result follows from combining these studies and extending the training range? The main new rule, cooldown, is motivated by ADANA and gives its main gains there. It is unclear how far this supports the paper's broader claim about optimizer memory schedules.

I am also unconvinced that weight decay should be classified as optimizer memory here. The four optimizers store past gradients or their statistics in their internal states, rather than storing past model parameters. Weight decay affects how past updates remain in the model parameters. It does not directly control how the optimizer states average past gradients. Section 3.1 and Appendix E.4 describe memory in the parameters. I would keep this separate from memory in the optimizer states.

  1. The proposed cooldown explanation is not directly tested

Section 6 suggests that recent gradients may become more useful as the learning rate approaches zero. Appendix G.1 proposes that the learning rate can change faster than the momentum state can respond. I do not see direct evidence for this connection. A small learning rate does not by itself show that long memory has become inappropriate. Shortening memory improves the final loss, but this does not establish that the proposed mismatch caused the plateau. Figure 2b compares optimal base learning rates for different fixed memory windows. That does not directly explain why memory should be shortened during the final learning rate decay.

A comparison with a rule that shortens memory based only on training progress would help test whether matching the two timescales adds anything. The paper leaves further analysis of the mechanism to future work. I would therefore describe cooldown as a useful heuristic whose explanation remains to be tested. It is also unclear how well the rule applies elsewhere. Appendix G.4 states that it has not been tested with Muon or SOAP, or with learning rate schedules other than cosine decay to zero.

  1. Important comparisons and evidence from larger models are missing

AdEMAMix is an important missing comparison. It also uses fast and slow momentum and schedules the slow momentum and mixing weight. The paper discusses how retuning its slow momentum coefficient changes the results for longer training. I would therefore expect a direct comparison that keeps its original schedules and tunes it for the training length. Those schedules serve a different purpose from cooldown, but the method is closely related to this paper's topic. In addition, AdamW and SOAP's fixed memory searches vary β2\beta_2 while fixing β1\beta_1 (Appendix F.1). The cited evidence concerns changes in batch size, so it does not establish that this choice is sufficient when training length changes. A small search over β1\beta_1 would make the comparison more complete.

The models are also relatively small, ranging from 51M to 458M. The maximum overtraining factors are 256×, 64×, 16×, and 8× as model size increases, and SOAP reaches only 2× at 458M. Fixed memory tuning is limited to 51M. Some efficiency estimates at high overtraining factors also rely on extrapolating the AdamW results. The paper acknowledges these limits, but they still weaken the evidence for applying its conclusions to larger language models.

Questions for authors

1 Contribution: Which result is the main new finding beyond prior ADANA work and the existing log time schedule for weight decay? Which claims are intended to apply across optimizers? Please distinguish memory in the parameters under weight decay from memory in the optimizer states.

2 Cooldown. Could the authors compare their rule with one that shortens memory based only on training progress, or vary the timing of learning rate decay and memory shortening independently? A test with a constant learning rate needs a separate rule for shortening memory, since the proposed cooldown does not activate there. Removing ADANA's adaptive normalization while retaining its memory and gradient mixture, and retuning the learning rate, would separately test whether the benefit depends on that component. This would not establish the timescale explanation by itself. Are there results for other learning rate schedules?

3 Comparisons and scope. Could the authors compare with AdEMAMix using its original schedules and tuning it for the training length? For AdamW and SOAP, a small search over or over β1\beta_1 and β2\beta_2 together, would address the concern about first moment memory. If these comparisons or evidence from larger models are unavailable, the corresponding conclusions should be limited to the settings actually tested.

1 comment

Thank you Reviewer nzPv for the detailed review and useful feedback. Please also see our top-level comment.

Clarity of main contribution: We focused the core contributions around the outscaling performance of ADANA, especially in light of the improved empirical performance in the revision experiments (Figure 1) due to a better hyperparameter setting for delta. We rewrote the abstract and contributions and hope this improves the clarity.

Weight decay and memory: We also agree with your comment that weight decay should not necessarily be framed as memory. We were originally following the framing in prior work on ADANA (Ferbach et al 2025), which frames the log-time weight decay as another "memory schedule" so that the ADANA momentum schedules causes the memory window to grow linearly throughout training, and the log-time weight decay schedule causes the "forgetting window" to have the same linear growth throughout training. However, especially in light of the recent burst of literature on effective learning rate perspectives for weight decay and hyperball methods, we agree it's better not to frame weight decay as memory. We have edited the writing to frame the momentum schedules (both the ADANA schedule and our cooldown schedule) as memory schedules and treat weight decay as its own entity.

Cooldown: We changed the cooldown formulation to be based on the training fraction rather than the instantaneous rate of change in the learning rate. We also added some experiments (see Appendix H.6) using cooldown on different learning rate schedules to test whether the mechanism depends on the learning rate decay, as you suggested. With a constant learning rate followed by linear decay over the last 20% or 40% of training, cooldown reduces the final loss by 7 to 8 mnats. With a constant learning rate and no decay, the runs with cooldown diverge near the end of training, so cooldown has to be combined with learning rate decay. We agree that cooldown is currently a useful heuristic whose explanation is not fully tested.

AdEMAMix: We added an initial set of AdEMAMix experiments in Appendix G. We keep its original schedules and sweep the learning rate and beta3 at each horizon on the 51M model. AdEMAMix is better than ADANA up to 4x OT, the two are within 3 mnats from 8x to 32x, and ADANA is better at 64x by 5.5 mnats. The best beta3 increases with the horizon.

Tuning beta1: We added some beta1 tuning experiments at 51M and 124M in Appendix F.3. The takeaway is that AdamW performance can slightly improve by increasing beta1 to 0.95 instead of 0.9, but this is a small fraction of the performance gap to ADANA. For SOAP, we used beta1=0.95 and our experiments using beta1=0.9 performed slightly worse.

Reviewer_HCLTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper compares a few optimizers (AdamW, SOAP, Muon, ADANA) at different training horizons under a thorough hyperparameter sweep.

  1. Compare optimizers across training horizons.
  2. Show that optimal hyperparameters for the optimizer may change depending on the training horizon.
  3. In the tested setting, ADANA improves over AdamW as the training horizon increases.
  4. Propose to reduce memory towards the end of training.
  5. The proposed momentum cooldown benefits ADANA moreso than AdamW.
Critical strengths
  1. Thorough hyperparameter sweep over training horizons.

The paper tests models from 51M to 458M in size and considers hyperparameters such as learning rate, learning rate schedule, weight decay, weight decay schedule, and memory (beta values or schedule for optimizer). These hyperparameters are tested across 1x to 256x (or less depending on the model size) chinchilla optimal token to parameter ratio.

This leads to findings such as how the memory should increase with training tokens and how optimal learning rate should decrease with memory (Figure 2).

  1. Proposes a competitive optimizer at long training horizons

Under each optimizer's best settings, ADANA seems to be slightly worse than the other optimizers at shorter training horizons but actually improves at longer training horizons.

Critical weaknesses
  1. Clarity

While the paper contains lots of experiments, the content is difficult to read.

  1. Missing data points in the plots

In the plots (Figures 1 - 4), some points are hollowed out and are connected with a dotted line. I assume that means extrapolation. If so, I'm not sure why some points on the lower end at 1x chinchilla optimal are hollowed out, since I would imagine those experiments are cheaper to run.

Questions for authors
  • I would find the contributions more clear if they were more succinct (e.g. 3 contributions) and focus on key results. For example, the change from linear to cosine learning rate schedule is listed as a contribution but is discussed in the Appendix.

  • For Figures 3 and 4 where the efficiency gain is extrapolated, it would be nice to actually run as many experiments as compute allows to show those results.

1 comment

Thank you Reviewer HCLT for your comments and positive score on the paper. Please see our top-level comment as well.

  1. Contributions. The introduction now lists three numbered contributions. The learning rate schedule is no longer listed as a contribution.

  2. Hollow points and dotted lines. Each caption now explains them. Translucent markers with dotted segments are comparisons where the loss is below the lowest loss we measured for AdamW at that model size, so the token multiplier extrapolates the AdamW fit. Open markers are learning rate sweeps that are still in progress.

  3. Hollow points at 1x. In the submission, ADANA was worse than AdamW at 1x, so matching its loss needed fewer tokens than AdamW's shortest run, which is also an extrapolation. With delta=20, ADANA is at or above AdamW at 1x at every model size, and no 1x point in Figure 1 is extrapolated.

  4. Extrapolated points at long horizons. These are points where an optimizer reaches a lower loss than AdamW's longest run. Removing them needs AdamW runs longer than our longest ones, which we could not complete in the rebuttal period. We mark these points in every figure.

  5. Clarity. We rewrote the abstract and the contributions to emphasize that the core contribution in the paper is the outscaling behavior of the ADANA optimizer.

Comments on the paper

1 comment

We thank the reviewers for the useful feedback and sincerely apologize for the delay in our rebuttal.

To respond to the points noted in multiple reviews:

  • All reviewers noted that the presentation of the paper needed significant improvements. We redesigned several main text figures and rewrote the abstract, contributions section, and the momentum cooldown section.
  • We discovered that we had poorly tuned the delta hyperparameter in ADANA and that our original choice of delta=8 was significantly limiting its performance. We reran the main comparison (Figure 1) and the scaling analysis (Figure 6) using delta=20. With this setting, ADANA slightly outperforms AdamW at 1x OT and then has an increasing advantage as we scale the overtraining factor. The ablations in Sections 4 to 6 still use delta=8. ADANA now surpasses both Muon and SOAP at the highest OT factor of every model size, and we have emphasized this as the core contribution of the paper.
  • We changed the formulation of the momentum cooldown so that the cooldown starts based on the training fraction rather than on the instantaneous decay rate of the learning rate schedule. This formulation can be applied under any learning rate schedule, and we test it under two additional schedules (Appendix H.6). The cooldown ablations in Figure 4 still use the original formulation.
  • We have added the requested experiments for beta1 tuning, an initial set of AdEMAMix experiments, and measurements of the noise floor based on multiple seeds.
  • The original submission was missing data for SOAP at 458M for 4x and 8x OT. These points are now included.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.