- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies optimizer accumulator decay schedules and their effect on the token-efficiency advantage of alternatives to AdamW (e.g., Muon, SOAP) over long training horizons (overtraining regime). The authors sweep multiples of the Chinchilla token budget and tune learning rates to argue that training horizon is an important axis for optimizer evaluation, finding ADANA to be surprisingly competitive in the overtraining regime. Their main contributions are (i) the framing of training horizon as an evaluation axis motivated by the overtraining of current open-weight models, and (ii) a momentum cooldown rule that caps each momentum timescale at the learning-rate decay timescale, which improves ADANA at longer horizons. They also show that a tuned fixed decay rates cannot match a dynamic one, indicating that the optimal decay rate is phase-dependent during training.
Critical strengths
- The core idea and motivation are strong. Connecting token efficiency to data scarcity in the overtraining regime, and arguing that training horizon is an important axis for comparing optimizers, is a well-grounded framing.
- The momentum cooldown rule (capping each momentum timescale at the learning-rate decay timescale) is a clean, well-motivated contribution that demonstrably helps ADANA at longer horizons.
- The finding that a tuned fixed horizon cannot beat a dynamic one, showing the optimal horizon is phase-dependent, is an informative result. The paper is also supported by detailed appendices.
Critical weaknesses
- Presentation severely undercuts the work. The paper discusses “optimizer memory” and “memory schedules” from the abstract onward but only defines what it means by memory around line 168 (four pages in). Readers should not have to guess the central concept for that long, and the term “memory” is itself confusing since it reads as system/hardware memory rather than a gradient horizon. I’d recommend adopting a clearer term such as “optimizer memory horizon” (cf. Ferbach et al.). The experiments section compounds this: lines 239–244 simply forward references figures and facts instead of walking through experiments one at a time, and passages like lines 250–252 (markers, inverted losses, dotted curves) appear without setup and are not understandable as written.
- Experimental design raises fairness and scale concerns. Only the base learning rate is tuned; the betas (central to the gradient-horizon claims) are tuned only at 51M and not at other scales, and AdamW’s beta_1 does not appear to be tuned in Section 4, making the comparison unfair. SOAP is also missing for the 458M model at overtraining factors 4 and 8 without explanation.
- The primary model size (51M) is too small for the weight of analysis placed on it. Only ~25M are transformer parameters, with nearly half going to the output projection, and such a tiny model also needs more seeds for the claims to be reliable. Code is not provided.
Questions for authors
- Line 177: why is the effective memory window M = 1/(1−β)? Could you derive or explain this?
- Line 181: what are Δ and δ? These symbols are not defined.
- Why was SOAP not run for the 458M model at overtraining factors 4 and 8?
- Why is AdamW’s beta_1 not tuned in Section 4, and why are the betas tuned only at 51M and not at other scales?
- What happens if the ADANA schedule is applied to Muon and SOAP?
- Figure 1 (top): what exactly is the “overtraining factor” used for that figure ?
Thank you Reviewer ayPC for the detailed review. Please also see our general comment.
To the weaknesses you raise:
Questions: