- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes COAST, a memory-efficient optimizer that separates the directions receiving updates in a rotated gradient basis from the directions maintaining Adam states. Motivated by approximately flat normalized update magnitudes and similar second-moment profiles across neighboring spectral directions, COAST stores states for a subset of directions and estimates the remaining updates through interpolation. A correction factor compensates for using the current gradient instead of momentum. An extension reduces basis storage by updating the omitted subspace as a single residual block. On LLaMA pre-training, COAST outperforms Adam with 0.42–0.63× Adam’s optimizer memory. It also improves on low-rank methods at similar memory. Halving the storage budget changes perplexity by less than 1% at every scale.
Critical strengths
COAST identifies an unnecessary coupling in low-rank projection methods and provides a concrete way to retain broad update coverage with fewer stored states.
Improvements appear across four model sizes, and the storage-budget experiments show unusually small degradation as state storage decreases.
The paper examines normalized update magnitudes, second-moment similarity, temporal correlations, and correction-factor sensitivity. These analyses make the method more informative than a purely empirical optimizer modification.
Critical weaknesses
Most baseline results are imported from earlier papers; only the 60M Fira result is explicitly reproduced. At the larger scales, the margin over LoRA-Pre is relatively small (Table 1): 16.14 versus 16.36 perplexity at 350M and 13.44 versus 13.53 at 1B, without reported seed variation. Meanwhile, Table 8 shows that the substantial persistent-state savings translate into only a small peak-memory reduction in the measured pretraining setting: 20.75 GB for Adam versus 20.64 GB for COAST and 20.59 GB for COAST with half storage. Timing comparisons omit Adam, Fira, and LoRA-Pre. These results do not negate the storage savings, but leave the practically achievable benefit insufficiently established.
Questions for authors
How are directions beyond the largest stored index handled?
When the basis and stored set change, are moment buffers reset, transported, or reused by buffer position?
3 comments
References
[1] Chen et al., "Fira: Can we achieve full-rank training of llms under low-rank constraint?." in NeurIPS, 2025.
[2] Wang et al., "Taming momentum: Rethinking optimizer states through low-rank approximation." in ICLR, 2026.
[3] Zhu et al., "Apollo: Sgd-like memory, adamw-level performance.", in MLSys, 2025.
Gentle Reminder
Dear Reviewer cKHB,
Thank you again for your time and thoughtful review. As the discussion period is coming to a close, we would like to kindly check whether our response has addressed your concerns. If you have any further questions or comments, we would be happy to address them.
Best regards,
Authors
Rebuttal by Authors
We would like to thank Reviewer cKHB for the careful, balanced assessment and precise questions, and for noting our strengths:
Below, we would like to give detailed responses to your questions.
To ensure a fair comparison, COAST strictly follows the common protocol of GaLore and its follow-up papers [1,2,3] without modification. We checked the reported numbers by reproducing Fira at 60M with its official code: the result, 30.80, is within 1% of the reported 31.06. This 0.26 PPL difference is about one-eighth of COAST’s margin over Fira (2.06 PPL), so it does not affect the comparison.
First, we report results over three seeds (mean ± std) for the pre-training (60M–350M) and fine-tuning experiments.
The table below compares the pre-training performance and state memory of LoRA-Pre and the newly added COAST$_{r/4}.COAST_{r/4}$ achieves lower perplexity than LoRA-Pre at every model size while requiring less state memory, offering a better trade-off between memory and performance. Due to the limited rebuttal period, we report a single-seed result of COAST$_{r/4}$ for 1B model.
The peak memory in Table 8 includes parameters, gradients, activations, and optimizer states, so the peak difference between methods that differ only in the optimizer is determined by the optimizer-state term. In this setting, COAST achieves peak savings on par with GaLore, a representative memory-efficient method. Because peak memory depends heavily on batch size, sequence length, and implementation, prior work, including GaLore, Fira, and LoRA-Pre, uses optimizer-state memory as the primary metric, and we follow this convention. Most importantly, COAST stays within 0.4% of GaLore’s peak memory while substantially lowering perplexity, and it outperforms Adam with less memory.
We compared COAST against the baselines in the paper and the newly added APOLLO on throughput (it/s), total training wall-clock time, and time to reach Adam's final loss. LoRA-Pre was excluded because, although its official GitHub repository exists, the code has not been released there as of this rebuttal.
The stored set is constructed to include the last direction, so every non-stored direction lies between two stored directions and Equation 14 is well-defined. We will state this explicitly in the main text.
Like existing gradient-subspace methods such as GaLore and Fira, COAST neither transforms nor resets the buffer when the subspace is updated; it is kept as is.