acceptodds
Under review as a conference paper at ICLR 2027

Rotate All, Store Few: Decoupling Coverage from Storage for Memory-Efficient Training

Abstract

Adam improves when the gradient is normalized in its singular basis, which we call . Full-rank rotation exceeds the memory of Adam by storing a basis and states for every direction, whereas low-rank gradient projection rotates only the top- directions but degrades sharply as decreases. This is because a single rank sets both the , the number of rotated directions, and the , the number of directions with optimizer states. The two decisions nonetheless need not be coupled. The rotation coverage should be full since the whitened update magnitude stays nearly flat along the spectral index, whereas the state storage can be reduced since adjacent second moments have high cosine similarity. We, therefore, propose , which decouples rotation verage nd state orage. rotates every direction but maintains optimizer states for only stored directions and estimates the updates of the rest from their stored neighbors. On LLaMA pre-training, outperforms Adam with – Adam's optimizer memory. It also improves on low-rank methods at similar memory. Halving the storage budget changes perplexity by less than 1% at every scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Efficient Training, Singular Value Decomposition

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_cKHBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes COAST, a memory-efficient optimizer that separates the directions receiving updates in a rotated gradient basis from the directions maintaining Adam states. Motivated by approximately flat normalized update magnitudes and similar second-moment profiles across neighboring spectral directions, COAST stores states for a subset of directions and estimates the remaining updates through interpolation. A correction factor compensates for using the current gradient instead of momentum. An extension reduces basis storage by updating the omitted subspace as a single residual block. On LLaMA pre-training, COAST outperforms Adam with 0.42–0.63× Adam’s optimizer memory. It also improves on low-rank methods at similar memory. Halving the storage budget changes perplexity by less than 1% at every scale.

Critical strengths

COAST identifies an unnecessary coupling in low-rank projection methods and provides a concrete way to retain broad update coverage with fewer stored states.

Improvements appear across four model sizes, and the storage-budget experiments show unusually small degradation as state storage decreases.

The paper examines normalized update magnitudes, second-moment similarity, temporal correlations, and correction-factor sensitivity. These analyses make the method more informative than a purely empirical optimizer modification.

Critical weaknesses

Most baseline results are imported from earlier papers; only the 60M Fira result is explicitly reproduced. At the larger scales, the margin over LoRA-Pre is relatively small (Table 1): 16.14 versus 16.36 perplexity at 350M and 13.44 versus 13.53 at 1B, without reported seed variation. Meanwhile, Table 8 shows that the substantial persistent-state savings translate into only a small peak-memory reduction in the measured pretraining setting: 20.75 GB for Adam versus 20.64 GB for COAST and 20.59 GB for COAST with half storage. Timing comparisons omit Adam, Fira, and LoRA-Pre. These results do not negate the storage savings, but leave the practically achievable benefit insufficiently established.

Questions for authors

How are directions beyond the largest stored index handled?

When the basis and stored set change, are moment buffers reset, transported, or reused by buffer position?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We would like to thank Reviewer cKHB for the careful, balanced assessment and precise questions, and for noting our strengths:

  • It identifies an unnecessary coupling in low-rank projection and keeps broad coverage with fewer stored states.
  • Gains span four model sizes, with unusually small degradation as storage shrinks.
  • The analyses make it more than a purely empirical optimizer modification.

Below, we would like to give detailed responses to your questions.


(W1) Most baseline results are imported from earlier papers; only the 60M Fira result is explicitly reproduced.

To ensure a fair comparison, COAST strictly follows the common protocol of GaLore and its follow-up papers [1,2,3] without modification. We checked the reported numbers by reproducing Fira at 60M with its official code: the result, 30.80, is within 1% of the reported 31.06. This 0.26 PPL difference is about one-eighth of COAST’s margin over Fira (2.06 PPL), so it does not affect the comparison.

(W2) At the larger scales, the margin over LoRA-Pre is relatively small (Table 1) without reported seed variation.

First, we report results over three seeds (mean ± std) for the pre-training (60M–350M) and fine-tuning experiments.

  • pre-training
Method60M130M350M
COAST$_r$28.82$_{±0.07}$22.12$_{±0.03}$16.10$_{±0.03}$
COAST$_{r/4}$29.12$_{±0.03}$22.23$_{±0.03}$16.17$_{±0.03}$
  • fine-tuning
MethodMemoryBoolQPIQASIQAHellaSwagWinoGrandeARC-eARC-cOBQAAvg.
Fira14.34G69.482.678.076.881.282.264.480.876.9
COAST$_r$14.34G69.2$_{±0.9}$82.6$_{±0.4}$77.7$_{±0.9}$85.4$_{±0.3}$80.6$_{±0.1}$81.9$_{±0.4}$66.1$_{±0.6}$79.3$_{±1.0}$77.9$_{±0.1}$
COAST$_{r/2}$14.28G69.6$_{±0.2}$83.1$_{±0.4}$77.7$_{±0.7}$85.8$_{±0.4}$81.2$_{±0.4}$81.7$_{±0.1}$65.6$_{±0.3}$78.5$_{±0.2}$77.9$_{±0.1}$

The table below compares the pre-training performance and state memory of LoRA-Pre and the newly added COAST$_{r/4}.COAST. COAST_{r/4}$ achieves lower perplexity than LoRA-Pre at every model size while requiring less state memory, offering a better trade-off between memory and performance. Due to the limited rebuttal period, we report a single-seed result of COAST$_{r/4}$ for 1B model.

PPL (State memory compared to Adam)60M130M350M1B
LoRA-Pre Adam32.57 (0.40×)23.78 (0.53×)16.36 (0.40×)13.53 (0.40×)
COAST$_{r/4}$29.12$_{±0.03}$ (0.35×)22.23$_{±0.03}$ (0.38×)16.17$_{±0.03}$ (0.35×)13.42 (0.35×)

(W3) Table 8 shows that the substantial persistent-state savings translate into only a small peak-memory reduction in the measured pretraining setting: 20.75 GB for Adam versus 20.64 GB for COAST and 20.59 GB for COAST with half storage.

The peak memory in Table 8 includes parameters, gradients, activations, and optimizer states, so the peak difference between methods that differ only in the optimizer is determined by the optimizer-state term. In this setting, COAST achieves peak savings on par with GaLore, a representative memory-efficient method. Because peak memory depends heavily on batch size, sequence length, and implementation, prior work, including GaLore, Fira, and LoRA-Pre, uses optimizer-state memory as the primary metric, and we follow this convention. Most importantly, COAST stays within 0.4% of GaLore’s peak memory while substantially lowering perplexity, and it outperforms Adam with less memory.

(W4) Timing comparisons omit Adam, Fira, and LoRA-Pre.

We compared COAST against the baselines in the paper and the newly added APOLLO on throughput (it/s), total training wall-clock time, and time to reach Adam's final loss. LoRA-Pre was excluded because, although its official GitHub repository exists, the code has not been released there as of this rebuttal.

Throughput (it/s)Total training wall-clock timeTime to reach Adam's final loss
Adam1.082:46:12Reference (10K steps)
GaLore1.092:44:03Not reached
Fira1.062:45:036k
Muon1.072:46:205k
APOLLO1.023:01:037k
COAST$_r$1.052:49:275k
COAST$_{r/2}$1.062:49:235k
COAST$_{r/4}$1.062:48:045k

(Q1) How are directions beyond the largest stored index handled?

The stored set is constructed to include the last direction, so every non-stored direction lies between two stored directions and Equation 14 is well-defined. We will state this explicitly in the main text.

(Q2) When the basis and stored set change, are moment buffers reset, transported, or reused by buffer position?

Like existing gradient-subspace methods such as GaLore and Fira, COAST neither transforms nor resets the buffer when the subspace is updated; it is kept as is.

AuthorsOpenReview ↗

References

[1] Chen et al., "Fira: Can we achieve full-rank training of llms under low-rank constraint?." in NeurIPS, 2025.

[2] Wang et al., "Taming momentum: Rethinking optimizer states through low-rank approximation." in ICLR, 2026.

[3] Zhu et al., "Apollo: Sgd-like memory, adamw-level performance.",  in MLSys, 2025.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer cKHB,

Thank you again for your time and thoughtful review. As the discussion period is coming to a close, we would like to kindly check whether our response has addressed your concerns. If you have any further questions or comments, we would be happy to address them.

Best regards,

Authors

Reviewer_mdGuOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes COAST, which separates rotation coverage from optimizer-state storage. It rotates gradients in a full singular-vector basis but keeps Adam moments for only a subset of directions. Updates for the remaining directions use interpolated second moments and update norms from stored neighbors. Experiments cover LLaMA pre-training and LoRA fine-tuning, and show the benefits of COAST compared to other memory-efficient variants.

Critical strengths
  1. Separating rotation coverage from state storage is an interesting approach to do memory-efficient optimization, making it similar to a memory-efficient version of rotation-based optimizers such as SOAP. The observations of nearly flat normalized update magnitudes and similar second-moment profiles provide a useful motivation for retaining update directions while reducing stored states.
  2. Extensive experiments are included to prove the effectiveness of the method.
Critical weaknesses
  1. The practical performance–memory advantage is not sufficiently established. The basic method stores m2+2nrm^2 + 2nr elements, ignoring small costs. This is at least near Muon's mnmn-element momentum buffer. Also, given the relations in the rotation based methods such as Muon and SOAP, I think it would worth doing a comparison with Muon. Arguing that COAST can work under smaller memory budget as in Section 5 may require comparison in a larger scale.
  2. The SFT setting is strange, although it follows what FIRA did. As the paper mentioned, COAST and other optimizers are applied to LoRA experiment, which actually provide a limited test of memory efficiency: Table 2 shows only 0.11 GB of estimated savings over LoRA with Adam (14.34 vs. 14.45 GB). I don't think this is a valid setting to establish a memory-efficient optimizer, since the optimizer states cost is just minor. COSAT brings benefits in this setting, then we need to figure out why it is specifically good for LoRA. Also, in the LoRA setting, it actually does not matter how large the optimizer state is, I think if this is the comparison setting, then Muon or even one-sided SOAP/Shampoo should also be considered, since they actually also introduce only minor memory footprint. Moreover, the full-fine-tuning baseline is substantially weaker than LoRA (58.6 vs. 74.7 average accuracy), and this large unexplained gap raises serious concerns about the evaluation setting.
Questions for authors
  1. Can you add the baseline of Muon in the pretraining setup, to see how COAST is compared to it, since the memory cost is similar?
  2. Can you clarify the source and configuration of the full-fine-tuning baseline? It looks unnaturally low. I would suggest doing full finetuning for the SFT experiment to validate the performance of a memory-efficient optimizer, or maybe we can explore why COAST is the best optimizer for LoRA?
8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We truly appreciate Reviewer mdGu’s thoughtful evaluation and the valuable perspective relating our work to rotation-based optimizers, as well as the recognition of our strengths in:

  • Decoupling rotation coverage from state storage is an interesting approach to memory-efficient optimization.
  • The observations on flat update magnitudes and similar second-moment profiles provide a useful motivation.
  • Extensive experiments are included to demonstrate the effectiveness of the method.

Below, we would like to give detailed responses to your questions.


(W1) The practical performance–memory advantage is not sufficiently established. This is at least near Muon's mnmn-element momentum buffer. It would worth doing a comparison with Muon.

Due to the limited rebuttal period, we added Muon [1], Adafactor [2], FRUGAL [3], and APOLLO [4] at 60M–350M. For Muon, we use the values reported by LoRA-Pre under the same protocol; for APOLLO, the values reported in its paper. For FRUGAL (official code) and Adafactor, we tuned the learning rate over {1e−2,1e−3}\{1e-2, 1e-3\} and report the best result, whereas our baseline uses a fixed learning rate of 1e-3. For Adam, for a fair comparison, we use the value shared by GaLore, Fira, and LoRA-Pre, which GaLore reports as the best over lr∈{0.01,0.005,0.001,0.0005,0.0001}lr \in \{0.01, 0.005, 0.001, 0.0005, 0.0001\}.

Method60M130M350M
Adam34.09 (0.36G)25.08 (0.76G)18.80 (2.06G)
Muon28.43 (0.26G)21.86 (0.54G)16.17 (1.43G)
GaLore34.88 (0.24G)25.36 (0.52G)18.95 (1.22G)
LoRA34.99 (0.36G)33.92 (0.80G)25.58 (1.76G)
Fira30.80 (0.24G)24.51 (0.52G)17.22 (1.22G)
LoRA-Pre (Adam)32.57 (0.24G)23.78 (0.56G)16.36 (1.31G)
LoRA-Pre (Muon)30.76 (0.23G)23.05 (0.47G)16.97 (1.07G)
FRUGAL36.45 (0.23G)26.37 (0.46G)22.03 (1.14G)
Adafactor32.21 (0.12G)23.80 (0.25G)17.01 (0.69G)
APOLLO w. SVD31.26 (0.24G)22.84 (0.52G)16.67 (1.22G)
APOLLO31.55 (0.24G)22.94 (0.52G)16.85 (1.22G)
APOLLO†31.26 (0.18G)23.18 (0.39G)16.98 (0.95G)
APOLLO-Mini31.93 (0.12G)23.53 (0.25G)17.18 (0.69G)
APOLLO-Mini‡30.95 (0.12G)22.85 (0.25G)16.63 (0.69G)
COAST$_r$28.74 (0.26G)22.11 (0.59G)16.14 (1.48G)
COAST$_{r/2}$28.95 (0.25G)22.14 (0.53G)16.15 (1.33G)
COAST$_{r/4}$29.09 (0.24G)22.23 (0.51G)16.19 (1.26G)

(W2) Arguing that COAST can work under smaller memory budget as in Section 5 may require comparison in a larger scale.

Thanks for kind suggestion. We run COAST with basis-reduction experiments in Section 5 at 60M, 130M, and 350M with γ=0.5\gamma = 0.5 for storage budgets rr and r/4r/4. Below table shows that the basis reduction remains effective at larger scales.

60M130M350M
COAST$_r$28.7422.1116.14
COAST$_r$ (γ=0.5\gamma=0.5)29.0422.1616.23
COAST$_{r/4}$29.0922.2316.19
COAST$_{r/4}$ (γ=0.5\gamma=0.5)29.3622.2816.28
AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

(W3) The SFT setting is strange, although it follows what FIRA did. As the paper mentioned, COAST and other optimizers are applied to LoRA experiment, which actually provide a limited test of memory efficiency: Table 2 shows only 0.11 GB of estimated savings over LoRA with Adam (14.34 vs. 14.45 GB). I don't think this is a valid setting to establish a memory-efficient optimizer, since the optimizer states cost is just minor. COSAT brings benefits in this setting, then we need to figure out why it is specifically good for LoRA.

Existing papers, including Fira, Adam-mini [5], and SGG [6], evaluate their optimizers on LoRA adapters in place of Adam. Since the adapters are kept identical across methods and only the optimizer is replaced, this setting isolates the effect of the optimizer and allows a direct comparison with prior methods. The memory in Table 2 counts everything, including the frozen 7B backbone that is identical across methods. The memory of only optimizer states is as follows.

Methodoptimizer state (GB)
Full-rank28
LoRA0.2244
GaLore0.1125
Fira0.1125
COAST$_r$0.1129
COAST$_{r/2}$0.0568

The 0.11 GB noted by the reviewer is a 33% reduction in the memory for gradients and optimizer states (0.337 → 0.225 GB with COAST$_r$). Combining LoRA with COAST$_{r/2}$ halves it relative to LoRA with Adam (0.337 → 0.169 GB) and reduces the optimizer state alone to a quarter (0.2244 → 0.0568 GB), while improving average accuracy by 3.1 points (74.7 → 77.8), which is a meaningful saving on top of LoRA.

Nevertheless, to fully address the reviewer's concern, we additionally apply COAST to full-weight fine-tuning rather than to LoRA adapters. We fine-tune LLaMA-3.2-1B on the commonsense reasoning benchmark, strictly following the setting of APOLLO [4] without any additional tuning for our method, so that the comparison is fair. As shown in the table below, COAST still outperforms the existing memory-efficient optimizers in this setting.

MethodsWGPIQASIQAOBQAHSBoolQARC-EARC-CAverage
AdamW68.1976.1272.3669.0069.1964.3472.2255.1268.07
LoRA67.5663.2871.6568.2019.1363.5867.3052.9959.21
DoRA68.9874.7072.4764.8063.9364.0169.3252.8266.38
GaLore62.7572.6368.1762.2047.8158.9968.9447.6161.14
Fira71.8277.2073.0869.0068.2164.3173.4054.7868.98
APOLLO w. SVD70.8877.6972.5270.6068.1963.0074.0355.7269.08
APOLLO70.4076.9372.7270.6063.7562.6973.4055.2068.21
APOLLO-Mini67.6476.5072.8869.6066.5464.2272.9855.4668.23
COAST$_r$69.8577.5374.3175.065.5066.6477.0260.5870.80
COAST$_{r/2}$71.0378.0773.1374.265.8466.2776.5662.5470.96

(W4) Also, in the LoRA setting, it actually does not matter how large the optimizer state is, I think if this is the comparison setting, then Muon or even one-sided SOAP/Shampoo should also be considered, since they actually also introduce only minor memory footprint.

Given the limited rebuttal period, we prioritized adding Muon [1] as a baseline. For a fair comparison, we tuned its learning rate over {1e−4,2e−4,3e−4}\{1e-4, 2e-4, 3e-4\} and report the result of the best configuration. For COAST, we additionally ran three random seeds and report the mean and standard deviation. Note that all memory-efficient methods, including COAST, use a fixed lr=1e−4lr=1e-4. (Please refer to the response in W5 for the updated results of the full-rank.)

MethodMemoryBoolQPIQASIQAHellaSwagWinoGrandeARC-eARC-cOBQAAvg.
Full-rank56.00G69.984.278.992.383.386.672.883.481.4
Fira14.34G69.482.678.076.881.282.264.480.876.9
Muon14.34G69.381.877.083.079.481.065.879.277.1
COAST$_r$14.34G69.2$_{±0.9}$82.6$_{±0.4}$77.7$_{±0.9}$85.4$_{±0.3}$80.6$_{±0.1}$81.9$_{±0.4}$66.1$_{±0.6}$79.3$_{±1.0}$77.9$_{±0.1}$
COAST$_{r/2}$14.28G69.6$_{±0.2}$83.1$_{±0.4}$77.7$_{±0.7}$85.8$_{±0.4}$81.2$_{±0.4}$81.7$_{±0.1}$65.6$_{±0.3}$78.5$_{±0.2}$77.9$_{±0.1}$
AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

(W5) Moreover, the full-fine-tuning baseline is substantially weaker than LoRA (58.6 vs. 74.7 average accuracy), and this large unexplained gap raises serious concerns about the evaluation setting.

Thank you for pointing this out. For a consistent comparison, we originally took the Full-rank result directly from Fira, whose fine-tuning experiments were conducted with the LLM-Adapters framework [7]. However, this result appears to be under-tuned: its average accuracy (58.6) falls well below that of every parameter-efficient method in Table 2, including LoRA (74.7).

To verify this, we ran full fine-tuning ourselves, following the LLM-Adapters setting [7] (batch size 1616 with micro-batch size 44, 33 epochs, and a linear learning-rate schedule with 100100 warm-up steps) and tuning only the learning rate over {3e−5,4e−5}\{3e-5, 4e-5\}. The better of the two runs reaches an average accuracy of 78.4, nearly 20 points above the number reported in Fira.

In addition, [8] reports 81.4 for full fine-tuning of LLaMA-7B under the same LLM-Adapters protocol. Since this is higher than our own result (78.4), we adopt it as the Full-rank baseline, so that the parameter-efficient methods, including ours, are compared against the strongest available full fine-tuning result. The corrected row is shown in the table in our response to W4. With this correction, full fine-tuning is, as expected, the strongest row in Table 2, at about 3.9× the memory of the parameter-efficient methods, and the comparison among those methods is unchanged.

(Q1) Can you add the baseline of Muon in the pretraining setup?

Please refer to the response of W1.

(Q2) Can you clarify the source and configuration of the full-fine-tuning baseline? It looks unnaturally low. I would suggest doing full finetuning for the SFT experiment to validate the performance of a memory-efficient optimizer, or maybe we can explore why COAST is the best optimizer for LoRA?

Please refer to the response of W3, W4 and W5.


References

[1] Jordan et al., “Muon: An optimizer for hidden layers in neural networks”, https://kellerjordan.github.io/posts/muon, 2024.

[2] Shazeer, Noam, and Mitchell Stern. "Adafactor: Adaptive learning rates with sublinear memory cost." in ICML, 2018.

[3] Zmushko et al., "FRUGAL: Memory-efficient optimization by reducing state overhead for scalable training." in ICML, 2025.

[4] Zhu et al., "Apollo: Sgd-like memory, adamw-level performance.",  in MLSys, 2025.

[5] Zhang et al., "Adam-mini: Use fewer learning rates to gain more." in ICLR, 2025.

[6] Li et al., "Taming LLMs with Gradient Grouping." in ACL, 2025

[7] Hu et al., "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models." in EMNLP, 2023. https://github.com/AGI-Edgerunners/LLM-Adapters

[8] Si et al., "Unleashing the power of task-specific directions in parameter efficient fine-tuning." in ICLR, 2025.

Reviewer_KzrCOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

COAST separates the directions receiving rotated updates from those retaining optimizer states. It stores Adam statistics for selected directions and estimates the remaining updates through interpolation and a momentum-based scale correction. The main contribution is showing that broad update coverage can be retained with substantially fewer states, although the computational trade-off remains incompletely evaluated.

Critical strengths
  1. Separating rotation coverage from state storage offers a useful perspective beyond simply choosing a smaller projection rank.
  2. Pretraining results consistently improve over Adam and several low-rank baselines, while halving the storage budget causes little deterioration. The budget and component ablations support the practical value of the design.
Critical weaknesses
  1. The block correction is not fully justified. Appendix H, Eq. (31), substitutes expected squared norms for realized ones and invokes informal weak dependence to argue that the error is small. Stationarity alone does not ensure this concentration; explicit additional assumptions and an error bound are needed.

  2. Practical efficiency remains insufficiently validated. Pretraining stops at 1B, and the 7B experiment only trains LoRA adapters, leaving benefits under larger, more memory-constrained full-model training unclear. Timing comparisons cover only GaLore, which also requires SVD, and omit total runtime and time to matched loss. Performance and timing comparisons with Adam/AdamW, Muon, and especially SVD-free APOLLO (Zhu et al., 2025) would better establish whether COAST’s rotation justifies its computational cost.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We are grateful to Reviewer KzrC for the time and care devoted to our submission and for the rigorous, constructive feedback, while also recognizing our strengths in:

  • Separating rotation coverage from state storage offers a useful perspective beyond choosing a smaller rank.
  • Pre-training results consistently improve over Adam and low-rank baselines, with little deterioration at half the storage.
  • The budget and component ablations support the practical value of the design.

Below, we would like to give detailed responses to your concerns.


(W1) The block correction is not fully justified. Appendix H, Eq. (31), substitutes expected squared norms for realized ones and invokes informal weak dependence to argue that the error is small. Stationarity alone does not ensure this concentration; explicit additional assumptions and an error bound are needed.

Thank you for the careful reading. We agree: Eq. (31) replaces expected squared norms with realized ones, and the submission gave neither the conditions for this nor an error bound. The revision adds both, with a measurement: Eq. (31) holds with a relative RMS error of 0.025 on average and at most 0.045. The algorithm is unchanged.

Let X:=(ρeff)2=∥M^tBD∥F2X:=(\rho^{\mathrm{eff}})^2=\lVert\hat M^B_tD\rVert_F^2, Y:=∥GtresD∥F2Y:=\lVert G^{\mathrm{res}}_tD\rVert_F^2, κ:=(1−β1)/(1+β1)\kappa:=\sqrt{(1-\beta_1)/(1+\beta_1)} and Δ:=X−κY\Delta:=\sqrt X-\kappa\sqrt Y. Eq. (31) is the claim Δ=0\Delta=0. Eqs. (30) and (22) are cXc\sqrt X and c κYc\,\kappa\sqrt Y times the same unit-norm matrix, so their difference has Frobenius norm c ∣Δ∣c\,\lvert\Delta\rvert.

Proposition H.2. For X,Y≥0X,Y\ge0 with finite second moments, let σX,σY\sigma_X,\sigma_Y be their relative standard deviations and μ:=κEY/EX\mu:=\kappa\sqrt{\mathbb{E}Y/\mathbb{E}X}. Then

E[Δ2]/EX  ≤  σX+∣1−μ∣+μ σY.\sqrt{\mathbb{E}[\Delta^2]/\mathbb{E}X}\;\le\;\sigma_X+\lvert1-\mu\rvert+\mu\,\sigma_Y .

Proof. Write Δ=(X−EX)+(EX−κEY)+κ(EY−Y)\Delta=(\sqrt X-\sqrt{\mathbb{E}X})+(\sqrt{\mathbb{E}X}-\kappa\sqrt{\mathbb{E}Y})+\kappa(\sqrt{\mathbb{E}Y}-\sqrt Y), apply Minkowski's inequality, and use ∣x−1∣≤∣x−1∣\lvert\sqrt x-1\rvert\le\lvert x-1\rvert for x≥0x\ge0. □\square

The inequality holds for any distribution and makes the conditions for Eq. (31) explicit: (i) XX and YY concentrate (σX,σY≪1\sigma_X,\sigma_Y\ll1), which Assumption 3.2 does not imply, as you point out, and which we verify empirically rather than derive; (ii) μ≈1\mu\approx1, which Lemma 3.3 gives for fixed DD in the uncorrelated case.

[Measurement.]

LLaMA-60M in the setting of Appendix G, with BB and the DD of Eq. (22) defined at γ=0.25\gamma=0.25, r=128r=128; expectations are averages over the 10,000 training steps. Mean (max) over 18 modules (q, k, v, up, gate, down projections of layers 0, 4 and 7).

LHSσX\sigma_XσY\sigma_Y∣1−μ∣\lvert1-\mu\rvertRHS
0.025 (0.045)0.015 (0.025)0.036 (0.053)0.017 (0.040)0.070 (0.103)

The error is about an order of magnitude above the informal ((m−p)n)−1/2((m-p)n)^{-1/2} estimate of the submission version, which supports your concern; we remove that estimate, also from Appendix F.3. For scale, cc multiplies the magnitude of the non-stored updates, and in the sweep of Fig. 5 (c) (γ=1\gamma=1) PPL changes by less than 0.1 between c⋆=2.29c^\star=2.29 and c=3c=3 and rises by 0.41 at c=1.5c=1.5. A deviation of 2.5–4.5% in magnitude is therefore expected to have little effect on perplexity.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

(W2) Practical efficiency remains insufficiently validated. Pretraining stops at 1B, and the 7B experiment only trains LoRA adapters, leaving benefits under larger, more memory-constrained full-model training unclear.

In the pre-training setting, the 7B protocol requires 150K steps, for which GaLore used 64 A100 GPUs, and completing it is beyond our compute budget. We therefore evaluate up to 1B, as noted in the Limitation section. Across 60M–1B, COAST consistently outperforms Adam by 12–16% in perplexity (15.7/11.8/14.1/13.6%) with 0.54–0.63× its optimizer memory, and the gain shows no downward trend as the model grows.

For SFT setting, existing papers, including Fira, Adam-mini [5], and SGG [6], evaluate their optimizers on LoRA adapters in place of Adam. Since the adapters are kept identical across methods and only the optimizer is replaced, this setting isolates the effect of the optimizer and allows a direct comparison with prior methods. Nevertheless, to fully address the reviewer's concern, we additionally apply COAST to full-weight fine-tuning rather than to LoRA adapters. We fine-tune LLaMA-3.2-1B on the commonsense reasoning benchmark, strictly following the setting of APOLLO without any additional tuning for our method, so that the comparison is fair. As shown in the table below, COAST still outperforms the existing memory-efficient optimizers in this setting.

MethodsWGPIQASIQAOBQAHSBoolQARC-EARC-CAverage
AdamW68.1976.1272.3669.0069.1964.3472.2255.1268.07
LoRA67.5663.2871.6568.2019.1363.5867.3052.9959.21
DoRA68.9874.7072.4764.8063.9364.0169.3252.8266.38
GaLore62.7572.6368.1762.2047.8158.9968.9447.6161.14
Fira71.8277.2073.0869.0068.2164.3173.4054.7868.98
APOLLO w. SVD70.8877.6972.5270.6068.1963.0074.0355.7269.08
APOLLO70.4076.9372.7270.6063.7562.6973.4055.2068.21
APOLLO-Mini67.6476.5072.8869.6066.5464.2272.9855.4668.23
COAST$_r$69.8577.5374.3175.065.5066.6477.0260.5870.80
COAST$_{r/2}$71.0378.0773.1374.265.8466.2776.5662.5470.96

(W3) Timing comparisons cover only GaLore, which also requires SVD, and omit total runtime and time to matched loss.

We compared COAST against the baselines in the paper and the newly added APOLLO [1] (no SVD) on throughput (it/s), total training wall-clock time, and time to reach Adam's final loss.

Throughput (it/s)Total training wall-clock timeTime to reach Adam's final loss
Adam1.082:46:12Reference (10K steps)
GaLore1.092:44:03Not reached
Fira1.062:45:036k
Muon1.072:46:205k
APOLLO1.023:01:037k
COAST$_r$1.052:49:275k
COAST$_{r/2}$1.062:49:235k
COAST$_{r/4}$1.062:48:045k

(W4) Performance and timing comparisons with Adam/AdamW, Muon, and especially SVD-free APOLLO (Zhu et al., 2025) would better establish whether COAST’s rotation justifies its computational cost.

Due to the limited rebuttal period, we added Muon [2], Adafactor [3], FRUGAL [4], and APOLLO [1] at 60M–350M. For Muon, we use the values reported by LoRA-Pre under the same protocol; for APOLLO, the values reported in its paper. For FRUGAL (official code) and Adafactor, we tuned the learning rate over {1e−2,1e−3}\{1e-2, 1e-3\} and report the best result, whereas other memory-efficient baselines use a fixed learning rate of 1e-2. For Adam, for a fair comparison, we use the value shared by GaLore, Fira, and LoRA-Pre, which GaLore reports as the best over lr∈{0.01,0.005,0.001,0.0005,0.0001}lr \in \{0.01, 0.005, 0.001, 0.0005, 0.0001\}.

Method60M130M350M
Adam34.09 (0.36G)25.08 (0.76G)18.80 (2.06G)
Muon28.43 (0.26G)21.86 (0.54G)16.17 (1.43G)
GaLore34.88 (0.24G)25.36 (0.52G)18.95 (1.22G)
LoRA34.99 (0.36G)33.92 (0.80G)25.58 (1.76G)
Fira30.80 (0.24G)24.51 (0.52G)17.22 (1.22G)
LoRA-Pre (Adam)32.57 (0.24G)23.78 (0.56G)16.36 (1.31G)
LoRA-Pre (Muon)30.76 (0.23G)23.05 (0.47G)16.97 (1.07G)
FRUGAL36.45 (0.23G)26.37 (0.46G)22.03 (1.14G)
Adafactor32.21 (0.12G)23.80 (0.25G)17.01 (0.69G)
APOLLO w. SVD31.26 (0.24G)22.84 (0.52G)16.67 (1.22G)
APOLLO31.55 (0.24G)22.94 (0.52G)16.85 (1.22G)
APOLLO†31.26 (0.18G)23.18 (0.39G)16.98 (0.95G)
APOLLO-Mini31.93 (0.12G)23.53 (0.25G)17.18 (0.69G)
APOLLO-Mini‡30.95 (0.12G)22.85 (0.25G)16.63 (0.69G)
COAST$_r$28.74 (0.26G)22.11 (0.59G)16.14 (1.48G)
COAST$_{r/2}$28.95 (0.25G)22.14 (0.53G)16.15 (1.33G)
COAST$_{r/4}$29.09 (0.24G)22.23 (0.51G)16.19 (1.26G)
AuthorsOpenReview ↗

References

[1] Zhu et al., "Apollo: Sgd-like memory, adamw-level performance.",  in MLSys, 2025.

[2] Jordan et al., “Muon: An optimizer for hidden layers in neural networks”, https://kellerjordan.github.io/posts/muon, 2024.

[3] Shazeer, Noam, and Mitchell Stern. "Adafactor: Adaptive learning rates with sublinear memory cost." in ICML, 2018.

[4] Zmushko et al., "FRUGAL: Memory-efficient optimization by reducing state overhead for scalable training." in ICML, 2025.

[5] Zhang et al., "Adam-mini: Use fewer learning rates to gain more." in ICLR, 2025.

[6] Li et al., "Taming LLMs with Gradient Grouping." in ACL, 2025

Reviewer_uiZXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that GaLore-style projection ties two choices to one rank r: rotation coverage (how many singular directions are rotated and updated) and state storage (how many directions keep Adam states). COAST separates them. It rotates the gradient in the full left-singular basis but keeps Adam states for only r rows: a contiguous head plus a random tail sample. Each non-stored row is whitened by a second moment interpolated from its stored neighbors, and its norm is set by interpolated neighbor norms times a closed-form constant c = 1/√(1−β12) ≈ 2.29. A basis-reduction variant (Sec. 5) also stores only part of the basis and treats the remaining directions as one block with coefficient 1/(1+β1). Under the GaLore/Fira protocol (LLaMA 60M–1B on C4, LLaMA-7B commonsense fine-tuning), COAST outperforms Adam and all low-rank baselines at 0.42–0.63× Adam’s optimizer-state memory.

Critical strengths

The paper shows a clean conceptual split that yields a simple and effective method. Under the standard GaLore/Fira protocol, COAST beats every low-rank baseline and Adam at all four scales (e.g., 13.44 vs. 15.56 PPL at 1B). Its perplexity is nearly insensitive to the storage budget: under 1% change at r/2 and at most 1.2% at r/4 (tested on 60M–350M). The paper shows that the correction constant is derived in closed form rather than tuned. The derivation (Lemma 3.3, Corollary F.1) is correct, and the predicted ratios 0.436 and 0.229 match the measurements in Fig. 4(b) and 6(c). Fig. 5(c) shows that c* lies on a flat plateau, within 0.1 PPL of the sweep minimum.

Critical weaknesses

The gains over Adam are not attributed to the proposed mechanism. Figure 1 shows that full-rank rotation alone (full-rank GaLore) reaches essentially the same PPL on 60M, yet the memory-matched full-coverage alternatives (Muon, Adam-mini, Adafactor, FRUGAL, SOAP) are cited but not compared, and the Adam baseline’s learning rate is neither reported nor re-tuned. The practical memory benefit and the robustness of the results are not demonstrated. The 0.42–0.63× figures are analytical and appear to cover only the projected linear layers; measured peak memory at 130M is 20.64 GB for COASTr vs. 20.75 GB for Adam (about −0.5%), and pre-training stops at 1B. No seeds or error bars are reported, all baselines except Fira-60M are copied from prior papers, and in Table 2 the +1.1 average gain over Fira comes entirely from HellaSwag (+8.7), with the other seven tasks tied (538.3 vs. 538.6 summed).

Questions for authors

Can you add memory-matched comparisons with Muon, Adam-mini, Adafactor, FRUGAL (ideally also one-sided SOAP) and a learning-rate-tuned Adam? Please also ablate the storage itself: a very small budget (e.g., r = 1–8), and a single shared or rank-1 factored second moment in place of interpolation, together with the element-wise relative error of v̄i against the true V̂i. Is the “optimizer memory” ratio computed only over the projected layers, excluding the embedding and output layers that are trained with Adam? With 131K tokens per step, 10K/20K/60K steps give 1.31B/2.62B/7.86B tokens, not the 1.1B/2.2B/6.4B listed in Table 1. Which is correct? Also, Table 3 lists 24 heads for the 1B model, but 2048 is not divisible by 24; are heads and layers swapped?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We sincerely thank Reviewer uiZX for the meticulous reading of our paper, down to the derivations and table entries, and for the constructive suggestions, highlighting our strengths in:

  • It presents a clean conceptual split that yields a simple and effective method.
  • COAST outperforms Adam and every low-rank baseline at all four model scales.
  • Its perplexity is nearly insensitive to the storage budget, even at r/2 and r/4.
  • The closed-form correction constant is correctly derived and agrees with the measurements without tuning.

Below, we would like to give detailed responses to your questions.


(W1) The gains over Adam are not attributed to the proposed mechanism. Figure 1 shows that full-rank rotation alone reaches essentially the same PPL.

As the reviewer notes, the gain over Adam comes from full-rank rotation; this observation is the premise of our paper. Existing full-rank rotation methods show promising performance but store a basis and states for every direction, so their memory exceeds that of Adam (1.29× for full-rank GaLore). Lowering the rank reduces the coverage, which makes the result worse than Adam (GaLore 34.88 vs. Adam 34.09; Figure 1). No existing method therefore achieves the performance of full-rank rotation with at most the memory of Adam. Our contribution is the decoupling of coverage from storage: COAST keeps full coverage and reduces only the storage, staying within 0.1 PPL of full-rank rotation while using 0.42× the memory of Adam, about one-third of full-rank rotation.

(W2) yet the memory-matched full-coverage alternatives (Muon, Adam-mini, Adafactor, FRUGAL, SOAP) are cited but not compared, and the Adam baseline’s learning rate is neither reported nor re-tuned.

Due to the limited rebuttal period, we added Muon [1], Adafactor [2], FRUGAL [3], and APOLLO [4] at 60M–350M. For Muon, we use the values reported by LoRA-Pre under the same protocol; for APOLLO, the values reported in its paper. For FRUGAL (official code) and Adafactor, we tuned the learning rate over {1e−2,1e−3}\{1e-2, 1e-3\} and report the best result, whereas our baseline uses a fixed learning rate of 1e−31e-3. For Adam, for a fair comparison, we use the value shared by GaLore, Fira, and LoRA-Pre, which GaLore reports as the best over lr∈{0.01,0.005,0.001,0.0005,0.0001}lr \in \{0.01, 0.005, 0.001, 0.0005, 0.0001\}.

Method60M130M350M
Adam34.09 (0.36G)25.08 (0.76G)18.80 (2.06G)
Muon28.43 (0.26G)21.86 (0.54G)16.17 (1.43G)
GaLore34.88 (0.24G)25.36 (0.52G)18.95 (1.22G)
LoRA34.99 (0.36G)33.92 (0.80G)25.58 (1.76G)
Fira30.80 (0.24G)24.51 (0.52G)17.22 (1.22G)
LoRA-Pre (Adam)32.57 (0.24G)23.78 (0.56G)16.36 (1.31G)
LoRA-Pre (Muon)30.76 (0.23G)23.05 (0.47G)16.97 (1.07G)
FRUGAL36.45 (0.23G)26.37 (0.46G)22.03 (1.14G)
Adafactor32.21 (0.12G)23.80 (0.25G)17.01 (0.69G)
APOLLO w. SVD31.26 (0.24G)22.84 (0.52G)16.67 (1.22G)
APOLLO31.55 (0.24G)22.94 (0.52G)16.85 (1.22G)
APOLLO†31.26 (0.18G)23.18 (0.39G)16.98 (0.95G)
APOLLO-Mini31.93 (0.12G)23.53 (0.25G)17.18 (0.69G)
APOLLO-Mini‡30.95 (0.12G)22.85 (0.25G)16.63 (0.69G)
COAST$_r$28.74 (0.26G)22.11 (0.59G)16.14 (1.48G)
COAST$_{r/2}$28.95 (0.25G)22.14 (0.53G)16.15 (1.33G)
COAST$_{r/4}$29.09 (0.24G)22.23 (0.51G)16.19 (1.26G)

(W3) The practical memory benefit and the robustness of the results are not demonstrated. The 0.42–0.63× figures are analytical and appear to cover only the projected linear layers; measured peak memory at 130M is 20.64 GB for COASTr vs. 20.75 GB for Adam (about −0.5%) and pre-training stops at 1B.

The peak memory in Table 8 includes parameters, gradients, activations, and optimizer states, so the peak difference between methods that differ only in the optimizer is determined by the optimizer-state term. In this setting, COAST achieves peak savings on par with GaLore, a representative memory-efficient method. Because peak memory depends heavily on batch size, sequence length, and implementation, prior work, including GaLore, Fira, and LoRA-Pre, uses optimizer-state memory as the primary metric, and we follow this convention. Most importantly, COAST stays within 0.4% of GaLore’s peak memory while substantially lowering perplexity (25.36 → 22.11 at 130M), and it outperforms Adam with less memory.

The 7B protocol requires 150K steps, for which GaLore used 64 A100 GPUs, and completing it is beyond our compute budget. We therefore evaluate up to 1B, as noted in the Limitation section. Across 60M–1B, COAST consistently outperforms Adam by 12–16% in perplexity (15.7/11.8/14.1/13.6%) with 0.54–0.63× its optimizer memory, and the gain shows no downward trend as the model grows.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

(W4) No seeds or error bars are reported.

As in prior work [1,2,3,4], we used a single seed; following the reviewer's suggestion, we now report results over three seeds for the pre-training (60M–350M) and fine-tuning experiments.

  • pre-training
Method60M130M350M
COAST$_r$28.82$_{±0.07}$22.12$_{±0.03}$16.10$_{±0.03}$
COAST$_{r/4}$29.12$_{±0.03}$22.23$_{±0.03}$16.17$_{±0.03}$
  • fine-tuning
MethodMemoryBoolQPIQASIQAHellaSwagWinoGrandeARC-eARC-cOBQAAvg.
Fira14.34G69.482.678.076.881.282.264.480.876.9
COAST$_r$14.34G69.2$_{±0.9}$82.6$_{±0.4}$77.7$_{±0.9}$85.4$_{±0.3}$80.6$_{±0.1}$81.9$_{±0.4}$66.1$_{±0.6}$79.3$_{±1.0}$77.9$_{±0.1}$
COAST$_{r/2}$14.28G69.6$_{±0.2}$83.1$_{±0.4}$77.7$_{±0.7}$85.8$_{±0.4}$81.2$_{±0.4}$81.7$_{±0.1}$65.6$_{±0.3}$78.5$_{±0.2}$77.9$_{±0.1}$

(W5) All baselines except Fira-60M are copied from prior papers, and in Table 2 the +1.1 average gain over Fira comes entirely from HellaSwag, with the others tied.

To ensure a fair comparison, COAST strictly follows the common protocol of GaLore and its follow-up papers [4,5,6] without modification. We checked the reported numbers by reproducing Fira at 60M with its official code: the result, 30.80, is within 1% of the reported 31.06. This 0.26 PPL difference is about one-eighth of COAST’s margin over Fira (2.06 PPL), so it does not affect the comparison.

Fira's accuracy in Table 2 are those reported by its authors in their own setting, and COAST is run with the same hyper-parameters without re-tuning (Appendix I.2). We agree that the gain comes from HellaSwag. Since a single model is trained on all eight tasks, we read it as an improvement without a trade-off: the mean of the other seven tasks is unchanged (76.8 over three seeds vs. 76.9), while HellaSwag rises from 76.8 to 85.4${\pm 0.3}$ / 85.8${\pm 0.4}$.

(Q1) Can you add memory-matched comparisons with Muon, Adam-mini, Adafactor, FRUGAL (ideally also one-sided SOAP) and a learning-rate-tuned Adam?

Please refer to the response of W2.

(Q2) Please also ablate the storage itself:

Thanks for your suggestion. We first explain why the perplexity is maintained even with a small storage budget. Since Eq. 16 renormalizes each non-stored row to norm c ρic\,\rho_i, the absolute scale of vˉi\bar{v}_i cancels and only its column profile affects the whitening. The profile is similar across the entire spectrum, and the update norm is flat along the spectral index. Hence the whitening is nearly identical for any estimate—interpolated, shared, or factored. If anything, this ablation strengthens our claim that performance is determined by the coverage rather than the storage.

(Q2-1) A very small budget (e.g., r=1–8r = 1–8).

Method60M
GaLore34.88
COAST r=1r=131.81
COAST r=2r=231.83
COAST r=4r=429.75
COAST r=8r=829.50
COAST r=16r=1629.38
COAST r=32r=3229.09
COAST r=64r=6428.95

Note that in the r=1r=1 case, Eq. 11 gives rhead=⌈1/2⌉=1r_{\text{head}}=\lceil 1/2\rceil=1 and rtail=0r_{\text{tail}}=0, so only direction 11 is stored and bi=min⁡∅=∞b_i=\min\emptyset=\infty for every non-stored ii. Eq. 14 then yields λi=0\lambda_i=0, and Eqs. 15–16 apply unchanged with vˉi=V^1\bar v_i=\hat V_1 and ρi=∥N1∥2\rho_i=\|N_1\|_2: every non-stored direction takes its second moment and update norm from direction 11. Even when the storage budget is reduced all the way to r=1r=1, COAST never exceeds 31.83 PPL, still 3.05 lower than GaLore at r=128r=128.

(Q2-2) A single shared or rank-1 factored second moment in place of interpolation.

60M, r=128r=12860M, r/4=32r/4=32130M, r=256r=256130M, r/4=64r/4=64
COAST28.7429.0922.1122.23
Single shared vector v^B\hat{v}^{\mathcal{B}}29.0529.1022.0922.21
Rank-1 factored28.6828.8921.8921.92

Our main contribution is to decouple rotation coverage from optimizer-state storage. Interpolation is one realization of this principle; the shared and factored variants show that substantial state compression can retain competitive performance.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

(Q2-3) Together with the element-wise relative error of vˉi\bar{v}_i against the true V^i\hat{V}_i.

On a LLaMA-60M run that keeps the states of all directions (r=m=512r = m = 512), we draw the stored set of Eq. 11 for the storage budget r=128r = 128, build vˉi\bar{v}_i for every non-stored direction by Eq. 14–15, and compare it with the true V^i\hat{V}_i.

Cos simRelative errorRelative error with rescaling
0.9860.250.15

Rescaled is the same error after multiplying vˉi\bar{v}_i by a single scalar per direction so that its scale matches V^i\hat{V}_i. The element-wise relative error is 0.25, yet the cosine similarity is as high as 0.986. Since vˉi\bar{v}_i is a combination of two such neighbors, vˉi\bar{v}_i and V^i\hat{V}_i point in nearly the same direction and differ mostly in overall scale; indeed, matching the scale of vˉi\bar{v}_i to that of V^i\hat{V}_i reduces the error to 0.15. COAST sets the size of the update not by vˉi\bar{v}_i but by cρic\rho_i in Eq. 16, and uses vˉi\bar{v}_i only to whiten the direction through the ratios between its elements, so this scale difference does not enter the update.

(Q3) Is the “optimizer memory” ratio computed only over the projected layers, excluding the embedding and output layers that are trained with Adam?

The reviewer is correct. Strictly following GaLore and Fira, the ratio counts only the optimizer memory of the attention and MLP linear layers. The embedding and output layers are excluded since they are trained with Adam.

(Q4) With 131K tokens per step, 10K/20K/60K steps give 1.31B/2.62B/7.86B tokens, not the 1.1B/2.2B/6.4B listed in Table 1. Which is correct?

Thank you for correction. All experiments were run with a batch size of 131K tokens for 10K / 20K / 60K / 100K steps, as specified in Appendix I (Table 3), which corresponds to 1.31B / 2.62B / 7.86B / 13.1B training tokens. Since the training budget is controlled by the same number of steps and batch size as the baselines including GaLore, the results and comparisons are unaffected. To avoid confusion, we have updated the row with the values above in the revision.

(Q5) Table 3 lists 24 heads for the 1B model, but 2048 is not divisible by 24; are heads and layers swapped?

Thank you for kindly pointing this out. This is a typo: the 1B model has 32 heads and 24 layers. We will correct Table 3 in revised paper.

References

[1] Jordan et al., “Muon: An optimizer for hidden layers in neural networks”, https://kellerjordan.github.io/posts/muon, 2024.

[2] Shazeer, Noam, and Mitchell Stern. "Adafactor: Adaptive learning rates with sublinear memory cost." in ICML, 2018.

[3] Zmushko et al., "FRUGAL: Memory-efficient optimization by reducing state overhead for scalable training." in ICML, 2025.

[4] Zhu et al., "Apollo: Sgd-like memory, adamw-level performance.",  in MLSys, 2025.

[5] Chen et al., "Fira: Can we achieve full-rank training of llms under low-rank constraint?." in NeurIPS, 2025.

[6] Wang et al., "Taming momentum: Rethinking optimizer states through low-rank approximation." in ICLR, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Updates

Summary of Rebuttal and Updates

We thank all reviewers for their constructive feedback. Below we summarize the strengths recognized by the reviewers and how we addressed each concern.

Strengths Recognized by Reviewers

  1. The novelty of decoupling rotation coverage from state storage (Reviewers uiZX, KzrC, mdGu, cKHB)
  2. Consistent improvements across all model scales with extensive experiments (Reviewers uiZX, KzrC, mdGu, cKHB)
  3. Robustness to the storage budget (Reviewers uiZX, KzrC, cKHB)
  4. The principled design, including a closed-form correction constant that matches measurements without tuning (Reviewers uiZX, mdGu, cKHB)
  5. The clear presentation (Reviewers uiZX, mdGu)

Addressing Reviewers' Concerns

We addressed every concern with new experiments, theory, or clarification. The new results consistently reinforce our original conclusions.

  1. Source of gains and additional baselines (Reviewers uiZX, KzrC, mdGu)

    • Full-rank rotation is our premise. Existing full-rank methods need 1.29× the memory of Adam, while COAST keeps full coverage at only 0.42×.
    • We added Muon, Adafactor, FRUGAL, and APOLLO with its SVD-free and Mini variants. COAST outperforms all of them except Muon, which it matches at comparable memory. COAST also beats LoRA-Pre at every scale with less state memory, and surpasses a tuned Muon in fine-tuning.
  2. Scalability to 7B pre-training (Reviewers uiZX, KzrC)

    • On LLaMA-7B, COAST reaches 11.95 PPL versus 14.61 for 8-bit Adam and 14.65 for 8-bit GaLore, a gap of 2.7 PPL. It does so with 22% less memory than 8-bit Adam while storing states for only about 3% of directions.
    • COAST leads at every checkpoint and passes both baselines' final PPL within roughly a third of the training budget. The basis-reduction variant also remains effective at 130M and 350M (Reviewer mdGu).
  3. Full-weight fine-tuning and SFT setting (Reviewers KzrC, mdGu)

    • On full-weight fine-tuning of LLaMA-3.2-1B, COAST reaches about 71 average accuracy, outperforming AdamW at 68.1 and the best memory-efficient baseline at 69.1.
    • In the LoRA setting, which isolates the optimizer effect, COAST cuts the optimizer state to a quarter while gaining 3.1 points. We also replaced the under-tuned full fine-tuning baseline with a properly tuned one, and our conclusions hold.
  4. Theoretical justification of the block correction (Reviewer KzrC)

    • Proposition H.2 now gives explicit conditions and an error bound for Eq. (31). The measured relative error is only 0.025 on average, with negligible effect on perplexity, confirming the correction is accurate in practice. The algorithm is unchanged.
  5. Robustness and ablations (Reviewers uiZX, cKHB)

    • Three-seed results for pre-training and fine-tuning confirm that the gains are stable, and our reproduction of Fira with its official code closely matches its reported result.
    • With only a single stored direction, COAST is still 3.05 PPL below GaLore at 60M, and shared or rank-1 factored second moments perform comparably. This directly confirms that coverage, not storage, drives performance.
  6. Memory and runtime (Reviewers uiZX, KzrC, cKHB)

    • We clarified the memory accounting and added timing comparisons. COAST's wall-clock time is within 2% of Adam's, and it reaches Adam's final loss in half the steps, faster than Fira and APOLLO.

Conclusion

With these additions, we believe all major concerns have been resolved, including those of Reviewers KzrC and mdGu. The new results strengthen our main claim: COAST achieves full-rank rotation coverage at a fraction of Adam's memory, matches or outperforms all baselines from 60M to 7B in both pre-training and fine-tuning, and runs at Adam's speed. We thank the reviewers and ACs for their time.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.