- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces an optimizer, coined Clean, that reduces SOAP’s memory cost while preserving much of its training performance.
It uses Nyström sketching to estimate the dominant directions of gradient covariance without storing large covariance matrices.
Clean keeps detailed Adam statistics in the retained subspace and uses simpler statistics for the remaining components, allowing them to contribute to the update.
Experiments on 350M and 1.3B language models show competitive perplexity with lower optimizer-state memory, while the quantized variant, Q-Clean, reduces memory further.
Critical strengths
1, Clean makes selective use of optimizer memory. it stores detailed momentum and second-moment statistics for the dominant gradient subspace, while using cheaper statistics for the residual components. This reduces memory without discarding their gradient contributions.
2, Nyström sketching estimates the dominant covariance directions without constructing the full covariance matrices, making SOAP-style optimization more memory-efficient.
3, Experiments on 350M and 1.3B language models trained on C4 demonstrate a useful performance–memory trade-off. At 1.3B, Clean reaches AdamW’s final validation loss in 64 hours instead of 81, reducing the training time needed to reach that loss by approximately 21%.
Critical weaknesses
1, Clean introduces sketching, decomposition, gradient projection, and moment transport. Although Figure 5 reports overall wall-clock time, it does not show how much time these operations take. I would like to see the costs of ordinary steps and refresh steps reported separately and how they change with rank and layer size.
2, The memory allocation across gradient components needs stronger justification. The method retains a fixed number of covariance directions and assigns different moment statistics to the four components. The rank ablation is useful, but it does not explain whether the same allocation is suitable across layers and training stages. More direct ablations are needed to justify pooling the second moments in A and B and removing momentum from C.
3, The training-quality experiments cover two model sizes on C4 with a sequence length of 1024. The 13B result demonstrates memory feasibility but does not establish convergence or training quality at that scale. Experiments at a larger scale or with longer contexts would make the claimed practical benefits more convincing.
Questions for authors
-
How much computational overhead does Clean introduce? Figure 5 reports the overall time to reach AdamW’s final loss. From the reported numbers, I estimate approximately 59.9 seconds per step for Clean and 58.3 seconds for AdamW, although these averages include other training overhead. Could the authors separately report the costs of sketching, decomposition, projection, and moment transport, including ordinary steps and refresh steps?
-
How should the retained rank and the statistics for each component be chosen? The method uses fixed ranks rather than a retained-energy threshold. Could the authors explain whether these ranks capture a similar fraction of gradient energy across layers and training stages? I would also like to see direct ablations supporting the pooled second moments in A and B and the removal of momentum from C.
-
How should the linear-memory claim be interpreted when rank increases with model width? The linear scaling holds when rank is fixed. However, Appendix D also considers rank proportional to width, which makes the state-memory formula quadratic in width. Could the authors clarify this distinction and explain the decreasing memory-per-parameter trend in Figure 7 under that setting?
-
Are the memory measurements directly comparable across methods and experiments? Could the authors specify the precision of each optimizer state and whether peak memory includes temporary decomposition and projection buffers? At 1.3B, Clean reduces optimizer-state memory by approximately 12%, but total peak memory by only 1.3% relative to AdamW. It would help to explain when the state savings translate into a substantial practical benefit, and whether the 13B result measures memory feasibility alone or also includes sustained training.
5 comments
Q3: The key distinction is that Clean’s optimizer memory has no or terms, whereas all baselines have at least one (Table 2). Thus, Clean is the only optimizer that fits a 13B model on one H100 GPU (Fig 1). We revised the manuscript to clarify that Clean scales linearly with model dimensions when the r is fixed (lines 84–86, 322–324, 882–886), reinforcing what was already included in line 96 of the original submission. In practice, the chosen rank often increases with model dimensions. To reflect this realistic setting, Fig 7 uses , rather than assuming a fixed r.
Regarding the decreasing trend in Fig. 7: Clean and all baselines keep plain Adam state on the embedding and output layers, and differ only on the attention and MLP blocks, where our state costs 0.83 bytes per parameter against AdamW's 4.0. Our per-parameter report is a weighted average of these two fixed costs, and declines toward our block cost as the weight shifts. AdamW is flat because both of its components cost 4.0, and SOAP, Muon and ALSO move only slightly because the embedding term is a far smaller fraction of their totals (line 892).
Q4: Table 3 reports each method in its native precision, so its optimizer-memory column is not directly comparable across methods. AdamW, Muon and ALSO store BF16 states. SOAP stores its Adam moments in BF16 and L, R and their eigenbases in FP32. Clean stores its states in FP32; with BF16 states, Clean’s optimizer memory will be 0.75G at 350M and 2.49G around 46% and 51% below AdamW respectively.
Nevertheless, to ensure a fair comparison of memory consumption, Fig. 1 reports usage with BF16 optimizer states for all methods, regardless of their original implementations.
Also, the peak memory is reported using torch.cuda.max_memory_allocated, accounting for all the parameters during training. Memory savings are less pronounced for smaller models and larger batch sizes or sequence lengths, as activations and other components dominate memory usage. Clean's primary advantage becomes significant at scales where memory constraints may prevent training even with a batch size of 1(Fig 1).
[1] icml.cc/virtual/2026/75854
[2] arXiv:1804.04235
[3] arXiv:2406.16793
Dear Reviewer,
With the discussion period nearing its conclusion, we wanted to follow up and kindly ask whether you have had an opportunity to consider our rebuttal. We would greatly appreciate knowing if any aspects of our response require further explanation or if there are additional concerns we could address. We sincerely value the time and effort you have devoted to evaluating our work. Your feedback has provided us with an important opportunity to reconsider and strengthen our submission. We have carefully considered each of your comments and made every effort to respond comprehensively to the points raised. We would welcome any further discussion that could help resolve outstanding concerns and improve the clarity and quality of our work. Thank you again for your thoughtful feedback and consideration. We would like to kindly ask the reviewer to reconsider their score if their concerns have been resolved.
Best regards,
We thank the reviewer for their constructive feedback. We provide answers to the questions below.
CW1,Q1: The gap inferred from Fig 5 is almost entirely related to the subspace‑refresh step, performed every T=10; ordinary steps add negligible cost. We report per‑step wall‑clock time for LLaMA‑1.3B, with computational complexity.
rank r, oversampling p, power iters q:
Wall‑clock, eliminating evaluation steps:
Clean's ordinary step is O(mnr) vs SOAP's O(m2n+mn2), and its refresh is O(mn(r+p)+(m+n)(r+p)2) vs SOAP's $O(m^3 + n^3). Please note that SOAP's few full-rank eigendecompositions leverage highly optimized GPU kernels, whereas Clean performs several small operations that might underutilize the GPU. This is a kernel-efficiency issue, not a scaling limitation, and will benefit Clean in larger models. Also, Clean's lower memory footprint enables larger batches and further wall-time savings; however the batch sizes are the same in our experiments for fair comparison. We added the results in the revision (lines 476, Appendix E).
CW2 and Q2-dynamic rank: We agree that adaptive rank selection is an interesting direction. However, since the primary objective of this paper is to reduce memory overhead while maintaining SOAP-like performance, we adopt a fixed-rank approach to ensure predictable and consistent memory usage. Thus, a more systematic investigation of adaptive rank selection and its potential benefits is very useful, which we leave for future work (line 497).
CW2 and Q2-pooled moment ablations: Pooling moments have been used previously in the literature [1-3], and we provide an ablation study to validate our pooling strategies. The ambient variant retains full first-moment estimates for block C, requiring O(mn) memory. The final variant retains full second-moments for blocks A and B, increasing their memory from O(2r) to O(mr+nr). As expected, ambient performs slightly better. We also expected the full second-moment variant to perform better than Clean, however, this result might be due to some regularization effect caused by pooling and needs further investigation. Nevertheless, both approaches achieve comparable perplexity and demonstrate that our pooling strategies substantially reduce memory overhead with negligible impact on perplexity. We have included this ablation in the revised manuscript (Table 11).
CW3: Second-order optimizers at scale are computationally expensive. For context, SOAP evaluates models up to 660M parameters, and ALSO[1] only goes up to 130M. Clean scales to 1.3B parameters, and to further demonstrate scalability, we evaluate against AdamW on a 7B-parameter model for 2k iterations (~3days on 4 H100 GPUs).
SOAP at 7B exceeds our computational budget. However, Clean outperforms AdamW with less than half its optimizer-state memory, highlighting the effectiveness of its second-order optimization.
Although none of our baselines fit within our H100 GPUs, we evaluate Clean's convergence on a 13B-parameter model over 3 days using 4 GPUs, as evidence of its scalability and convergence.
These results are included in the revision (Appendix E).