acceptodds
Under review as a conference paper at ICLR 2027

Clean: Second-order LLM Training at Linear Memory Cost via Nyström Sketching

Abstract

Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature, whereas full-curvature methods such as SOAP can accelerate convergence at prohibitive memory costs. We introduce **Clean**, a memory-efficient and full-curvature optimizer designed to resolve this bottleneck. Clean leverages the randomized Nyström method to accurately approximate the left and right preconditioners in SOAP, and to reduce the optimizer's memory complexity from quadratic to linear in terms of model dimension. We subsequently reintegrate the off-subspace components to capture curvature information beyond the low-rank approximation, preserving rich curvature at minimal memory cost. We further propose **Q-Clean**, a low-precision variant that aggressively compresses optimizer states. Q-Clean reduces optimizer memory consumption by **over 50%** compared to Muon when pre-training a LLaMA-1.3B architecture, all while maintaining strong and competitive predictive performance. Notably, Clean operates with a smaller optimizer-state footprint than standard AdamW while reaching AdamW's final performance **26% faster** in wall-clock time. Furthermore, our methods uniquely enable the pre-training of a 13B-parameter model on a single 80GB GPU, providing a scalable, efficient, and accessible approach to large-scale model optimization.

Primary area:
optimization
Keywords:
memory-efficient optimization, low-rank gradient, second-order information, subspace tracking, Nystrom sketching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eEMsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an optimizer, coined Clean, that reduces SOAP’s memory cost while preserving much of its training performance.

It uses Nyström sketching to estimate the dominant directions of gradient covariance without storing large covariance matrices.

Clean keeps detailed Adam statistics in the retained subspace and uses simpler statistics for the remaining components, allowing them to contribute to the update.

Experiments on 350M and 1.3B language models show competitive perplexity with lower optimizer-state memory, while the quantized variant, Q-Clean, reduces memory further.

Critical strengths

1, Clean makes selective use of optimizer memory. it stores detailed momentum and second-moment statistics for the dominant gradient subspace, while using cheaper statistics for the residual components. This reduces memory without discarding their gradient contributions.

2, Nyström sketching estimates the dominant covariance directions without constructing the full covariance matrices, making SOAP-style optimization more memory-efficient.

3, Experiments on 350M and 1.3B language models trained on C4 demonstrate a useful performance–memory trade-off. At 1.3B, Clean reaches AdamW’s final validation loss in 64 hours instead of 81, reducing the training time needed to reach that loss by approximately 21%.

Critical weaknesses

1, Clean introduces sketching, decomposition, gradient projection, and moment transport. Although Figure 5 reports overall wall-clock time, it does not show how much time these operations take. I would like to see the costs of ordinary steps and refresh steps reported separately and how they change with rank and layer size.

2, The memory allocation across gradient components needs stronger justification. The method retains a fixed number of covariance directions and assigns different moment statistics to the four components. The rank ablation is useful, but it does not explain whether the same allocation is suitable across layers and training stages. More direct ablations are needed to justify pooling the second moments in A and B and removing momentum from C.

3, The training-quality experiments cover two model sizes on C4 with a sequence length of 1024. The 13B result demonstrates memory feasibility but does not establish convergence or training quality at that scale. Experiments at a larger scale or with longer contexts would make the claimed practical benefits more convincing.

Questions for authors
  1. How much computational overhead does Clean introduce? Figure 5 reports the overall time to reach AdamW’s final loss. From the reported numbers, I estimate approximately 59.9 seconds per step for Clean and 58.3 seconds for AdamW, although these averages include other training overhead. Could the authors separately report the costs of sketching, decomposition, projection, and moment transport, including ordinary steps and refresh steps?

  2. How should the retained rank and the statistics for each component be chosen? The method uses fixed ranks rather than a retained-energy threshold. Could the authors explain whether these ranks capture a similar fraction of gradient energy across layers and training stages? I would also like to see direct ablations supporting the pooled second moments in A and B and the removal of momentum from C.

  3. How should the linear-memory claim be interpreted when rank increases with model width? The linear scaling holds when rank is fixed. However, Appendix D also considers rank proportional to width, which makes the state-memory formula quadratic in width. Could the authors clarify this distinction and explain the decreasing memory-per-parameter trend in Figure 7 under that setting?

  4. Are the memory measurements directly comparable across methods and experiments? Could the authors specify the precision of each optimizer state and whether peak memory includes temporary decomposition and projection buffers? At 1.3B, Clean reduces optimizer-state memory by approximately 12%, but total peak memory by only 1.3% relative to AdamW. It would help to explain when the state savings translate into a substantial practical benefit, and whether the 13B result measures memory feasibility alone or also includes sustained training.

5 comments

We thank the reviewer for their constructive feedback. We provide answers to the questions below.

CW1,Q1: The gap inferred from Fig 5 is almost entirely related to the subspace‑refresh step, performed every T=10; ordinary steps add negligible cost. We report per‑step wall‑clock time for LLaMA‑1.3B, with computational complexity.

rank rr, oversampling pp, power iters qq:

StageWhenCleanSOAP
Proj. (Eq. 10–11)every stepO(mnr)O(m²n+mn²)
Moment updates (Eq. 12–13)every stepO(mn)O(mn)
Proj. back (Eq. 16)every stepO(mnr)O(m²n+mn²)
Sketching (Eq. 8 / Alg. 3)refreshO((q+1)mn(r+p))—
DecompositionrefreshO((m+n)(r+p)²+(r+p)³)O(m³+n³)
Moment transport (Eq. 14–15)refreshO((m+n)r²)—

Wall‑clock, eliminating evaluation steps:

Ordinary stepRefreshRefresh add-onAmortized
AdamW57.5 s——57.5 s
SOAP57.5 s60.8 s+3.3 s57.8 s
Clean57.8 s75.2 s+17.4 s59.5 s

Clean's ordinary step is O(mnr)O(mnr) vs SOAP's O(m2n+mn2)O(m^2n + mn^2), and its refresh is O(mn(r+p)+(m+n)(r+p)2)O(mn(r+p)+(m+n)(r+p)^2) vs SOAP's $O(m^3 + n^3). Please note that SOAP's few full-rank eigendecompositions leverage highly optimized GPU kernels, whereas Clean performs several small operations that might underutilize the GPU. This is a kernel-efficiency issue, not a scaling limitation, and will benefit Clean in larger models. Also, Clean's lower memory footprint enables larger batches and further wall-time savings; however the batch sizes are the same in our experiments for fair comparison. We added the results in the revision (lines 476, Appendix E).

CW2 and Q2-dynamic rank: We agree that adaptive rank selection is an interesting direction. However, since the primary objective of this paper is to reduce memory overhead while maintaining SOAP-like performance, we adopt a fixed-rank approach to ensure predictable and consistent memory usage. Thus, a more systematic investigation of adaptive rank selection and its potential benefits is very useful, which we leave for future work (line 497).

CW2 and Q2-pooled moment ablations: Pooling moments have been used previously in the literature [1-3], and we provide an ablation study to validate our pooling strategies. The ambient variant retains full first-moment estimates for block C, requiring O(mn)O(mn) memory. The final variant retains full second-moments for blocks A and B, increasing their memory from O(2r)O(2r) to O(mr+nr)O(mr+nr). As expected, ambient performs slightly better. We also expected the full second-moment variant to perform better than Clean, however, this result might be due to some regularization effect caused by pooling and needs further investigation. Nevertheless, both approaches achieve comparable perplexity and demonstrate that our pooling strategies substantially reduce memory overhead with negligible impact on perplexity. We have included this ablation in the revised manuscript (Table 11).

Val PPL
Clean [Ours]15.87
Clean-ambient15.68
Clean-full second-moments for A and B16.12

CW3: Second-order optimizers at scale are computationally expensive. For context, SOAP evaluates models up to 660M parameters, and ALSO[1] only goes up to 130M. Clean scales to 1.3B parameters, and to further demonstrate scalability, we evaluate against AdamW on a 7B-parameter model for 2k iterations (~3days on 4 H100 GPUs).

Val PPLOpt Mem.
AdamW17.0925.71G
Clean (r = 1024)14.1911.66G

SOAP at 7B exceeds our computational budget. However, Clean outperforms AdamW with less than half its optimizer-state memory, highlighting the effectiveness of its second-order optimization.

Although none of our baselines fit within our H100 GPUs, we evaluate Clean's convergence on a 13B-parameter model over 3 days using 4 GPUs, as evidence of its scalability and convergence.

Step 50Step 300Step 600Step 900
Val PPL2356.2866.0126.9822.59

These results are included in the revision (Appendix E).

Q3: The key distinction is that Clean’s optimizer memory has no mn,m2mn, m^2 or n2n^2 terms, whereas all baselines have at least one mnmn (Table 2). Thus, Clean is the only optimizer that fits a 13B model on one H100 GPU (Fig 1). We revised the manuscript to clarify that Clean scales linearly with model dimensions when the r is fixed (lines 84–86, 322–324, 882–886), reinforcing what was already included in line 96 of the original submission. In practice, the chosen rank often increases with model dimensions. To reflect this realistic setting, Fig 7 uses r≈d/8r\approx d/8, rather than assuming a fixed r.

Regarding the decreasing trend in Fig. 7: Clean and all baselines keep plain Adam state on the embedding and output layers, and differ only on the attention and MLP blocks, where our state costs 0.83 bytes per parameter against AdamW's 4.0. Our per-parameter report is a weighted average of these two fixed costs, and declines toward our block cost as the weight shifts. AdamW is flat because both of its components cost 4.0, and SOAP, Muon and ALSO move only slightly because the embedding term is a far smaller fraction of their totals (line 892).

Q4: Table 3 reports each method in its native precision, so its optimizer-memory column is not directly comparable across methods. AdamW, Muon and ALSO store BF16 states. SOAP stores its Adam moments in BF16 and L, R and their eigenbases in FP32. Clean stores its states in FP32; with BF16 states, Clean’s optimizer memory will be 0.75G at 350M and 2.49G around 46% and 51% below AdamW respectively.

Nevertheless, to ensure a fair comparison of memory consumption, Fig. 1 reports usage with BF16 optimizer states for all methods, regardless of their original implementations.

Also, the peak memory is reported using torch.cuda.max_memory_allocated, accounting for all the parameters during training. Memory savings are less pronounced for smaller models and larger batch sizes or sequence lengths, as activations and other components dominate memory usage. Clean's primary advantage becomes significant at scales where memory constraints may prevent training even with a batch size of 1(Fig 1).

[1] icml.cc/virtual/2026/75854

[2] arXiv:1804.04235

[3] arXiv:2406.16793

Dear Reviewer,

With the discussion period nearing its conclusion, we wanted to follow up and kindly ask whether you have had an opportunity to consider our rebuttal. We would greatly appreciate knowing if any aspects of our response require further explanation or if there are additional concerns we could address. We sincerely value the time and effort you have devoted to evaluating our work. Your feedback has provided us with an important opportunity to reconsider and strengthen our submission. We have carefully considered each of your comments and made every effort to respond comprehensively to the points raised. We would welcome any further discussion that could help resolve outstanding concerns and improve the clarity and quality of our work. Thank you again for your thoughtful feedback and consideration. We would like to kindly ask the reviewer to reconsider their score if their concerns have been resolved.

Best regards,

Reviewer_bLpSOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Clean, a memory-efficient second-order optimizer for LLM training. The method uses randomized Nystrom approximations to maintain compact representations of the left and right gradient covariance matrices, avoiding the quadratic memory overhead of optimizers such as SOAP. To preserve information outside the low-rank subspace, Clean decomposes the gradient into four blocks and applies Adam-style updates with different levels of moment compression. The paper further introduces Q-Clean, which uses 8-bit quantization to reduce optimizer-state memory. The method is evaluated on LLaMA-based pretraining at 350M and 1.3B parameters, with additional memory measurements for a 13B model.

Critical strengths
  1. The empirical results are promising. On LLaMA-1.3B, Clean achieves perplexity close to SOAP (11.65 versus 11.52) while using substantially less optimizer memory (4.48 GB versus 13.62 GB). The paper also reports wall-clock convergence improvements over AdamW. Q-Clean further reduces optimizer memory to 1.28 GB while maintaining performance close to AdamW.
  2. Nystrom approximation is a promising idea for compressing LtL_t and RtR_t. The reviewer found this idea to compress the Kronecker factors to be novel.
Critical weaknesses
  1. While reviewing the paper, the reviewer found several notation inconsistencies:
  • In Section 2, there is a logical inconsistency in what exactly LtL_t and RtR_t are approximating. The reviewer understands that all the statements are correct, however the background section will benefit from accurate claims regarding why the matrix power is used and why it is 1/41/4 initially.
  • In (7), before stating that r1=r2=rr_1 = r_2 = r authors should clearly define what is r_1 and r_2. It seems like Ut,L∈Rm×r1,Ut,R∈Rn×r2U_{t, L} \in \mathbb{R}^{m \times r_1}, U_{t, R} \in \mathbb{R}^{n \times r_2}, but it needs to be written explicitly.
  1. Authors does not provide a valid comparison between their quantized optimizer version and other quantized optimizers, which makes the evalution of the proposed quantized optimizer difficult.
  2. The choice second moment estimates for blocks is not well motivated by the task structure. The motivation would be much more solid if the choice of second moment estimates for all blocks was guided by the solution of some approximation task.
Questions for authors
  1. How do authors explain the imbalance of BB and CC blocks gradient norms across all layers. At the same time, for 3e, 3f and 3g the norm of Main does not dominate the norm of the A block, which poses a question whether the approximation found by the method is accurate.
  2. Did authors tried to maintain true rank-1 approximation of the second moment for blocks AA and BB? Current choice of the shared scale for AA and BB is also rank-1, but a particular one, with fixed row for AA and column for BB. Is it true that in the case of the unconstrained rank-1 approximationn of the second moment there is no analogue to Eq.(15)? If so, authors should explain this analysis in the revised version.
3 comments

We thank the reviewer for their constructive feedback and for finding our Nystrom approximation novel. We address their questions below.

CW1-a Based on the reviewer’s feedback, we improved the background section(line 125) in the revision as follows:

Full-matrix AdaGrad[1] maintains the preconditioner Pt=∑gtgt⊤∈Rmn×mnP_t = \sum g_t g_t^\top\in \mathbb{R}^{mn\times mn}, where gt∈Rmng_t\in \mathbb{R}^{mn} is the vectorized gradient, and updates the weights as wt+1=wt−ηPt−1/2gtw_{t+1} = w_t - \eta P_t^{-1/2}g_t. Shampoo[2] approximates this full-matrix preconditioning using two smaller covariance matrices, LtL_t and RtR_t, as defined in Eq. 6. Shampoo originally uses the exponent 1/4, corresponding to the approximation Pt−1/2≈Lt−1/4⊗Rt−1/4P_t^{-1/2}\approx L_t^{-1/4}\otimes R_t^{-1/4}. Anil et al.[3] empirically found that using 1/2 instead of 1/4 improves performance. Building on this, Morwani et al.[4] provide a theoretical perspective on Shampoo's preconditioning, offering further motivation for using 1/2 rather than the original 1/4. Thus, Clean adopts the 1/2 for its low-rank preconditioning factors.

CW1-b: We appreciate the reviewer’s input and incorporated these details in the revised version(line 196).

CW2: We added Adam-8bit[5] and Q-GaLore[6]. Q-Clean achieves better validation perplexity while having lower optimizer memory than Adam-8bit and similar to Q-GaLore. We also emphasize that quantization is not the primary contribution of this work; rather, we aim to show that Clean remains effective under quantization.

Val PPLOpt Mem.
Q-Clean16.440.38
Adam-8bit19.640.71
Q-GaLore18.620.32

These results are included in the revision for 350M model(Table 6), and we will add 1B results in the camera-ready version.

CW3 and Q2: All four choices in Table 1 follow from a single approximation problem: Let (VX=Eβ2[GX⊙2])(V_X=\mathbb{E}_{\beta_2}[G_X^{\odot 2}]) denote the entrywise second moment of block XX. Our estimates are precisely the solutions to Frobenius-norm approximation problems under block-specific structural constraints:

&\text{Main:}\widehat V_{\text{Main}}=V_{\text{Main}},\\ &\text{A:} \widehat V_A=\mathbf{1}_m a^\top/(m-r),\\ &\text{B:} \widehat V_B=b\mathbf{1}_n^\top/(n-r),\\ &\text{C:}\widehat V_C=c\mathbf{1}_m\mathbf{1}_n^\top/((m-r)(n-r)). \end{align}$$ Minimizing $(|V_X-\widehat V_X|_F^2)$ gives \$ (a=V_A^\top\mathbf{1}_m), (b=V_B\mathbf{1}_n)\$, and $(c=\mathbf{1}_m^\top V_C\mathbf{1}_n)$, exactly recovering our pooled second moments, up to the normalization convention in Table 1. This structure is motivated by the Nyström decomposition[7-8], which rescales each tracked eigen direction individually and acts as the identity on the complements. We apply pooling to these complementary dimensions. Since the top eigen directions are captured by the main block, the complementary dimensions consist of directions associated with smaller eigenvalues. Regarding Eq. (15), consider a more general rank-1 approximation \$\widehat V_A=\rho a^\top/(m-r) \$, where $\rho\in\mathbb{R}^m$ is a free factor along the complement dimension (whereas our method fixes $\rho=\mathbf {1} _ m$). When $U_{t,R}$ rotates, the columns of $G_{t,A}$ are mixed, but their row norms remain unchanged. Thus, a free $\rho$ can be preserved using Eq. (15). However, when $U_{t,L}$ is refreshed, the complement subspace changes. Thus, the energy distribution across the m coordinates of $\rho$ also changes. However, neither approach permits exact transport of entrywise second moments under arbitrary basis rotations without the discarded cross-coordinate correlations, making Eq. (15) an approximation, as it is already stated in the paper (line 291). We have added this discussion in the revision (line 1242). **Q1:** The imbalance between A and B arises from the different alignment of the gradient with the left and right principal subspaces. Thus, the larger contribution of A indicates that the retained right subspace captures more of the gradient energy than the left subspace. This asymmetry is possible because the two subspaces are estimated from different exponentially averaged covariance matrices, which need not have the same spectral concentration. In addition, the Nyström approximation has strong and provable guarantees[7-8], and a large contribution from A does not imply an inaccurate approximation. Instead, it highlights the importance of retaining off-subspace information, which is a central motivation for Clean. We clarified these points in the revised version (255 and Appendix J). We thank the reviewer for the constructive feedback. We addressed their concerns with additional results and clarifications. We welcome further discussion and kindly ask the reviewer to reconsider their score if their concerns have been resolved. [1] jmlr.org/papers/v12/duchi11a.html [2] arXiv:1802.09568 [3] arXiv:2002.09018 [4] arXiv:2406.17748 [5] arXiv:2110.02861 [6] arXiv:2407.08296 [7] arXiv:2110.02820 [8] arXiv:1303.1849

Dear Reviewer,

With the discussion period nearing its conclusion, we wanted to follow up and kindly ask whether you have had an opportunity to consider our rebuttal. We sincerely hope that our responses have adequately addressed your concerns and that the clarifications provided have helped resolve any issues. We have made every effort to carefully consider and respond to all the points raised.

We would also like to respectfully note that, given the generally positive tone of your overall assessment, we wonder whether the current score might not fully reflect your overall evaluation of the work, particularly in light of the clarifications and revisions we have provided during the discussion. If you feel that our responses have sufficiently addressed your concerns, we would be truly grateful if you would consider revisiting your score.

Thank you once again for your time, thoughtful feedback, and consideration. We genuinely appreciate the opportunity to engage in this discussion and improve our work. We would like to kindly ask the reviewer to reconsider their score if their concerns have been resolved.

Best regards,

Thank you for your response. The authors conducted several additional experiments demonstrating the benefits of their method. So I consider my concerns mostly addressed and increase my score.

Reviewer_Mc1bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CLEAN approximates SOAP’s covariance factors with Nyström sketches and uses different moment representations for the projected core and residual blocks. Q-CLEAN further compresses the stored states through quantization. The method offers a useful memory–convergence trade-off, but the strongest scaling claims extend beyond the training evidence provided.

Critical strengths
  1. Nyström sketches avoid storing dense covariance factors, while residual updates retain information outside the main subspace. This is a practical approach to reducing the cost of two-sided preconditioning.
  2. The evaluation includes actual wall-clock convergence: on 1.3B, CLEAN reaches AdamW’s final loss in 64 rather than 81 hours. This is more informative than reporting iteration counts alone.
Critical weaknesses
  1. Linear memory growth holds at fixed rank, but Appendix D scales rank with layer width, which restores quadratic growth for fixed-aspect-ratio matrices. Moreover, Eq. (14) omits momentum transfers between blocks when the basis changes, so it is not an exact transport of the represented momentum. Neither this approximation nor the claim that warm starts keep second-moment errors small is sufficiently evaluated.

  2. APOLLO (Zhu et al., 2025) is cited but not evaluated, and Q-CLEAN lacks an 8-bit Adam comparison, leaving the memory–time–quality advantage over simpler alternatives unclear. Convergence results stop at 1.3B; the 7B/13B results demonstrate memory feasibility rather than training quality. Larger-model convergence and downstream evaluations are needed to support the broader claims.

Questions for authors

See weaknesses.

5 comments

We thank the reviewer for their constructive feedback and provide answers to the raised questions below:

CW1:

Linear Memory Growth: The key distinction is that Clean’s optimizer memory has no mn,m2mn, m^2 or n2n^2 terms, whereas all baselines have at least one mnmn (Table 2). Thus, Clean is the only optimizer that fits a 13B model on one H100 GPU (Fig 1). We revised the manuscript to clarify that Clean scales linearly with model dimensions when the r is fixed (lines 84–86, 322–324, 882–886), reinforcing what was already included in line 96 of the original submission. In practice, the chosen rank often increases with model dimensions. To reflect this realistic setting, Fig 7 uses r≈d/8r\approx d/8, rather than assuming a fixed r.

First-Moment Transfer: Indeed, Eq. 14 omits the overlap terms arising from changes in subspace, as accounting for cross-block transfers does not mean exact information transfer. Each block's updates depend on its local moments' histories. So, the transferred first moments remain coupled to block-specific optimizer states. We therefore retain the formulation in Eq. 14, which only rotates within-block information.

However, we implemented the variant of Eq. 14 that accounts for cross-block overlaps (Eq. 23 in the revision). Pretraining the 350M model with this variant yields a val ppl. of 15.96, matching our reported result. This confirms that the additional terms do not provide performance improvement. We added the corresponding experiment to our revised draft (lines 289-291, 1247-1257.)

Warm-Start Effect: Warm-starting limits drift between subspaces and stabilizes the moments' estimates. Although a fixed subspace eliminates drift, it goes stale. To evaluate this, we compare warm- and cold-starting using 3 metrics:

-Subspace Overlap: Mean squared cosine of the principal angles between consecutive subspaces, measuring the similarity of their spans.

-Eigenvector Direction Retention (EVDR): Mean squared cosine between corresponding basis vectors across consecutive steps, measuring the stability of individual directions.

-Validation Perplexity: The impact on model performance.

We added the related plots in the revision (lines 325-326, 1057-1128, Fig 10). For both LtL_t and RtR_t, we report metrics averaged across layers, as well as results for the layer type with the largest warm and cold difference in drift. While average EVDR is generally similar across the two approaches, it is more sensitive to changes in directions with smaller eigenvalues. We therefore report EVDR restricted to basis vectors in the 4th eigenvalue quantile. The following table confirms that reducing subspace drift does not come at the cost of degraded performance.

Val PPL
Warm-Start15.87
Cold-Start16.02

CW2:

Larger Models: Second-order optimizers at scale are computationally expensive. For context, SOAP [1] evaluates models up to 660M parameters, and ALSO [2] only goes up to 130M. Clean scales to 1.3B parameters, and to further demonstrate scalability, we evaluate against AdamW on a 7B-parameter model for 2k iterations (~3days on 4 H100 GPUs).

Val PPLOpt Mem.
AdamW17.0925.71G
Clean (r = 1024)14.1911.66G

SOAP at 7B exceeds our computational budget. However, Clean outperforms AdamW with less than half its optimizer-state memory, highlighting the effectiveness of its second-order optimization.

Although none of our baselines fit within our H100 GPUs, we evaluate Clean's convergence on a 13B-parameter model over 3 days using 4 GPUs, as evidence of its scalability and convergence.

Step 50Step 300Step 600Step 900
Val PPL2356.2866.0126.9822.59

Additional Baselines: We evaluated APOLLO[3], Adam-8bit[4], and Q-GaLore[5] on a 350M-parameter model. We initially omitted methods such as APOLLO, as we already benchmarked against full-rank AdamW. Both Clean and Q-Clean outperform all 3 baselines due to their second-order nature.

Val PPLOpt Mem.
Clean15.871.25
Q-Clean16.440.38
Adam-8bit19.640.71
Q-GaLore18.620.32
APOLLO26.640.62

Downstream Tasks: We report the downstream evaluation results of the pretrained 1B-parameter models.

HellaSwagPIQAARC-eSciQSIQACSQAAvg.
AdamW34.268.748.378.738.430.649.8
SOAP35.769.847.677.939.529.149.9
Muon35.069.248.979.238.829.250.0
ALSO34.369.048.477.939.030.149.8
Clean35.469.449.779.338.929.950.4
Q-Clean34.769.247.678.339.230.950.0

We included all of these results in the revised manuscript. (lines 430-431, and Appendix E).

We thank the reviewer for the constructive feedback. We addressed their concerns with additional results and clarifications. We welcome further discussion and kindly ask the reviewer to reconsider their score if their concerns have been resolved.

[1] arXiv:2409.11321

[2] icml.cc/virtual/2026/75854

[3] arXiv:2412.05270

[4] arXiv:2110.02861

[5] arXiv:2407.08296

Dear Reviewer,

With the discussion period nearing its conclusion, we wanted to follow up and kindly ask whether you have had an opportunity to consider our rebuttal. We would greatly appreciate knowing if any aspects of our response require further explanation or if there are additional concerns we could address. We sincerely value the time and effort you have devoted to evaluating our work. Your feedback has provided us with an important opportunity to reconsider and strengthen our submission. We have carefully considered each of your comments and made every effort to respond comprehensively to the points raised. We would welcome any further discussion that could help resolve outstanding concerns and improve the clarity and quality of our work. Thank you again for your thoughtful feedback and consideration. We would like to kindly ask the reviewer to reconsider their score if their concerns have been resolved.

Best regards,

Thank you for the clarifications and additional experiments.

However, the warm-start results mainly measure subspace and basis stability, rather than the error in the second-moment approximation or the resulting preconditioned updates. The unchanged PPL in one 350M experiment is reassuring, but leaves the reliability of these approximations under other settings unclear. The new 7B and 13B runs also provide useful evidence of feasibility, while remaining too short to establish sustained convergence benefits at scale.

These concerns are only partially resolved, so I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.