- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors analyze how to reduce the memory usage in LLM training. At the same time, the authors want to preserve the convergence guarantees and practical performance. The authors present two algorithms, Subspace Momentum SGD with Residual Correction (SMRC) and Projected Muon with Residual Correction (PMRC). Both algorithms has the convergence rate under the standard assumptions.
Critical strengths
First, in this paper, the authors give the theoretical proofs for the convergence guarantees, and the theoretical analysis only replies to the standard assumptions. The convergence of SMRC accommodates predictable, gradient-dependent subspaces, which addresses the limitations of Galore.
Second, the convergence analysis seems non-trivial, especially for the PMRC. It is more challenging since the matrix sign is not linear. Therefore, the authors develop a weighted momentum tracking analysis and control the interaction between nuclear-norm and Frobenius-norm.
Third, the algorithm is straightforward and easy to follow, and the paper presents substantial experimental results. In addition, the authors give a detailed discussion on related work in Appendix A. It makes it clear what the novelties are.
Critical weaknesses
First, I believe this paper gives sufficient and novel theoretical analysis, but the algorithmic procedure seems standard. For example, the residual correction was analyzed in prior works. Fira combines subspace optimization with residual gradient correction, and Nguyen & Nguyen (2025) analyzes the projected momentum with SGD updates. Therefore, it seems that making the momentum in a subspace while applying SGD outside that subspace is related to prior works.
Second, I believe that there is a gap between the theoretical result and the experiment. The practical implementation uses , which is not shown in the theoretical PMRC algorithm. The paper shows the convergence of PMRC, but it is unknown whether or not the analysis can be applied to the practical implementation.
Questions for authors
Please see weaknesses.
Response to Reviewer wrr1
Many thanks for your constructive review and positive assessment of our paper. We have uploaded our revised paper, with the changes highlighted in blue for ease of reference.
Response. We thank the reviewer for recognizing our theoretical contributions. Although residual correction has been investigated in prior work, our study advances both algorithmic design and theoretical analysis, with the key distinctions outlined below.
Fira incorporates residual correction into GaLore to recover the discarded gradient information. Fira is built upon AdamW and employs a heuristically designed approach. More importantly, it does not establish convergence guarantees. In contrast, our work provides rigorous convergence guarantees for both momentum SGD and Muon with residual correction.
Although Nguyen & Nguyen (2025) also combine projected momentum SGD with residual correction, we identify a mathematical error in their convergence analysis. Specifically, the proof of Lemma D.1 omits a nonnegative term from an upper bound, which invalidates a key step of their proof. Consequently, their claimed convergence guarantee is not established by the existing analysis, as detailed in Appendix A.1. Moreover, their analysis is restricted to a fixed projection basis and only considers an unweighted residual update. In contrast, we establish convergence guarantees for multi-stage SMRC with periodically refreshed subspaces by developing a boundary correction technique that ensures the consistency of virtual iterates across stage transitions.
We further extend the residual correction framework to Muon and propose PMRC, which applies the matrix sign to the projected momentum. The nonlinear dependence of the update prevents an application of the virtual-iterate analysis developed for SMRC. To address this challenge, we develop a weighted momentum tracking analysis that accounts for the interaction between nuclear-norm and Frobenius-norm geometries. Based on this analysis, we establish convergence guarantees for both single-stage and multi-stage PMRC under a mixed stationarity measure.
Response. We thank the reviewer for pointing out this issue. We acknowledge that there is a minor difference between the theoretical analysis and the practical implementation. To address this concern, we extend the analysis of Theorem 4 to accommodate the residual cap, with the detailed proof provided in Appendix E.2.1 (page 31). Under Assumptions 1–4, with a fixed ρ and the parameter settings in Theorem 4, we can ensure
N1∑k=1K∑t=1TE[∣Pk⊤∇F(Wk,t)∣∗+∣Qk∇F(Wk,t)∣F]≤O(N−1/4)+2γN1∑k=1K∑t=1TE[(γ−γk,t)2∣QkGk,t∣F2].
The second term characterizes the effect of the residual cap. In particular, if N1∑k=1K∑t=1TE[(γ−γk,t)2∣QkGk,t∣F2]=O(1), the capped PMRC retains the original O(N−1/4) convergence rate.
To further evaluate the effect of the residual cap, we conduct additional experiments to examine the cap activation frequency and the magnitude of the coefficient reduction during training, with the cap activation frequency calculated separately for each interval rather than over all training steps.
These results further support the conclusion that the residual cap primarily improves training stability without substantially altering the original PMRC updates.
References.
[1] Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint? NeurIPS, 2025.
[2] Lean and Mean Adaptive Optimization via Subset-norm and Subspace-momentum with Convergence Guarantees. ICML, 2025.
We hope that our responses address your concerns and are happy to answer any further questions during the reviewer–author discussion period.