- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors show that there are two main weaknesses of the existing memory efficient LLM training methods: first the projection subspace may become outdated as gradients evolve, and second they might not be sufficient to recover gradient information discarded by low-rank projection. To address these problems, the authors propose LoSARE that builds upon GaLore and Fira.
Critical strengths
First, this paper studies an important and practical problem in LLM training. LoSARE may address this memory efficiency problem without restricting the model parameter to low rank. It may keep the full rank parameter updates while allowing the low-rank optimizer states.
Second, this new approach is computationally efficient and easy to implement. Replacing the periodic SVD with the Gram–QR iteration can reduce the computational cost.
Third, the authors have analyzed the model pretraining on different scales, from 60M to 7B. LoSARE consistently archives better validation perplexity than GaLore and Fira.
Critical weaknesses
My major concern regarding this paper is its novelty. I believe that Gram–QR subspace iteration is a well-studied approach. The update is a standard orthogonal iteration step with a warm start. In addition, the residual recovery framework seems to follow from Fira. The major change is that the authors introduce .’
Second, I believe that there is a gap between the theoretical analysis and the actual algorithm. In Lemma 1, the authors show that an undamped QR iteration contracts the principal-angle distance to the dominant singular subspace, whereas the actual LoSARE algorithm uses a damped update .
Questions for authors
Please see weaknesses.
We sincerely thank the reviewer for the constructive comments. We address the main concerns below.
Q1. Novelty of the adaptive subspace tracking and residual enhancement mechanisms.
Response:
We agree that Gram-based subspace iteration and QR orthogonalization are established techniques. Our contribution is not to introduce these operations independently, but to integrate warm-started Gram–QR iteration with a damped basis update for efficient gradient subspace tracking. Specifically, Eq. (10) uses the previous basis and an adaptation rate to avoid abrupt basis replacement and repeated full SVD computations.
Similarly, LoSARE builds upon Fira's residual recovery but introduces energy-aware residual scaling (Eq. (14)), which adjusts the residual contribution according to the discarded gradient energy. We additionally conduct ablation studies on the 60M model, which support the complementary benefits of these components: adaptive projection improves GaLore from 34.09 to 32.08 PPL, while residual enhancement improves Fira from 30.30 to 30.26 PPL. The complete LoSARE achieves 30.19 PPL. Thus, our contribution lies in addressing subspace staleness and projection information loss jointly.
Q2. Theoretical justification and the gap between Lemma 1 and the implemented damped update.
Response:
Thank you for identifying this distinction. Lemma 1 analyzes the underlying Gram–QR subspace iteration, explaining its tendency to capture dominant singular directions under the stated assumptions. The actual implementation further introduces the interpolation controlled by (ηa) in Eq. (10) to improve stability during nonstationary training.
We acknowledge that Lemma 1 does not directly establish convergence of the complete damped update under arbitrary time-varying gradients. Our intention is to theoretically characterize the underlying subspace-seeking mechanism, rather than claim a global convergence guarantee for the entire optimizer. We will clarify this scope to avoid overstating the theoretical result.