acceptodds
Under review as a conference paper at ICLR 2027

LoSARE: Low-Rank Optimization for Training LLMs with Adaptive Subspace Tracking

Abstract

Low-rank optimization has emerged as an effective approach to reducing the memory cost for training Large Language Models (LLMs). Existing methods are still insufficient in adapting to evolving gradient subspaces and recovering gradient information discarded by low-rank projection. We present LoSARE, a memory-efficient low-rank optimization method that addresses both issues through adaptive subspace tracking and energy-aware residual enhancement. LoSARE incrementally updates the projection subspace according to the current gradient geometry. The gradient subspace is continually tracked with low-complexity QR decomposition, avoiding the need for costly SVD re-computation in GaLore. It further enhances the discarded gradient residual according to its relative energy, while low-rank optimizer states and stable norm-growth control are still retained. Extensive experiments on LLaMA models demonstrate that LoSARE consistently outperforms GaLore and Fira in memory-efficient pre-training and fine-tuning, under the constraints of preserving their low memory footprint. Notably, on the LLaMA-7B pre-training task, LoSARE with projection rank 64 achieves comparable or better performance than GaLore with ranks 512 and 1024, using and smaller ranks, respectively. At the same time, LoSARE achieves and faster training speed than GaLore with rank 1024 and Fira with rank 64, respectively. Furthermore, LoSARE maintains strong performance under substantially lower rank constraints and exhibits more effective late-stage optimization. These results demonstrate that our method provides an effective balance among performance, memory efficiency, and training cost for LLM training.

Primary area:
optimization
Keywords:
low-rank optimization, adaptive subspace tracking, large language models, memory-efficient training, LLM fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_MFyEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors show that there are two main weaknesses of the existing memory efficient LLM training methods: first the projection subspace may become outdated as gradients evolve, and second they might not be sufficient to recover gradient information discarded by low-rank projection. To address these problems, the authors propose LoSARE that builds upon GaLore and Fira.

Critical strengths

First, this paper studies an important and practical problem in LLM training. LoSARE may address this memory efficiency problem without restricting the model parameter to low rank. It may keep the full rank parameter updates while allowing the low-rank optimizer states.

Second, this new approach is computationally efficient and easy to implement. Replacing the periodic SVD with the Gram–QR iteration can reduce the computational cost.

Third, the authors have analyzed the model pretraining on different scales, from 60M to 7B. LoSARE consistently archives better validation perplexity than GaLore and Fira.

Critical weaknesses

My major concern regarding this paper is its novelty. I believe that Gram–QR subspace iteration is a well-studied approach. The update Ptnew=qf(GtGt⊤Pt−1)P_t^{\mathrm{new}}=\mathrm{qf}(G_tG_t^\top P_{t-1}) is a standard orthogonal iteration step with a warm start. In addition, the residual recovery framework seems to follow from Fira. The major change is that the authors introduce γt=1+αrρt1+ρt\gamma_t=1+\alpha_r\frac{\rho_t}{1+\rho_t}.’

Second, I believe that there is a gap between the theoretical analysis and the actual algorithm. In Lemma 1, the authors show that an undamped QR iteration contracts the principal-angle distance to the dominant singular subspace, whereas the actual LoSARE algorithm uses a damped update Pt=qf((1−ηa)Pt−1+ηaPtnew)P_t = \mathrm{qf}((1 - \eta_a) P_{t - 1} + \eta_a P_t^{\mathrm{new}}).

Questions for authors

Please see weaknesses.

1 comment

We sincerely thank the reviewer for the constructive comments. We address the main concerns below.

Q1. Novelty of the adaptive subspace tracking and residual enhancement mechanisms.

Response:

We agree that Gram-based subspace iteration and QR orthogonalization are established techniques. Our contribution is not to introduce these operations independently, but to integrate warm-started Gram–QR iteration with a damped basis update for efficient gradient subspace tracking. Specifically, Eq. (10) uses the previous basis and an adaptation rate to avoid abrupt basis replacement and repeated full SVD computations.

Similarly, LoSARE builds upon Fira's residual recovery but introduces energy-aware residual scaling (Eq. (14)), which adjusts the residual contribution according to the discarded gradient energy. We additionally conduct ablation studies on the 60M model, which support the complementary benefits of these components: adaptive projection improves GaLore from 34.09 to 32.08 PPL, while residual enhancement improves Fira from 30.30 to 30.26 PPL. The complete LoSARE achieves 30.19 PPL. Thus, our contribution lies in addressing subspace staleness and projection information loss jointly.

Q2. Theoretical justification and the gap between Lemma 1 and the implemented damped update.

Response:

Thank you for identifying this distinction. Lemma 1 analyzes the underlying Gram–QR subspace iteration, explaining its tendency to capture dominant singular directions under the stated assumptions. The actual implementation further introduces the interpolation controlled by (ηa\eta_a) in Eq. (10) to improve stability during nonstationary training.

We acknowledge that Lemma 1 does not directly establish convergence of the complete damped update under arbitrary time-varying gradients. Our intention is to theoretically characterize the underlying subspace-seeking mechanism, rather than claim a global convergence guarantee for the entire optimizer. We will clarify this scope to avoid overstating the theoretical result.

Reviewer_Mjz9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoSARE, a memory-efficient optimizer for LLM training that combines two mechanisms: 1- Adaptive subspace tracking: Instead of periodically recomputing a projection basis using a full SVD, LoSARE applies one warm-started Gram/subspace-iteration step. 2- Energy-aware residual enhancement: Building on Fira’s residual recovery, LoSARE computes the energy ratio between the discarded residual and projected gradient, then uses a bounded gain to strengthen the residual contribution. The paper reports improvements over GaLore and Fira, including stronger late-stage optimization at rank 64 in the 7B experiment.

Critical strengths

1- The motivation is clear; periodically updating the projection may lead to stale subspace as gradient geometry changes, while discarded residual information may still be useful. Combining continual basis adaptation with residual recovery is a reasonable and relevant direction for memory-efficient LLM training.

2- The warm-started Gram–QR update provides a computationally efficient way to adapt the projection basis while encouraging continuity between consecutive updates. The residual gain is also bounded and interpretable.

3- The paper evaluates pre-training from 60M to 7B parameters and also includes LLaMA-7B commonsense fine-tuning. Also the rank ablation on 60M reports strong performance at very small ranks, and the inclusion of lower ranks on 7B experiments covers the down side of small model size in the ablation.

Critical weaknesses

1- LoSARE changes both the projection update and the residual recovery mechanism, but the experimental section does not provide a clean component-wise ablation. Specifically, the role of subspace continual adjustments, residual recovery, and energy-aware enhancement are not isolated.

2- As the projection basis changes, the algorithm directly reuses the low-rank Adam moments in the coordinates of the updated basis. The paper does not empirically evaluate the effect of this approximation or compare it with alternatives such as transporting or rotating the moments into the new basis, as considered in prior work such as LDAdam [1]. This is important because two basis matrices can represent the same subspace while having different coordinates. Consequently, directly reusing coordinate-dependent optimizer states may produce different updates under equivalent basis representations.

3- The paper discusses InGaLore [2], SubTrack++ [3], and LDAdam [1], but the experiments compare primarily against GaLore and Fira. Given that the main contribution is continual adaptive tracking, comparison with at least one tracking-aware baseline would be important.

4- Reporting empirical runtime and peak-memory measurements would better support the efficiency claims. Results across multiple random seeds, at least for a smaller model such as the 350M setting, would also strengthen the evaluation, especially because some of the improvements over Fira are relatively small.

5- The claimed tracking mechanism is not directly measured. A measurement of principal-angles distant to current SVD, or tracking error vs. GaLore, incremental SVD, or SubTrack++, would help in framing the contribution.


References

[1] Robert, T., Safaryan, M., Modoranu, I.-V., and Alistarh, D. LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. ICLR 2025.

[2] Meng, X., Abdelgawad, M. A. A., Jing, P., Cheung, R. C. C., and Yan, H. InGaLore: Memory-Efficient Neural Network Training via Gradient Compression Through Continuous Basis Tracking. Neurocomputing, 680:133278, 2026.

[3] Rajabi, S., Nonta, N., and Rambhatla, S. SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. 2025.

Questions for authors

Please refer to weaknesses.

4 comments

We sincerely thank the reviewer for the detailed suggestions. We conducted additional experiments and clarify the remaining implementation details below.

Q1. Insufficient ablation studies.

Response:

We conducted additional 60M ablations:

MethodPPL
GaLore34.09
GaLore + adaptive projection32.08
Fira30.30
Fira + residual enhancement30.26
LoSARE30.19

These demonstrate the individual effectiveness of the two proposed components, both of which are capable of improving performance, and their combination yields the best results.

Q2. Optimizer moment reuse when the projection basis changes.

Response:

LoSARE retains the low-rank Adam moments when updating the projection basis, without explicitly transporting or resetting them. We acknowledge that this is not mathematically equivalent to exact moment transport under arbitrary basis changes. However, the damped basis update in Eq. (10) encourages gradual adaptation and mitigates abrupt coordinate changes. This design also avoids additional state transformations and preserves the memory efficiency shown in Table 1. Explicit moment transport remains a potential extension.

Q3. Additional baseline comparisons.

Response:

Due to limited time to modify the code and reproduce the results, we supplement the comparison with 10K-step LDAdam and SubTrack++ results reported in SubTrack++[1], converting their reported losses to PPL using (PPL=exp⁡(Loss)\mathrm{PPL}=\exp(\mathrm{Loss})). Since the hyperparameter configurations for the 60M, 130M, and 350M models are identical to those in SubTrack++[1], we restrict the comparison to these three model sizes. Moreover, we report LoSARE's PPL at 10K steps to ensure a fair comparison.

ModelLDAdamSubTrack++LoSARE
60M33.7830.8730.19
130M31.1825.5325.70
350M39.2526.8424.11

We explicitly clarify that LDAdam and SubTrack++ are literature-derived results, not independent reproductions. LoSARE uses our own measured 10K-step checkpoints. LoSARE outperforms these references at 60M and 350M and remains competitive at 130M.

Q4. Runtime and stability across random seeds.

Response:

We measured the 7B training time under identical GPU configurations and other training settings:

MethodConfigurationTime
LoSARERank 642,383 min
FiraRank 642,901 min
GaLoreRank 10242,682 min

We conducted three-seed experiments at different model scales:

ModelStepsPPL (three seeds)Mean ± Std
60M10K30.09 / 30.19 / 30.2230.17 ± 0.07
130M20K22.53 / 22.62 / 22.6422.60 ± 0.06
350M60K16.75 / 16.79 / 16.8716.80 ± 0.06

The small variations demonstrate stable performance across random seeds. We explicitly report the different training budgets rather than implying a uniform number of steps. Furthermore, we emphasize that rather than pursuing a substantial performance margin over methods like Fira, our contribution lies in providing a novel perspective, which is evidenced by LoSARE even outperforming the Full-Rank baseline in terms of PPL.

Q5. Direct subspace tracking diagnostics.

Response:

We agree that principal-angle measurements could provide additional geometric insights. Although we have not directly measured subspace tracking errors, the ablation results provide indirect evidence of the effectiveness of adaptive tracking: adding adaptive projection improves GaLore from 34.09 to 32.08 PPL on 60M. Given the limited time available to conduct comprehensive experiments, we therefore focus our current empirical claims on the demonstrated optimization improvements rather than claiming unmeasured geometric guarantees.


References

[1] Rajabi, S., Nonta, N., and Rambhatla, S. SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. 2025.

Thank you for the response. In Q1, the adaptive subspace tracking + Fira residual recovery, but without energy-aware enhancement is not reported, thus the isolation is still not complete; also Q2 and Q5 are not answering my questions. I am maintaining my current evaluation of the paper.

Q1. Completeness of component ablations

We thank the reviewer for pointing out this missing comparison. We agree that reporting only GaLore with adaptive projection does not fully isolate the contributions of adaptive subspace tracking and energy-aware residual enhancement.

Our initial ablation focused on GaLore because the improvement was more pronounced and thus provided a clearer demonstration of the effect of adaptive projection itself (34.09 →\rightarrow 32.08 PPL). However, we should also have reported the corresponding Fira-based variant. We have obtained this additional result: Fira + adaptive projection, without energy-aware enhancement, achieves 30.24 PPL, compared with 30.19 PPL for full LoSARE.

This additional comparison completes the component-level evaluation: adaptive projection improves the GaLore baseline, while adding energy-aware residual enhancement to the Fira-based adaptive variant yields a further, albeit modest, improvement. We will include this missing result and clarify the respective contributions without overstating the magnitude of the residual enhancement gain.

Q2. Reuse of optimizer moments under basis changes

We thank the reviewer for clarifying this concern. We agree that directly reusing coordinate-dependent Adam moments after a basis update is an approximation, and that our current experiments do not quantify its effect against explicit moment transport.

An important subtlety is that, although the first moment can be transformed under an orthogonal change of basis, Adam's coordinate-wise second-moment estimates do not generally admit an equivalent transformation under arbitrary rotations. Exact treatment would require maintaining or transforming richer second-order statistics, introducing additional computational and memory costs.

The primary focus of LoSARE is to develop an adaptive low-rank optimization framework that improves optimization performance while retaining low-rank optimizer states and computational efficiency, rather than to introduce a new optimizer-state transport mechanism. We therefore retain the existing low-rank moments as an efficiency-oriented design choice. The damped basis update in Eq. (10) encourages gradual basis adaptation, but does not guarantee coordinate consistency or eliminate the approximation.

We acknowledge that a controlled comparison with explicit moment transport would provide a more complete understanding of this design choice. We will clarify this limitation and leave a systematic investigation of optimizer-state transport for future work.

Q5. Direct diagnostics of subspace tracking

We appreciate the reviewer’s suggestion and fully understand the need for direct geometric evaluations, such as principal-angle measurements against the current SVD subspace and tracking-error comparisons with GaLore, incremental SVD, or SubTrack++.

We would like to clarify that we have not overlooked this concern or intended to avoid the requested evaluation. Rather, conducting these additional experiments would require substantial implementation, checkpoint-level analysis, and computational resources. Unfortunately, the time and resources available during the rebuttal period do not permit us to complete such a systematic evaluation reliably.

Our existing ablation results (34.09 →\rightarrow 32.08 PPL on 60M when adding adaptive projection to GaLore) demonstrate the optimization benefit of the adaptive projection strategy, but we acknowledge that they do not directly measure geometric tracking accuracy. We will therefore clarify this distinction and avoid making claims about superior subspace tracking accuracy over existing methods without direct evidence. We consider the proposed principal-angle and tracking-error analyses valuable future work for further understanding the geometric behavior of LoSARE.

We hope this clarifies that the missing evaluation reflects a limitation in the scope of the current experimental study, rather than a lack of consideration of the reviewer’s concern.

We appreciate the reviewer’s suggestion and reply.

For Q2, The primary focus of LoSARE is to develop an adaptive low-rank optimization framework that improves optimization performance while strictly retaining low-rank optimizer states. It is not intended to introduce a new optimizer-state transport mechanism. Therefore, retaining the existing low-rank moments is a principled efficiency-oriented design choice. Our proposed approach is dedicated to establishing an adaptive low-rank optimization mechanism. Forcibly incorporating these techniques would appear as an "A+B+C" patchwork rather than a cohesive design, which deviates from the core objective of our work.

For Q5, We want to clarify that we did not intend to avoid this. Conducting these additional experiments would require checkpoint-level analysis, substantial code implementation, and computational resources, which is practically infeasible within the limited time of the rebuttal period. Our existing ablation results (improving GaLore from 34.09 to 32.08 PPL on 60M with adaptive projection) serve as indirect but concrete evidence of the optimization benefits brought by adaptive tracking. We will frame the principal-angle analysis explicitly as valuable future work.

To the Area Chairs: We respectfully ask the ACs to consider that the reviewer's decision to maintain their evaluation is based on the absence of specific empirical diagnostics rather than flaws in our core methodology. As clarified, exact moment transport is mathematically non-trivial for Adam (due to the second moment) and would violate our low-rank memory constraints. Furthermore, while principal-angle diagnostics are insightful, they fall outside the central scope of this paper. We believe the current extensive empirical validation supports the paper's core contributions and kindly ask the ACs to weigh these points carefully in the final decision.

Reviewer_YpU8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LoSARE, a low-rank optimizer for training large language models. The method maintains Adam’s first- and second-moment states in a low-dimensional space while improving gradient utilization through two mechanisms. First, it continuously adapts the projection subspace using a gradient-driven, warm-started Gram–QR update and basis interpolation. Second, it applies a bounded, dynamic enhancement to Fira-style residual recovery based on the relative energy of the out-of-subspace residual. Experiments cover C4 pre-training at scales from 60M to 7B parameters, as well as LLaMA-7B fine-tuning on eight commonsense reasoning benchmarks. The results show promising performance while retaining low-rank persistent states, particularly under small projection ranks.

Critical strengths
  1. The problem is practically relevant, and the method is relatively simple. The paper addresses two related limitations of low-rank optimization: stale projection subspaces and discarded gradient information. Updating the projection basis through matrix products and reduced QR avoids repeated full SVD computations during training, while residual enhancement does not require full-sized Adam states. This design has practical potential.
  2. The experimental results are promising, particularly at low ranks. In Table 2, LoSARE achieves lower final perplexity than the reproduced Fira and GaLore baselines at all four model scales. For example, on the 1B model, perplexity decreases from 14.96 with Fira to 14.62. In the 7B experiment, rank-64 LoSARE achieves final performance comparable to or better than GaLore at ranks 512 and 1024. The results under very low ranks in Figure 3a are also noteworthy.
Critical weaknesses
  1. The experiments do not sufficiently isolate the contributions of the two proposed mechanisms. Residual recovery and norm-growth control are inherited from Fira, while Gram–QR is closely related to classical subspace iteration. Consequently, the strength of the contribution depends substantially on whether the specific adaptive tracking strategy and energy-aware enhancement each provide verifiable benefits. The current hyperparameter sensitivity experiments do not fully answer this question. I suggest comparing the following under identical experimental settings: Fira; Fira with energy-aware enhancement; Fira with only periodic SVD replaced by the proposed subspace tracking procedure; the complete LoSARE method. A tuned constant residual-gain baseline is also needed to distinguish the value of dynamic energy-aware enhancement from simply increasing residual recovery strength. The αr=0\alpha_r=0 comparison in Figure 3b is useful, but it remains insufficient to establish the independent contributions of both components. Direct measurements of projection error or dominant-subspace alignment would also provide stronger support for the proposed tracking mechanism.
  2. The 7B comparison protocol and practical efficiency evidence require further clarification. Section 5.2 describes a comparison against a historical GaLore checkpoint, whereas Figure 2 presents complete training trajectories for several methods and ranks. It is unclear which curves were reproduced in this work, which were obtained from historical results, and how the horizontal axes and training budgets were aligned. Under the reported settings, 150K steps with batch size 128 and sequence length 256 correspond to approximately 4.915B tokens, whereas 40K steps with batch size 512 correspond to approximately 5.243B tokens. The token budgets are therefore not strictly identical. Different methods also use different learning rates and scaling factors in the 1B and 7B experiments. Method-specific hyperparameters are reasonable, but the tuning budgets and selection criteria should be explained. Furthermore, the paper reports 1.12× and 1.22× training speedups without providing absolute timings, throughput, or a sufficiently clear timing configuration. The memory tables primarily estimate parameters and persistent states, which cannot substitute for measurements of full training peak memory. Since LoSARE updates its subspace every step while the baselines refresh it every 200 steps, the complexity of a single decomposition is insufficient to establish an end-to-end speed advantage.
  3. The theoretical results should be more explicitly scoped to the procedures they actually analyze. The contraction lemma in Appendix G.1 concerns an undamped Gram–QR step under a spectral-gap assumption. It does not cover the implemented basis interpolation, stochastic time-varying gradients, or Adam-state reuse. The authors acknowledge this limitation, so I view the result as motivation for the design rather than a tracking guarantee for the complete algorithm. In addition, Equations (40)–(45) in Appendix G.3 analyze the residual before the norm-growth limiter. With the previous residual norm fixed, the implemented limiter caps the norm of the residual applied to the parameters. Therefore, the affine and unbounded growth of the unrestricted residual ratio with αr\alpha_r does not directly characterize the complete algorithm. Column-dependent gains may still alter the residual direction, but this requires additional analysis. This observation does not contradict the empirical instability in Figure 4; rather, it limits the explanatory strength of the current theory for that phenomenon.
Questions for authors
  1. Can the authors provide the four controlled component comparisons described above, together with a comparison between dynamic energy-aware enhancement and a tuned constant gain? Do these results establish the need for each of the two proposed mechanisms?
  2. What is the source of each curve in Figure 2? Can the authors report end-to-end throughput, absolute training time, and peak memory under clearly specified budgets?
  3. Does basis alignment or moment transport change performance and stability? Without these procedures, how large are the coordinate changes encountered during training?
1 comment

We sincerely thank the reviewer for the valuable comments on experimental fairness, hyperparameter sensitivity, and theoretical justification.

W1. Insufficient ablation studies and the effectiveness of dynamic residual enhancement.

Response:

We conducted additional 60M ablations:

MethodPPL
GaLore34.09
GaLore + adaptive projection32.08
Fira30.30
Fira + residual enhancement30.26
LoSARE30.19

These demonstrate the individual effectiveness of the two proposed components, both of which are capable of improving performance, and their combination yields the best results.

We also compared adaptive enhancement against fixed residual gains:

ConfigurationPPL
Fixed gain = 1.2530.33
Fixed gain = 1.530.36
Fixed gain = 2.030.25
Fixed gain = 3.030.52
Fixed gain = 5.0174.15
Adaptive LoSARE30.19

The results show that the dynamic energy-aware enhancement outperforms the fixed residual gain and demonstrates its practical value. Simply increasing residual magnitude is insufficient and can destabilize training. Energy-aware enhancement achieves the best result among the tested configurations.

W2. Fairness of the 7B experiments and the 150K-versus-40K-step comparison and Runtime.

Response:

We apologize for the ambiguity in our original description. We clarify that all curves in Figure 2 are obtained from our own reproduced experiments using batch size 128 and 150K steps for all methods. No curves from different batch sizes or training budgets are mixed.

The discussion of 40K versus 150K steps was only intended to explain the approximate token-budget correspondence with the original GaLore experiment. With sequence length 256, our setting processes (150K×128×256=4.9152)(150\mathrm{K}\times128\times256=4.9152)B tokens, compared with (40K×512×256=5.24288)(40\mathrm{K}\times512\times256=5.24288)B tokens in the original setting, a difference of approximately 6.25%. We also observed comparable final PPL in our GaLore reproduction under these two token-matched settings. Thus, the 150K-step setting is a reasonable resource-constrained choice, while Figure 2 maintains a fully consistent experimental protocol.

We measured the 7B training time under identical GPU configurations and other training settings:

MethodConfigurationTime
LoSARERank 642,383 min
FiraRank 642,901 min
GaLoreRank 10242,682 min

Under identical GPU and training configurations, LoSARE-64 takes 2,383 minutes, compared with 2,901 minutes for Fira-64 and 2,682 minutes for GaLore-1024. This corresponds to 1.12×1.12\times and 1.22×1.22\times training speedups, respectively. These actual measurements further demonstrate the practical efficiency of LoSARE.

W3. Theoretical justification of the adaptive update and norm-growth limiter.

Response:

We acknowledge that Lemma 1 does not directly establish convergence of the complete damped update under arbitrary time-varying gradients. Our intention is to theoretically characterize the underlying subspace-seeking mechanism, rather than claim a global convergence guarantee for the entire optimizer. We will clarify this scope to avoid overstating the theoretical result.

The norm-growth limiter, retained from Fira, restricts the relative growth of the enhanced residual to 1% between consecutive iterations. It serves as a practical stabilization mechanism rather than an independently proven convergence guarantee. We will clarify these distinctions in the manuscript.

Q3. Optimizer moment reuse when the projection basis changes and direct subspace tracking diagnostics.

Response:

LoSARE retains the low-rank Adam moments when updating the projection basis, without explicitly transporting or resetting them. We acknowledge that this is not mathematically equivalent to exact moment transport under arbitrary basis changes. However, the damped basis update in Eq. (10) encourages gradual adaptation and mitigates abrupt coordinate changes. This design also avoids additional state transformations and preserves the memory efficiency shown in Table 1. Explicit moment transport remains a potential extension.

Although we have not directly measured subspace tracking errors, the ablation results provide indirect evidence of the effectiveness of adaptive tracking: adding adaptive projection improves GaLore from 34.09 to 32.08 PPL on 60M. Given the limited time available to conduct comprehensive experiments, we therefore focus our current empirical claims on the demonstrated optimization improvements rather than claiming unmeasured geometric guarantees.

Reviewer_X5RjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies memory-efficient LLM training via low-rank optimizer states. The proposed LoSARE introduces a continuously updated projection basis obtained through Gram-QR iterations and an energy-dependent scaling of the gradient residual outside that basis. The method computes full gradients similarly to related method, but claims memory savings on the optimizer states. Validation includes LLaMA pre-training from 60M to 7B parameters, commonsense reasoning fine-tuning, and rank and hyperparameter sensitivity experiments. The reported results improve over GaLore and, generally by smaller margins, Fira, which is the closest method the authors built on.

Critical strengths
  1. The paper addresses a relevant problem, combining subspace tracking with residual recovery is a reasonable approach to improving optimization while retaining low-rank optimizer states
  2. The evaluation covers several model scales and both pre-training and fine-tuning. The experiments at very small ranks are particularly relevant to the intended memory-constrained setting.
Critical weaknesses
  1. The contribution is incremental over Fira, and the experiments do not sufficiently isolate the value of the additions. Residual recovery and norm-growth control are inherited from Fira; the additions are a tracker based on Gram–QR iteration and an energy-dependent residual multiplier. The zero-enhancement ablation supports increasing the residual contribution, but does not establish the benefit of energy dependence over a tuned constant multiplier. The method also changes the projection basis while reusing Adam moments in their previous coordinates, even though this is acknoledged it would be better to evaluate its impact. I also found missing comparisons with LDAdam or SubTrack++, which address optimizer states under changing subspaces, and so they would be particularly informative. For fine-tuning, LoFT is also a relevant missing comparison: it addresses optimizer-state alignment and evaluates LLaMA-7B on the same commonsense benchmarks.
  2. The performance–efficiency comparison is insufficiently controlled or documented. The stated 7B comparison uses LoSARE after completing learning-rate decay versus an intermediate GaLore checkpoint from a longer schedule. Matching token counts would be not enough in this case, and it is unclear which Figure 2 trajectories were rerun under matched schedules. The reported speedups also lack a defined timing protocol, while the per-update complexity comparison does not account for QR running every step versus SVD every 200 steps. Those confounders make it difficult to clearly assess the claimed quality-cost advantage.
Questions for authors
  1. About Fira implementation formual. Equation 23 describes Fira as applying the scale factor only to the projected update, while Fira’s published algorithm scales the combined projected and residual update. Which implementation was used in experiments? Is this a notation discrepancy, or does the reproduced baseline use a different relative weighting of the two components?
  2. Hparam tuning. At 1B and 7B, Table 5 gives different learning rates and scale factors for GaLore versus Fira and LoSARE. How were these selected, and did each method receive a comparable tuning budget? Currently, it is not established that imporovements may come from unequal tuning
  3. Significance of gains wrt Fira. Were any experiments repeated across seeds? In Table 3, most of the average improvement over Fira comes from HellaSwag, while the average difference across the remaining seven tasks is approximately 0.06 percentage points, rather small to establish better optimization. Is the HellaSwag improvement reproducible, and what variability was observed in the smaller pre-training gains?
6 comments

We sincerely thank the reviewer for carefully examining our implementation and experimental evaluation.

W1. Insufficient ablation studies and additional baseline comparisons.

We conducted additional 60M ablations:

MethodPPL
GaLore34.09
GaLore + adaptive projection32.08
Fira30.30
Fira + residual enhancement30.26
LoSARE30.19

These demonstrate the individual effectiveness of the two proposed components, both of which are capable of improving performance, and their combination yields the best results.

Due to limited time to modify the code and reproduce the results, we supplement the comparison with 10K-step LDAdam and SubTrack++ results reported in SubTrack++[1], converting their reported losses to PPL using (PPL=exp⁡(Loss)\mathrm{PPL}=\exp(\mathrm{Loss})). Since the hyperparameter configurations for the 60M, 130M, and 350M models are identical to those in SubTrack++[1], we restrict the comparison to these three model sizes. Moreover, we report LoSARE's PPL at 10K steps to ensure a fair comparison.

ModelLDAdamSubTrack++LoSARE
60M33.7830.8730.19
130M31.1825.5325.70
350M39.2526.8424.11

We explicitly clarify that LDAdam and SubTrack++ are literature-derived results, not independent reproductions. LoSARE uses our own measured 10K-step checkpoints. LoSARE outperforms these references at 60M and 350M and remains competitive at 130M.

W2. Fairness of the 7B experiments and the 150K-versus-40K-step comparison and Runtime.

Response:

We apologize for the ambiguity in our original description. We clarify that all curves in Figure 2 are obtained from our own reproduced experiments using batch size 128 and 150K steps for all methods. No curves from different batch sizes or training budgets are mixed.

The discussion of 40K versus 150K steps was only intended to explain the approximate token-budget correspondence with the original GaLore experiment. With sequence length 256, our setting processes (150K×128×256=4.9152)(150\mathrm{K}\times128\times256=4.9152)B tokens, compared with (40K×512×256=5.24288)(40\mathrm{K}\times512\times256=5.24288)B tokens in the original setting, a difference of approximately 6.25%. We also observed comparable final PPL in our GaLore reproduction under these two token-matched settings. Thus, the 150K-step setting is a reasonable resource-constrained choice, while Figure 2 maintains a fully consistent experimental protocol.

We measured the 7B training time under identical GPU configurations and other training settings:

MethodConfigurationTime
LoSARERank 642,383 min
FiraRank 642,901 min
GaLoreRank 10242,682 min

Under identical GPU and training configurations, LoSARE-64 takes 2,383 minutes, compared with 2,901 minutes for Fira-64 and 2,682 minutes for GaLore-1024. This corresponds to 1.12×1.12\times and 1.22×1.22\times training speedups, respectively. These actual measurements further demonstrate the practical efficiency of LoSARE.

Q1. Inconsistency between the Fira algorithm and its implementation.

Response:

Thank you for identifying this issue. We carefully checked the released Fira implementation. The appendix pseudocode contains a scaling-placement inconsistency: according to the released code, the coefficient (α\alpha) is applied to the reconstructed low-rank update, while the residual is added separately:G~tFira=αPtNt+S^t.\widetilde G_t^{\mathrm{Fira}}=\alpha P_tN_t+\widehat S_t.

Our Fira implementation and formulation follow the released code, rather than the inconsistent appendix pseudocode.

Q2. Hparam tuning.

It is a conventional default in LLM training to use smaller learning rates for larger models. Furthermore, we empirically observe that on the 1B and 7B models, GaLore degrades significantly more if it uses the same small learning rate as Fira and LoSARE, which motivates our choice of hyperparameters.


References

[1] Rajabi, S., Nonta, N., and Rambhatla, S. SubTrack++: Gradient Subspace Tracking for Scalable LLM Training. 2025.

Q3. Fine-tuning reliability and the absence of additional multi-seed results.

We conducted three-seed experiments at different model scales:

ModelStepsPPL (three seeds)Mean ± Std
60M10K30.09 / 30.19 / 30.2230.17 ± 0.07
130M20K22.53 / 22.62 / 22.6422.60 ± 0.06
350M60K16.75 / 16.79 / 16.8716.80 ± 0.06

The small variations demonstrate stable performance across random seeds. We explicitly report the different training budgets rather than implying a uniform number of steps. Furthermore, we emphasize that rather than pursuing a substantial performance margin over methods like Fira, our contribution lies in providing a novel perspective, which is evidenced by LoSARE even outperforming the Full-Rank baseline in terms of PPL.

We acknowledge that additional multi-seed fine-tuning experiments would provide further evidence. Nevertheless, our existing results cover two complementary evaluation settings. On OBQA (500 examples), LoSARE is close to the best Fira result, with fewer than 10 fewer correct predictions. On HellaSwag (10,042 examples), LoSARE achieves the highest accuracy, with substantially more correct predictions than Fira. The larger evaluation set provides complementary evidence that the observed advantage is not merely due to a handful of examples. We therefore believe the two benchmarks together support LoSARE's competitive downstream performance, while avoiding an unsupported claim of multi-seed statistical significance. Due to time and computational resource constraints, we supplement multi-seed experiments for pre-training but are unable to do so for fine-tuning. Nevertheless, the multi-seed pre-training results are sufficient to demonstrate the small variance.

Thank you for the additional experiments and clarifications. The rebuttal resolves some concerns but leaves others unanswered and raises questions about fair comparison. For ease of reading I will follow the original points in my review.

W1. Ablations and baseline comparisons

The new ablations partially address this concern. I looked up at response to reviewer YpU8 about the fixed-gain comparison: it shows only a marginal advantage for energy-dependent enhancement, comparable to the reported standard deviation. It may still reduce tuning, but I would not claim a significant performance advantage. Moment reuse across changing bases remains unevaluated, and LoFT is not addressed.

The SubTrack++ comparison uses mismatched settings, as their Table 4 implies total batch sizes of 256/256/128, while LoSARE uses approximately 131K tokens per update. At sequence length 256, your 10K-step checkpoints process 2×/2×/4× as many nominal tokens. SubTrack++ also completes its schedule at 10K steps, whereas your 130M/350M checkpoints come from 20K/60K-step schedules. Learning rates differ too: (10^{-3}) versus (10^{-2}). These configurations are not identical as claimed, and are overall very confusing.

A fair comparison requires reruns with matched budgets and scheduler horizons, preferably reproducing the baselines under your protocol with comparable tuning. Otherwise, the imported results provide context rather than a controlled comparison.

W2. Performance–efficiency comparison

The clarification that all Figure 2 curves use reproduced runs at batch size 128 for 150K steps largely resolves the provenance concern. However, it conflicts with section 5.2’s reference to GaLore’s original 40K-step checkpoint. Please verify a common scheduler horizon and correct the manuscript.

The different-rank GaLore comparison shows a configuration-level trade-off, but does not isolate the tracker’s computational advantage. Whether it persists against baselines with a tuned SVD refresh interval remains unclear.

Q2. Hyperparameter selection

This concerns remains unanswered. Showing that GaLore performs worse at a smaller learning rate actually reinforces that methods may need different settings, and why transferring the hyperparameters does not establish comparable tuning effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.