acceptodds
Under review as a conference paper at ICLR 2027

GrabLoRA: A Grassmannian Framework for Tracking Gradient Subspace Rotation in LoRA

Abstract

Low-rank adaptation (LoRA) is a widely used approach in parameter-efficient fine-tuning, and a recent line of work augments it by initializing LoRA with gradient-informed subspaces. Such one-time placement, however, overlooks the evolution of the gradient space during training. In this work, we investigate two questions: Does the latent gradient subspace rotate as LoRA training proceeds? If yes, then how to keep track of the rotating subspace? To study these problems in a geometry-aware setting, we introduce GrabLoRA, a Bayesian Grassmannian LoRA framework that treats the low-rank subspace as an evolving state rather than a fixed initialization. To realize this, GrabLoRA inherits the information-rich early gradient subspace as a prior, realigns the left and right subspaces on the Grassmann manifold using incoming gradient evidence, and applies an uncertainty-aware preconditioner that balances historical geometric uncertainty with newly observed gradients. Across natural language understanding and generation tasks, GrabLoRA achieves consistent improvements over standard LoRA and gradient-initialized baselines. We also provide theoretical insights for gradient-space coverage and convergence of GrabLoRA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Gradient Subspace Tracking, Uncertainty-Aware Preconditioning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Rqf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

GrabLoRA periodically tracks left and right gradient subspaces through a Grassmannian reconstruction objective with temporal regularization. It combines uncertainty-scaled diagonal coefficients with two tangent-strip corrections. Experiments include six T5-base GLUE tasks, three Llama-3.1-8B tasks, three seeds, component ablations, and rotation/coverage diagnostics. The reported results show improvements over several low-rank adaptation baselines.

Critical strengths
  1. Concrete geometric formulation. Equation (1) balances gradient reconstruction against projector-distance penalties. Each alternating block admits a leading-eigenspace solution. The full tangent projection also yields a useful product bound on left/right tracking errors. [p. 5; Appendix A.3, p. 16; B.1–B.2, pp. 18–22]

  2. Meaningful empirical breadth. The reported GLUE average is 86.47 versus 83.74 for LoRA-One. Learning-rate searches, SubTrack comparisons, five component ablations, and multilayer diagnostics support the method's promise. [Tables 1–4, pp. 7–9; Appendix D.2, pp. 27–28; Appendix E]

Critical weaknesses
  1. The cumulative model class is unspecified. Equation (3), Algorithms 1–2, and Eq. (A.15) add proposals directly to previous weights without a stated retraction or truncation ensuring rank⁡(Wt−W0)≤r\operatorname{rank}(W_t-W_0)\le r. With tracking rank one, a core and two tangent strips can already produce
\qquad \det(\Delta W)=-ab,$$ which has rank two when $ab\ne0$. Repeated additive updates can further increase cumulative rank. The manuscript therefore needs to distinguish compressed optimizer state from a bounded-rank task adapter and account for the resulting capacity and storage. Peak GPU memory alone does not settle that comparison. [pp. 5–6; Table 3, p. 8; Algorithm 2, p. 17; Eq. (A.15), p. 18] 2. Full tangent coverage overstates the specified update space. At a non-refresh step, let $U=V=[e_1,e_2]\subset\mathbb R^3$ and $G=e_1e_2^T+e_2e_1^T$. Then $$\Pi_T(G)=P_UG+GP_V-P_UGP_V=G, \qquad u_k^TGv_k=0.$$ Coverage is one, but both strips vanish. With zero stored first moment and finite positive denominators, the diagonal implementation therefore produces zero update. The main text appropriately calls its descent theorem idealized; that theorem cannot establish the behavior of this smaller update space. [coverage definition, p. 9; Eqs. (A.7), (A.10)–(A.14), pp. 16–18; Appendix B.3, p. 23]
Questions for authors
  1. What is trained and saved in the evaluated implementation? Please provide its cumulative parameterization, task-delta size, numerical rank, optimizer memory, and matched runtime. Does existing evidence establish a fair resource–quality comparison with rank-rr adapters?

  2. How are off-diagonal core gradients handled between refreshes? Please resolve the example and report coverage of the actual update space. If full-core retention or per-step re-diagonalization was already implemented, identify the omitted operations and confirm which reported results use them.

2 comments
AuthorsOpenReview ↗

Reply to W1 and Q1

We really appreciate all your keen observations about our paper and the insightful questions! Below we report more details of the paper's experiment results, and conduct more quantification experiments regarding your concerns.

What is trained and Saved in LoRA

ComponentStored tensorsScalars per layerT5-baseLlama-8B
GrabLoRA
Optimizer momentsμ,P\mu,P2r2r1.5 KiB2.0 KiB
ParametersU,V,mU,V,mr(dout+din)+rr(d_{\text{out}}+d_{\text{in}})+r1.13 MiB6.50 MiB
Bookkeepingρ,σ0,β\rho,\sigma_0,\beta3r3r2.25 KiB3.0 KiB
Totalr(dout+din)+6rr(d_{\text{out}}+d_{\text{in}})+6r1.13 MiB6.51 MiB
LoRA
Optimizer momentsAdam m,vm,v on A,BA,B2r(dout+din)2r(d_{\text{out}}+d_{\text{in}})2.25 MiB13.0 MiB
ParametersA,BA,Br(dout+din)r(d_{\text{out}}+d_{\text{in}})1.12 MiB6.50 MiB
Total3r(dout+din)3r(d_{\text{out}}+d_{\text{in}})3.38 MiB19.5 MiB

GrabLoRA keep UU and VV, so the storage is essentially the same as LoRA. For optimizer memory, GrabLoRA replace Adam second order moment denominator with a per-direction uncertainty estimate, therefore, which shrink the optimizer state.

cumulative parameterization and terminology

We thank the reviewer for this point. GrabLoRA is more accurately described as low-rank subspace optimization with a compressed optimizer state rather than as a bounded rank adapter in the strict LoRA sense. We used the LoRA name because the method builds on LoRA's core idea of updating on low-dimensional subspace. We'll make this distinction explicit in the revision and position the method accordingly.

The cumulative parameterization is WT−W0=η∑t=0T−1ΔWt=η∑t=0T−1[UtCtVt⊤+(I−PtU)YˉtPtV+PtUYˉt(I−PtV)]W_T -W_ 0 =\eta \sum_{t=0}^{T-1} \Delta W_t = \eta \sum_{t=0}^{T-1} [U_t C_t V_t^\top +(I-P_t^U)\bar Y_t P_t^V+P_t^U\bar Y_t (I-P_t^V)] Each step's update lives in low-dimensional left UU and right VV subspace. Because the subspaces are refreshed during training, the cumulative update is not limited to rank r=8r=8. While keeping the same parameter storage as LoRA, GrabLoRA extend the rank capacity [1]. To measure this, we track the task-delta size (rank and magnitude) of ΔWt=Wt−W0\Delta W_t = W_t-W_0 on GLUE SST2 with T5 base (d=768d=768) model.

task-delta size, numerical rank

steprank 1% (q) [min-max]rank 1% (v) [min max]∥ΔW∥F(q)\| \Delta W\|_F (q)∥ΔW∥F(v)\| \Delta W\|_F (v)
1000326 [89–533]276 [54–636]2.4280.5305
2000388 [90–562]272 [43–641]2.5910.5985
3000393 [90–564]270 [43–641]2.5980.6012

Ths indicates the subspace tracking explores a much richer update space than a fixed rank-rr adapter while the training state stays at LoRA scale.

running time

MethodGSM8KHumanEvalMMLUAvg. GLUE
LoRA73151779991355
GrabLoRA8738 (1.19×)2047 (1.15×)1183 (1.19×)413 (1.16×)
LoRA-One7840 (1.07×)1776 (1.00×)982 (0.99×)362 (1.02×)
RoSA8496 (1.16×)1945 (1.09×)1003 (1.01×)403 (1.14×)
SubTrack7351 (1.00×)1860 (1.05×)880 (0.89×)396 (1.12×)

GrabLoRA adds 15-19% wall clock time over LoRA, which is comparable to RoSA. Here we address the concern of the potential overhead of GrabLoRA mechanism by two kinds of work. Between refreshes, each step updates only the rr directional coefficients, which is cheap. The more expensive subspace refreshes happen only periodically. We will report these timing in the revision and state the overhead.

Finally, considering the expressiveness of GrabLoRA with same scale of memory as LoRA, and the accuracy gain in Table 1 and 2, we believe the comparison of GrabLoRA and other LoRA variants, as well as the tradeoff of computation and quality are reasonable. We are grateful to reviewer prompting these analyses.

[1] Zhang, F., & Pilanci, M. (2024). Spectral Adapter: Fine-Tuning in Spectral Space. arXiv preprint arXiv:2405.13952.

AuthorsOpenReview ↗

Reply to W2 and Q2

Influence of non-refresh off-diagonal core

We appreciate the abstract example for illustration. In that example, the core gradient is zero and the informative signal lies entirely in the off-diagonal components, which our setting discards. Noted, the numerator of the coverage defined in the paper: ΠT(G)=PUG+GPV−PUGPV\Pi_T(G) = P_U G + G P_V - P_U G P_V measures how well the UU and VV subspaces themselves capture the gradient, as the quality of subspace we identify, which does not show how well the actual updated weight cover the gradient. There is a gap between the real coverage of update weight on gradient, as Theorem5.3 first part proof need, and the claimed coverage in our definition. Thank you for pointing it out!

To measure this gap directly, we conduct quantification on GLUE COLA, comparing the claimed coverage with the real coverage (∥UGVT+strips∥F2\Vert UGV^T + \textbf{strips}\Vert _F^2 as numerator). We report the median across layers, averaged over training steps.

Notation:

  • Gradient projection onto UU and VV space: Ct=UtYˉtVt⊤C_{t} = U_{t} \bar Y_{t} V_t^\top for every step (especially non-fresh step).

  • Strips: St=PUYˉt(I−PV)+(I−PU)YˉtPVS_t = P_U \bar Y_t (I-P_V) + (I-P_U)\bar Y_t P_V.

  • Claimed coverage: ∥ΠT(Yˉt)∥F2∥Yˉt∥F2=∥UtCtVt⊤+St∥F2∥Yˉt∥F2\dfrac{\lVert \Pi_T(\bar Y_t) \rVert_F^2}{\lVert \bar Y_t \rVert_F^2} = \dfrac{\lVert U_t C_t V_t^{\top} + S_t \rVert_F^2}{\lVert \bar Y_t \rVert_F^2} [P9 row 449].

  • Real coverage: ∥Utdiag⁡(Ct)Vt⊤+St∥F2∥Yˉt∥F2\dfrac{\lVert U_t \operatorname{diag}(C_t) V_t^{\top} + S_t \rVert_F^2}{\lVert \bar Y_t \rVert_F^2}.

  • Gap = claimed coverage - real coverage=∥Utoffdiag⁡(Ct)Vt⊤∥F2∥Yˉt∥F2\dfrac{\lVert U_t \operatorname{offdiag}(C_t) V_t^{\top} \rVert_F^2}{\lVert \bar Y_t \rVert_F^2}.

COLAτ=1\tau=1 (per-step rediag)τ=20\tau = 20 (paper setting)τ=100\tau = 100
claimed coverage0.870.740.73
real coverage0.870.730.72
gap-0.000.010.01
accuracy81.0079.5079.10
wall-clock time (s)694317300

Empirically, discarding the off-diagonal core costs little -- the gap is small (≤0.02\leq 0.02) across all steps, which is supported by both observations and our design. First, the off-diagonal core carries only a small share of the gradient energy (different from the Reviewer's example). Second, we rediagonalize every τ=20\tau=20 steps, keeping the basis aligned with the gradient, which keeps the off-diagonal components that are discarded between refreshes from building up.

Reason for nonrefresh steps

Another reason for choosing refresh interval is balancing between cost and performance. Rediagonalizing at every step hugely increase the wall-clock time while relaxing the refresh interval larger (τ=100\tau=100) saves time, but loosen the rotation and accuracy. Therefore, we pick the moderate interval to balance between accuracy and computation overhead, as a cost of discarding the off diagonal gradient core, as your example shows.

We'll clearly clarify the distinction between subspace coverage and update coverage in the paper statement, and appendix theorem proof part.


We hope this response addresses the reviewer’s concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification if needed.

Reviewer_7qg8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes the GrabLoRA framework to address the issue of continuous evolution of gradient subspace directions during the LoRA fine-tuning stage. This framework models the left and right low-rank subspaces of LoRA as dynamically evolving states on the Grassmann manifold, and continuously aligns the subspaces using new gradient evidence during the training process. To address the issue of "subspace rotation causing the Adam optimizer to fail to effectively scale gradients", this work replaces Adam's empirical second-order moments with rotation-aware uncertainty estimation that dynamically adjusts the update step size for each direction according to the degree of subspace rotation.

Critical strengths

1.The manuscript focuses on the core issue of "continuous evolution in the direction of gradient subspace" during the LoRA fine-tuning stage, with a clear research motivation. It holds significant theoretical and practical implications for understanding and improving the dynamic adaptation mechanism of LoRA.

2.The proposed GrabLoRA framework models the gradient subspace as a dynamically evolving state on the Grassmann manifold and utilizes new gradient evidence for continuous alignment. This method effectively alleviates the mismatch between the direction of the gradient subspace and the dominant gradient direction in traditional methods during the later stages of training.

3.The manuscript keenly identifies the derivative problem of "subspace rotation leading to the failure of the empirical second-order moment of the Adam optimizer" and proposes a significant innovation by replacing traditional second-order momentum with rotation-aware uncertainty estimation to solve this problem.

4.The manuscript experiment is relatively complete, fully verifying the effectiveness of the proposed method. The theoretical analysis significantly enhances the depth of the research.

Critical weaknesses

1.Regarding the phenomenon of "LoRA fine-tuning stage gradient subspace rotation", some preliminary work such as [1] has already explored it. Therefore, using this phenomenon as the contribution of this paper may face questions about originality. It is recommended that the author rethink the motivation of this paper, clarify the essential differences between this paper and the aforementioned work.

[1] Piontkovskaia, I., & Nikolenko, S. (2026), First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers ,arXiv:2607.16821

2.The Related Work section of the manuscript mainly focuses on classic dynamic subspace tracking (GROUSE, SubTrack) and LoRA initialization/structural variants, failing to fully cover these paper discussing gradient subspace rotation. The author should conduct a more thorough investigation of existing research and supplement with corresponding references.

3.The methods compared in the main experiments of the manuscript (Tables 1 and 2) are quite limited, and some methods mentioned in related work have not been included in the comparison (such as AdaLoRA and PiSSA). In addition, the author should also compare strong baselines such as LoRA+ and DoRA to verify the progressiveness of the methods.

4.The theory seems somewhat hollow and disconnected from the algorithm. The greatest innovation of this manuscript lies in the "rotation-aware uncertainty preconditioner". Although Theorem 5.3 provides a proof of the algorithm's convergence, it merely states that "as long as the step size is bounded, gradient descent will converge", without offering sufficient support for the benefits of the core innovation of this paper.

Questions for authors

Refer to the Weaknesses.

2 comments

We thank the reviewer for taking the time to provide us with many suggestions to improve the clarity and presentation of the paper.

Originality and Motivation

Thank you for highlighting this connection. We completely agree the gradient subspace rotation is not a new observation [1,2]. Rather than claiming a new finding, we aim to empirically prove the existence of rotation during training, and naturally introduce the rotation-guided lora framework. [1] directly uses the existence of rotation in training and aim for a more efficient training model, while [2] as a diagnostic of rotation, reject the hypothesis of training lies in a fixed space. GrabLoRA is a empirically motivated training framework.

Moreover, our rotation analysis examines geometric structure. GrabLoRA analyze fixed r=8r=8 rank left and right subspace from SVD, and measure the principal angle between subspace (Ut,Ut−τ)(U_t, U_{t-\tau}) and (Vt,Vt−τ)(V_t, V_{t-\tau}). This also reveals the asymmetry of left and right subspace rotation during training. In contrast, [2] analyze the top10 task-gradient subspaces (described in P8,11,33 as gradient-covariance subspaces). Although overlap and principal angles are equivalent, the underlying geometric objects differ.

Overall, both studies achieve consensus on the rotation of gradient subspace. GrabLoRA further translate this observation into a training model, which is our primary motivation and intuition.

[1] Rajabi, S., Nonta, N., & Rambhatla, S. (2025). SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training.

[2] Piontkovskaia, I., & Nikolenko, S. (2026). First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers.

related work of gradient subspace rotation

Thank you for pointing out the gap in the current related work. We've added extented related work in appendix F (see revised PDF), where we add two more discussions on subspace rotation quantification, and geometry induced methods in prior work.

More baselines

Follow the suggestion of the reviewer, we conduct more baseline methods on GLUE tasks, and get the following result (averaged over 3 seeds as setting same in experiment). These experiments confirm that GrabLoRA outperforms the extra baselines.

MethodCoLAMRPCQNLIQQPRTESST-2Avg
PiSSA77.7084.3190.4388.1375.8191.7884.69
AdaLoRA71.7385.2189.7789.3375.0989.6483.46
LoRA+76.1383.6681.9080.7378.2288.4281.51
DoRA72.7784.5691.1089.0776.6592.7884.49
GrabLoRA79.5087.1792.0189.7076.7793.6986.47

Theory where preconditioner enters

We thank the reviewer for going into the details of the proof. Theorem 5.2 answers how the U and V space identified in GrabLORA capture the gradient, while Theorem 5.3 proves when coverage is close enough (see reviewer Rqf8 W2 Q2 for "subspace coverage" vs "update coverage" distinction), with the learning rate under certain bound, our algorithm will converge. The bound term 0<η≤d−[1−(etuetV)2]Ld+20< \eta\leq \frac{d_-[1-(e_t^ue_t^V)^2]}{Ld_+^2} is where the preconditioner enters the theorem.

Recall the definition dt,k=1st,k+ϵd_{t,k} = \frac{1}{\sqrt{s_{t,k}}+\epsilon} [Def B.11], where st,k=Pt,k−+Rt,ks_{t,k}=P_{t,k}^- + R_{t,k} combines the uncertainty propagated from the past, the rotation induced uncertainty and sample noise uncertainty. d−≤min⁡1,dt,1,…,dt,rd_- \leq \min\\{1,d_{t,1},\dots,d_{t,r}\\} gives a conservative lower bound on how strongly the update uses the gradient, while d+≥max⁡1,dt,1,…,dt,rd_+ \geq\max\\{1, d_{t,1},\dots, d_{t,r}\\} represents the largest possible movement. If the uncertainty is small on direction kk, we take larger step on that direction (dt,kd_{t,k} is large, therefore d+d_+ is large, the progress on that direction is amplified). If the uncertainty is large on direction jj, we take smaller step on that direction (dt,jd_{t,j} is small, therefore d−d_- is small, the progress on that direction is small). In this case, the loss decrease slow (and may increase) : L(Wt+1)−L(Wt)≤−η d−[1−(etUetV)2]∥Gt∥F2+Lη2d+22∥Gt∥F2 [P29 row 1305]\mathcal{L}(W_{t+1}) - \mathcal{L}(W_t) \leq -\eta\ d_- \left[1 - (e_t^U e_t^V)^2\right] \Vert G_t \Vert_F^2 + \frac{L \eta^2 d_+^2}{2} \Vert G_t \Vert_F^2\ \text{[P29 row 1305]} On the right handside, the first term (with d−d_-) is the loss descent, while the second term (with d+d_+) can be seen as the penalty induced by large steps therefore large curvature. By choosing 0<η≤d−[1−(etuetV)2]Ld+20< \eta\leq \frac{d_-[1-(e_t^ue_t^V)^2]}{Ld_+^2} , η\eta serves to ensure the penalty cannot overwhelm the loss decrease.

We hope this response addresses the reviewer’s concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification if needed. If the concerns are resolved, we would be grateful if the reviewer considers updating their score accordingly.

Thank you for the authors' response. Regarding the first issue, although I find part of the authors' argument reasonable, they have not clearly submitted revisions to the manuscript. Given the seriousness of this issue and the fact that addressing it would require substantial changes to the manuscript (including the abstract, introduction, contributions, and conclusion), I can only maintain my original view for now. If the authors provide a revised manuscript, I will reconsider this point. As for issue 5, I still find the authors' explanation rather unconvincing. Based on the above two points, I have decided to keep my score of 2 for the time being.

Reviewer_D87ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether the task-relevant gradient subspace changes during LoRA fine-tuning and proposes GrabLoRA to track that change. The method updates left and right low-rank subspaces on Grassmann manifolds using current gradient evidence and a continuity term, then adapts coefficient updates with an uncertainty-aware preconditioner. The paper also introduces a gradient-subspace coverage analysis and gives bounds on tracking error and descent under stated assumptions. Experiments cover six GLUE tasks with T5-base and GSM8K, HumanEval, and MMLU with Llama-3.1-8B.

Critical strengths
  1. The paper identifies a meaningful weakness of one-time gradient-informed initialization: its useful directions may become misaligned as fine-tuning proceeds.
  2. Tracking a subspace rather than a particular basis is a natural geometric choice. The combination of temporal continuity, new gradient evidence, and uncertainty-aware updates is a coherent method design, and the ablations attempt to isolate several components.
  3. The evaluation includes both language understanding and generation, multiple initialization and subspace-tracking baselines, and three-seed results. The reported gains are strongest across the GLUE tasks, while the generation results are competitive and the reported memory use is close to LoRA baselines.
Critical weaknesses
  1. Although the paper introduces the standard LoRA parameterization, it remains unclear whether GrabLoRA actually employs trainable LoRA factors A and B or directly updates the weight matrices. The authors should clarify the architecture and trainable parameters used in GrabLoRA.
  2. The empirical evidence for generality is more limited than the paper suggests. The GLUE evaluation uses T5-base, while the generation evaluation uses one backbone (Llama-3.1-8B), three tasks, and updates only query/value projections. It is unclear how well the method transfers to other model families, LoRA ranks, and additional modules, such as MLP weight matrices.
  3. The computational overhead is not sufficiently evaluated. GrabLoRA requires gradient evidence collection, periodic subspace rotation, SVD or eigenspace computations, and uncertainty estimation. However, Table 3 only reports the training time of GrabLoRA without providing the corresponding training time of standard LoRA. Although peak GPU memory is comparable, this does not establish that the proposed method is computationally efficient. A direct comparison of training time, per-step latency, and subspace refresh overhead is necessary.
  4. The necessity of the uncertainty-aware preconditioner is not sufficiently demonstrated. Although the ablation study shows a substantial performance drop when the preconditioner is removed, it remains unclear whether the proposed uncertainty modeling is necessary or whether simpler alternatives could achieve similar improvements. For example, standard Adam, resetting or transporting momentum after subspace rotation, and simple rotation-aware scaling could also address the mismatch between historical optimizer statistics and evolving gradient directions.
  5. The motivation and design of the uncertainty-aware preconditioner are not sufficiently justified. The authors argue that subspace rotation makes historical optimizer statistics unreliable, which is a reasonable motivation. However, a larger rotation does not necessarily imply greater uncertainty, as a substantially rotated gradient direction may still be estimated reliably. Moreover, it is unclear why greater rotation should lead to more conservative updates, since rapid changes in gradient directions may instead require faster adaptation. The paper does not sufficiently justify these assumptions underlying the proposed uncertainty formulation.
Questions for authors
  1. Does GrabLoRA preserve the standard LoRA parameterization?
  2. How well does GrabLoRA generalize to other models and adaptation settings?
  3. What is the actual computational overhead of GrabLoRA compared with standard LoRA?
  4. Why should larger subspace rotation imply greater uncertainty and more conservative updates?
4 comments
AuthorsOpenReview ↗

Reply to W4, W5, Q4

We thank the reviewer for recognizing our motivation of carrying part of the past geometry forward by ρt,k2Pt−1,k+\rho_{t,k}^2P_{t-1,k}^+, which is one central design of GrabLoRA, as similar to Subtrack, SHAMPOO, SOAP[1,2,3]. Below, we’d like to share our reasoning for the remaining two parts: q0+q1(1−ρt,k2)q_0+q_1(1-\rho_{t,k}^2) and RR.

Recall the denominator st,k=Pt,k−+Rt,k=ρt,k2Pt−1,k++q0+q1(1−ρt,k2)+(ntβt,k)−1s_{t,k} = P_{t,k}^-+R_{t,k} = \rho_{t,k}^2 P_{t-1,k}^+ +q_0+q_1(1-\rho_{t,k}^2) + (n_t \beta_{t,k})^{-1} which follows the state model ct,k=ρt,kct−1,k+wt,kc_{t,k} = \rho_{t,k}c_{t-1,k}+w_{t,k} [A.2]. Each coefficient ct,kc_{t,k} is tracked over time like Kalman filtering. The previous posterior (which is carried forward through the rotation ρt,k\rho_{t,k}) forms the prior at time tt, and the incoming minibatch evidence then update it into a new posterior. The denominator st,ks_{t,k} therefore combines how uncertain the carried over belief is (Pt,k−P_{t,k}^-) with how noisy the new evidence is (Rt,kR_{t,k}).

The q0+q1(1−ρt,k2)q_0+q_1(1-\rho_{t,k}^2) part:

Motivation: Our primary motivation and intuition are similar to the idea of ReLoRA [4] (where after merging the new weight, the learning rate is set to zero, giving the optimizer a few small steps to settle into the new geometry) and SwitchLoRA [5] (which similarly reset the optimizer state and freezes affected parameters when parameters are switched). GrabLoRA applies the similar idea: q0+q1(1−ρt,k2)q_0+q_1(1-\rho_{t,k}^2) grows with the amount of rotation, so a direction that has just rotated takes a moderately smaller step while its new statistics build up.

Relationship of rotation and uncertainty: Fira [6] also observes and identifies subspace switching is a cause of the instability. Our empirical observation agree with this claim. In Figure4, for the weakly rotated directions (Q1Q1), the inherited history ρ2P+\rho^2 P^+ is still a major part of the denominator. As rotation grows, its share decreases and the denominator is dominated by the observation noise from the incoming evidence. In other words, large rotations tend to occur together with noisier, less stable evidence. This observation is consistent with prior work.

At the same time, we fully agree with the reviewer that a strongly rotated direction may still be estimated reliably. To address this concern, we therefore empirically ablated the rotation-related terms (q1=0q_1=0 and q1=q0=0q_1=q_0=0). Result shows that without q1q_1, without q0q_0 and q1q_1, the average performance on GLUE have a average on GLUE will drop, indicating the necessity of the design.

methodcolamrpcqnliqqprtesst2Avg
GrabLoRA79.5087.1792.0189.7076.7793.6986.47
q1=q0=0q_1=q_0=068.5068.7160.3089.2055.3554.8566.15
q1=0q_1=076.7082.1189.6083.9074.7391.1783.04
R=0R=069.9370.4264.5089.5755.7253.4867.27

The RR part:

Structurally, RR comes from the Bayesian filtering view, which treats each minibatch as a noisy observation and keeps a Kalman-filter model of the evolution of the unknown parameters [7,8]. Under this view, the filter updates both its belief PP and the observation noise RR, which is also where the reviewer’s case is handled. When the evidence along a rotated direction has low variability, RR is small and the direction takes a relatively large step. In this sense, RR acts as a measure of how reliable each rotated direction currently is.

Beyond this structural interpretation, we also ablate RR (see table above). Removing it causes a substantial drop in performance (from 86.47 to 67.27 on average), indicating the incoming evidence is important.

Alternative preconditioner

Following the suggestions from reviewer, we conduct further experiment with alternative preconditioners the reviewer proposed:

methodcolamrpcqnliqqprtesst2Avg
GrabLoRA79.5087.1792.0189.7076.7793.6986.47
Adam76.5085.7888.0086.7072.5689.1183.11
resetting71.7084.3189.8085.9077.2691.8683.47
rotation only70.4069.3665.8088.5050.1852.1866.07

where rotation only is by setting R=q1=q0=0R=q_1=q_0=0. Result shows design of preconditioner in GrabLoRA outperforms other alternatives.

Taken together, more experiments guided by Reviewer makes our work more complete. We'll add the above justification, as well as the additional experiments in the draft.

AuthorsOpenReview ↗

Reply to W1 Q1, W2 Q2, W3 Q3.

Thank you for taking time to provide meaningful insights for our work!

Parameterization (W1, Q1)

Thank you for asking about the parameterization, which is also raised by Reviewer Rqf8. GrabLoRA does not train LoRA factors AA and BB. Instead, it keeps rank-rr left and right subspaces UU and VV, similar to the SVD-style LoRA variants structure, together with coefficients on the core between them. At each step, the gradient is projected onto these subspaces, the preconditioner rescale the step size of each direction and the result low-rank update is applied to the weight matrix.

Memory: storing UU and VV takes about the same memory as LoRA's AA and BB, while the optimizer state is much smaller since the uncertainty-aware preconditioner keeps only a few scalars per direction, instead of Adam's two moments for every entry of AA and BB. Please see respond to Reviewer Rqf8 W1 Q1 for the detailed memory comparison.

Rank: Each individual update is low-rank, but the actual accumulated update is not, so GrabLoRA is not a LoRA in the strict sense.

Since it keeps a LoRA-scale memory and the low-rank structure of each update, we still use the LoRA terminology and will clarify this distinction in the revision.

More epxeriments (W2, Q2)

Following the reviewer's suggestion, we conduct more experiments to examine how well GrabLoRA generalize across settings. Considering the time, we focus on GLUE task.


Model family: apart from the encoder-decoder T5 base model in the paper, we conduct experiments on encoder only model RoBERTa, and decoder only model Qwen-7B GLUE tasks.

RoberTaCoLAMRPCQNLIQQPRTESST-2Qwen7BCoLAMRPCQNLIQQPRTESST-2
LoRA79.4087.2578.5083.5075.8192.66LoRA81.7087.2589.4087.2089.8994.84
GrabLoRA81.0089.2283.3085.5077.2692.78GrabLoRA83.8088.4892.0089.7090.6195.64

Rank: we extend our work from r=8 to r=32 and r=128 for further comparison.

GrabLoRACoLAMRPCQNLIQQPRTESST-2
r=879.5087.1792.0189.7076.7793.69
r=3280.7090.2089.9089.0077.5092.74
r=12881.0090.4489.8788.1771.4893.00

Modules: different from the paper where GrabLoRA is applied on Query and Value, here we apply to all linear models, including the up, down, gate in MLP.

GrabLoRACoLAMRPCQNLIQQPRTESST-2
all linear layer81.0087.3490.8089.8776.9092.89

Across model families, ranks and target modules, GrabLORA performs stably on most tasks, which suggests that the transferability of GrabLoRA.

Computation Overhead (W3, Q3)

Thank you for pointing out the missing runtime comparison with LoRA, which Reviewer Rqf8 also raised. We acknowledge that GrabLoRA adds computational overhead. To keep it limited, we perform the full subspace refresh only every τ=20\tau=20 steps, which balance the cost and performance (see Table in Response to Reviewer Rqf8 W2 Q2). The overall training times over 3000 steps compared with LoRA and other baselines is given in our response to Reviewer Rqf8 W1 Q1. Overall, GrabLORA takes 1.1.5-1.19$\times$ the training time of LoRA, which is comparable to ROSA. To show a detailed comparison of computation overhead per step, we report it in the below, using GLUE SST2 as an example):

stepcalculationtime(s)
GrabLoRA refresh stepaverage over 150 refreshes0.841
GrabLORA nonrefresh stepaverage over 2850 the rest steps0.114
LoRA stepaverage over 3000 steps0.127

On nonrefresh steps, GrabLoRA is slightly faster than LoRA (0.90$\times$ per step), while the refresh steps are expensive, about 7.4 ×\times slower than nonrefresh steps. Because a refresh occurs only once every 20 steps, the faster nonrefresh steps in between make up for part of its cost. Overal, this gives 1.18$\times$ the training time of LoRA. Also, this refresh does not hurt accuracy (see the τ\tau comparison in our response to Reviewer Rqf8 W2 Q2).

References in Reply to W4, W5, Q4

[1] Rajabi, S., Nonta, N., & Rambhatla, S. (2025). SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training.
[2] Gupta, V., Koren, T., & Singer, Y. (2018). Shampoo: Preconditioned Stochastic Tensor Optimization.
[3] Vyas, N., Morwani, D., Zhao, R., Kwun, M., Shapira, I., Brandfonbrener, D., … Kakade, S. (2025). SOAP: Improving and Stabilizing Shampoo using Adam.
[4] Lialin, V., Muckatira, S., Shivagunde, N., & Rumshisky, A. (2024). ReLoRA: High-Rank Training Through Low-Rank Updates.
[5] Zhou, K., Wang, S., & Xu, J. (2025). SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information.
[6] Chen, X., Feng, K., Li, C., Lai, X., Yue, X., Yuan, Y., & Wang, G. (2025). Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
[7] Ollivier, Y. (2018). Online Natural Gradient as a Kalman Filter.
[8] Davtyan, A., Sameni, S., Cerkezi, L., Meishvilli, G., Bielski, A., & Favaro, P. (2025). KOALA: A Kalman Optimization Algorithm with Loss Adaptivity.


In summary, we thank the reviewer for their important questions. We will revise the manuscript accordingly. If further experimentation or clarification is warranted, we would be happy to provide it. If the concerns are resolved, we would be grateful if the reviewer considers updating their score accordingly.

Thank you for your detailed response and additional experiments. I appreciate the authors' efforts in addressing my concerns regarding the parameterization, computational overhead, and the necessity and motivation of the uncertainty-aware preconditioner. These concerns have been largely resolved by the rebuttal. However, I still have concerns about the generalizability of the proposed method. Although the additional experiments are helpful, I believe comparisons with more competitive baselines beyond standard LoRA across these models and adaptation settings would further strengthen the empirical evidence. Overall, I believe the experimental evaluation should be further expanded to provide more comprehensive validation of the proposed method. Therefore, I will maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for all the reviewers' efforts in helping improve our work!

We thank all reviewers for their constructive feedback, which significantly strengthened this work. During the rebuttal and discussions, all concerns are discussed, and all the reviewers maintain the score. For your reference, we summarize the major discussion points and paper improvements.

Parameterization and computation overhead. We provide a detailed summary table reporting GrabLoRA memory cost (similar to LoRA memory), as well as the computation overhead (1.15-1.19$\times$ LoRA wall lock time, relative to ROSA). (Rqf8, D87Z)

Necessity of uncertainty-aware preconditioner: Beyond the reasoning from the intuition of the design, we further conduct a thorough ablations on the components of the preconditioner. The decrease of performance further proves the necessity of our design. (D87Z)

Robustness and generalizability. Under the reviewers 7qg8 D87Z suggestions, we further compare GrabLoRA with more baselines, on different LLM models, different transformer modules, and different ranks. Under these cases, GrabLoRA shows its robustness and performance over other baselines.

Originality and related work: GrabLoRA is motivated by the empirical evidence of subspace rotation (principal angle), where the observation of subspace rotation are also identified by prior works through different quantification methods on different objects (rank, energy, overlap). Apart from the observation, GrabLoRA further put forward a geometric-aware method in LoRA, which serves the different aim from prior works. (7qg8) We further discussed the position of this paper in "gradient susbpace rotation" area, by adding the related work part in revised Appendix F (P30). (7qg8)

Theory mismatch: GrabLoRA's uncertainty-aware preconditioner enters the theorem through rescaling the step size of each direction, which further decides the bound for weight update learning rate. (7qg8) During non-refreshing steps, the unchanged UU and VV spaces causes the gap between discarding the off-diagonal core matrix, with the full core matrix. We appreciate reviewer Rqf8 pointing it out, and further empirically quantify the gap, which is small (≤0.02\leq 0.02).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.