- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
GrabLoRA periodically tracks left and right gradient subspaces through a Grassmannian reconstruction objective with temporal regularization. It combines uncertainty-scaled diagonal coefficients with two tangent-strip corrections. Experiments include six T5-base GLUE tasks, three Llama-3.1-8B tasks, three seeds, component ablations, and rotation/coverage diagnostics. The reported results show improvements over several low-rank adaptation baselines.
Critical strengths
-
Concrete geometric formulation. Equation (1) balances gradient reconstruction against projector-distance penalties. Each alternating block admits a leading-eigenspace solution. The full tangent projection also yields a useful product bound on left/right tracking errors. [p. 5; Appendix A.3, p. 16; B.1–B.2, pp. 18–22]
-
Meaningful empirical breadth. The reported GLUE average is 86.47 versus 83.74 for LoRA-One. Learning-rate searches, SubTrack comparisons, five component ablations, and multilayer diagnostics support the method's promise. [Tables 1–4, pp. 7–9; Appendix D.2, pp. 27–28; Appendix E]
Critical weaknesses
- The cumulative model class is unspecified. Equation (3), Algorithms 1–2, and Eq. (A.15) add proposals directly to previous weights without a stated retraction or truncation ensuring . With tracking rank one, a core and two tangent strips can already produce
Questions for authors
-
What is trained and saved in the evaluated implementation? Please provide its cumulative parameterization, task-delta size, numerical rank, optimizer memory, and matched runtime. Does existing evidence establish a fair resource–quality comparison with rank- adapters?
-
How are off-diagonal core gradients handled between refreshes? Please resolve the example and report coverage of the actual update space. If full-core retention or per-step re-diagonalization was already implemented, identify the omitted operations and confirm which reported results use them.
2 comments
Reply to W2 and Q2
Influence of non-refresh off-diagonal core
We appreciate the abstract example for illustration. In that example, the core gradient is zero and the informative signal lies entirely in the off-diagonal components, which our setting discards. Noted, the numerator of the coverage defined in the paper: measures how well the and subspaces themselves capture the gradient, as the quality of subspace we identify, which does not show how well the actual updated weight cover the gradient. There is a gap between the real coverage of update weight on gradient, as Theorem5.3 first part proof need, and the claimed coverage in our definition. Thank you for pointing it out!
To measure this gap directly, we conduct quantification on GLUE COLA, comparing the claimed coverage with the real coverage ( as numerator). We report the median across layers, averaged over training steps.
Notation:
-
Gradient projection onto and space: for every step (especially non-fresh step).
-
Strips: .
-
Claimed coverage: [P9 row 449].
-
Real coverage: .
-
Gap = claimed coverage - real coverage=.
| COLA | (per-step rediag) | (paper setting) | |
|---|---|---|---|
| claimed coverage | 0.87 | 0.74 | 0.73 |
| real coverage | 0.87 | 0.73 | 0.72 |
| gap | -0.00 | 0.01 | 0.01 |
| accuracy | 81.00 | 79.50 | 79.10 |
| wall-clock time (s) | 694 | 317 | 300 |
Empirically, discarding the off-diagonal core costs little -- the gap is small () across all steps, which is supported by both observations and our design. First, the off-diagonal core carries only a small share of the gradient energy (different from the Reviewer's example). Second, we rediagonalize every steps, keeping the basis aligned with the gradient, which keeps the off-diagonal components that are discarded between refreshes from building up.
Reason for nonrefresh steps
Another reason for choosing refresh interval is balancing between cost and performance. Rediagonalizing at every step hugely increase the wall-clock time while relaxing the refresh interval larger () saves time, but loosen the rotation and accuracy. Therefore, we pick the moderate interval to balance between accuracy and computation overhead, as a cost of discarding the off diagonal gradient core, as your example shows.
We'll clearly clarify the distinction between subspace coverage and update coverage in the paper statement, and appendix theorem proof part.
We hope this response addresses the reviewer’s concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification if needed.
Reply to W1 and Q1
We really appreciate all your keen observations about our paper and the insightful questions! Below we report more details of the paper's experiment results, and conduct more quantification experiments regarding your concerns.
What is trained and Saved in LoRA
GrabLoRA keep U and V, so the storage is essentially the same as LoRA. For optimizer memory, GrabLoRA replace Adam second order moment denominator with a per-direction uncertainty estimate, therefore, which shrink the optimizer state.
cumulative parameterization and terminology
We thank the reviewer for this point. GrabLoRA is more accurately described as low-rank subspace optimization with a compressed optimizer state rather than as a bounded rank adapter in the strict LoRA sense. We used the LoRA name because the method builds on LoRA's core idea of updating on low-dimensional subspace. We'll make this distinction explicit in the revision and position the method accordingly.
The cumulative parameterization is WT−W0=η∑t=0T−1ΔWt=η∑t=0T−1[UtCtVt⊤+(I−PtU)YˉtPtV+PtUYˉt(I−PtV)] Each step's update lives in low-dimensional left U and right V subspace. Because the subspaces are refreshed during training, the cumulative update is not limited to rank r=8. While keeping the same parameter storage as LoRA, GrabLoRA extend the rank capacity [1]. To measure this, we track the task-delta size (rank and magnitude) of ΔWt=Wt−W0 on GLUE SST2 with T5 base (d=768) model.
task-delta size, numerical rank
Ths indicates the subspace tracking explores a much richer update space than a fixed rank-r adapter while the training state stays at LoRA scale.
running time
GrabLoRA adds 15-19% wall clock time over LoRA, which is comparable to RoSA. Here we address the concern of the potential overhead of GrabLoRA mechanism by two kinds of work. Between refreshes, each step updates only the r directional coefficients, which is cheap. The more expensive subspace refreshes happen only periodically. We will report these timing in the revision and state the overhead.
Finally, considering the expressiveness of GrabLoRA with same scale of memory as LoRA, and the accuracy gain in Table 1 and 2, we believe the comparison of GrabLoRA and other LoRA variants, as well as the tradeoff of computation and quality are reasonable. We are grateful to reviewer prompting these analyses.
[1] Zhang, F., & Pilanci, M. (2024). Spectral Adapter: Fine-Tuning in Spectral Space. arXiv preprint arXiv:2405.13952.