acceptodds
Under review as a conference paper at ICLR 2027

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Abstract

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding **LoRA-accessible gradient space** and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the **normal gradient**. Based on this decomposition, we propose **GDLoRA (Gradient-Decomposed Low-Rank Adaptation)**. GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-rank adaptation, LLM Fine-tuning, Parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wAtpOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GDLoRA, a fine-tuning method that supplements standard LoRA optimization with updates outside the current adapter-induced first-order weight-space directions. It characterizes the range of the Euclidean-gradient-induced equivalent-gradient operator and shows that this range coincides with the tangent space of the current LoRA parameterization. This characterization yields a two-sided orthogonal projection of the full weight gradient onto the corresponding normal space. GDLoRA reconstructs the full weight gradient from forward activations and backward signals, applies an additional input-count normalization to the normal branch, and directly updates the base weights using the resulting normal component. The LoRA factors retain standard AdamW optimization, and the base-weight branch introduces no additional first- or second-moment optimizer buffers. The appendices analyze finite-step normal displacements of LoRA updates, their reachable set under a factor-step budget, and local properties of direct normal compensation.

Critical strengths
  1. The paper connects a precise description of the current adapter-accessible space to an implementable normal-gradient update that leaves the adapter optimizer unchanged (Sec. 3, pp. 3–6). The construction uses thin adapter factors for basis computation and avoids additional base-weight momentum and variance states. Importantly, Appendix E explicitly accounts for the normal displacement already available through the bilinear remainder of a finite LoRA step, and characterizes its rank and nuclear-norm constraints. The authors also distinguish local gradient matching from guarantees about complete optimization trajectories. This combination of an actionable method and careful theoretical scope is consequential to my positive assessment.

  2. Tables 1–4 report improvements over same-rank LoRA across all 42 model–task pairs, with average gains of 1.75–2.50 points across the six backbone settings. The experiments include FFT, Fira, and several relevant LoRA variants, with three-seed reporting stated in Appendix G. The Full/Tangent/Normal comparison and the learning-rate sweep in Figure 2 provide meaningful evidence that the result is not explained solely by adding an arbitrary full-gradient branch or selecting one favorable base-update learning rate. Table 9 further shows gains at ranks 8, 16, and 32. The paper also reports the actual cost: 21.6 GB versus 20.4 GB for LoRA in Figure 2(c), together with runtime measurements in Table 6. These controls make the empirical contribution more convincing than a comparison of final task scores alone.

Critical weaknesses
  1. Important comparison details remain unspecified, and the practical advantage at a matched total-memory budget is not yet established. Appendix G reports independent learning-rate searches for the language experiments, which is a positive feature, but does not fully specify the selected baseline configurations or the validation and checkpoint-selection protocol. For vision, Table 12 provides GDLoRA's adapter and head learning rates while leaving FFT's base-weight learning rate and its selection unclear. Fira's projection rank, target modules, and residual-scaling configuration are also insufficiently documented. Several evaluation details, including exact checkpoints, task metrics, and reasoning prompts and answer extraction, are needed to make the comparisons reproducible. These omissions matter because the central empirical claim depends on the strength of the FFT and memory-efficient full-rank baselines; they do not, by themselves, demonstrate an unfair comparison. The resource comparison also has a narrower scope than a full practical trade-off analysis. Table 9 reports 70.80 for rank-32 LoRA versus 70.01 for rank-8 GDLoRA, whereas Figure 2(c) compares memory at rank 8. Without memory and runtime measurements across the existing rank sweep, it remains unclear how GDLoRA compares with allocating the additional memory to larger LoRA adapters. This limits the assessment of practical significance beyond the demonstrated same-rank optimizer-state comparison.

  2. The fidelity of the numerical normal update to the exact construction needs validation. Tables 10–11 report BF16 training, while Appendix H.1 accumulates full weight gradients and applies an input-count-scaled normal update directly to the base weights. The manuscript does not adequately specify the storage and computation dtypes of the base weights, accumulated gradients, projection buffers, and final weight addition, or whether master weights or another mechanism preserve small increments. BF16 training alone does not determine these choices, so I am not asserting that the updates are rounded away. Nevertheless, the choices matter both for the implemented mechanism and for the memory result in Figure 2(c). There is also an acknowledged limitation in Appendix H.2: unpivoted QR followed by diagonal thresholding need not recover the complete column space of nonzero rank-deficient factors. Consequently, the numerical residual need not be orthogonal to the full adapter-accessible space in such cases, although the exact theorems remain valid. The paper does not report how often this occurs or whether it affects the results. Representative measurements of actual base-weight changes, retained ranks, and projection error against reliable column-space bases would resolve an important gap between the mathematical construction and the implementation.

Questions for authors
  1. What exact configurations were used for FFT and Fira, particularly the vision FFT base-weight learning rates and Fira's projection rank, target modules, and residual-scaling settings? Please also specify the validation and checkpoint-selection procedure, exact model checkpoints, GLUE metrics, and reasoning evaluation protocol. Were hyperparameters and checkpoints selected independently of the final evaluation data?

  2. What dtypes are used for the base weights, full-gradient accumulation, QR/projection, residual buffers, and final weight addition? Are FP32 master weights or any compensation mechanism used? For a representative reasoning run, can you report the discrepancy between the intended normal step and the actual base-weight change, and explain how all relevant buffers are included in the memory measurement?

  3. How often does the thresholded QR procedure in Appendix H.2 discard directions that belong to the factor column spaces during the reported runs? Can you provide retained-rank and orthogonality diagnostics against bases obtained using a reliable thin-factor decomposition, together with a representative check of whether using that decomposition changes accuracy or runtime?

  4. Can you report peak memory and training time for the existing LoRA and GDLoRA rank-8/16/32 configurations in Table 9? Given that rank-32 LoRA scores 70.80 while rank-8 GDLoRA scores 70.01, these measurements would materially clarify which configuration is preferable under a fixed total-memory or training-time budget.

8 comments
AuthorsOpenReview ↗

Response to Reviewer wAtp [Part I]

Hi, Reviewer wAtp:

Thank you for recognizing our practical method, finite-step analysis, and broad evaluation.

Your concerns are (i) reproducibility, (ii) numerical fidelity, and (iii) fixed-budget efficiency. We address these concerns below.

W1/Q1: Exact FFT/Fira settings; hyperparameter/checkpoint selection independent of final evaluation?

R1: Thank you for requesting these reproducibility details. Our code is public; Tables 7, 10–12 give training settings; Appendix G covers LR searches, three-seed reporting, and Fira refresh intervals.

Selection. All methods select LRs on independent validation data, never final evaluation results. Each language baseline, including FFT/Fira, independently searches 1,3,5,7,9,10×10−5\\{1,3,5,7,9,10\\}\times10^{-5}. Vision jointly searches each method's main/head LRs:

  • Main: 10−3,5×10−2,5×10−3,10−2,4×10−2,2×10−2\\{10^{-3},5\times10^{-2},5\times10^{-3},10^{-2},4\times10^{-2},2\times10^{-2}\\}.
  • Head: 10−2,2×10−2,4×10−2,5×10−2,5×10−3,10−3\\{10^{-2},2\times10^{-2},4\times10^{-2},5\times10^{-2},5\times10^{-3},10^{-3}\\}.

DTD fits/selects/evaluates on official train/validation/test splits, never validating on test. Validation: seeded 90/10 train splits for CIFAR-10/100; 10% train holdouts for OxfordPets/StanfordCars; supplied FGVC-Aircraft/RESISC45/EuroSAT splits. Multi-epoch candidates choose and compare checkpoints by peak primary validation metric. GLUE fits official training data with separate LR/checkpoint validation. Math/commonsense: one epoch per candidate; LR selected by independent validation accuracy, never test accuracy.

Final LRs. FFT/Fira use 1e-4 on RoBERTa-Base/Large's six GLUE tasks and Llama-3-8B/Math10K; 3e-5 on Gemma-7B/Commonsense170K. Vision cells give (FFT/Fira main LR, FFT head LR); Fira head optimization/LRs follow GDLoRA (Table 12).

DatasetViT-BaseViT-Large
OxfordPets1e-3,1e-34e-2,2e-2
StanfordCars/DTD/FGVC-Aircraft/RESISC455e-2,5e-22e-2,1e-2
CIFAR-105e-3,1e-31e-2,5e-3
EuroSAT5e-2,1e-21e-2,2e-2
CIFAR-1005e-2,5e-21e-2,5e-3

Fira, Llama-3-8B/Math10K. Gradient-projection rank 8 is not adapter rank. Fira directly updates q_proj.weight, k_proj.weight, v_proj.weight in every Transformer layer; all other parameters are frozen, with no LoRA adapters. One-sided SVD (proj_type="std") projects right for an m×nm\times n gradient with m≥nm\ge n, left otherwise. Refresh: 50 optimizer steps; 100/200-step runs are also saved. Optimizer: betas=(0.9,0.999), eps=1e-6, correct_bias=True, LR=1e-4, weight decay=0.

Residual scaling has two stages, not just alpha=1 (unrelated to lora_alpha). Let Gt,gt,Ut,BtG_t,g_t,U_t,\mathcal B_t denote the full gradient, projected gradient, low-rank Adam direction, and back-projection. First,

R_t=s_t\odot[G_t-\alpha\mathcal B_t(g_t)],\quad\alpha=1.$$ Axis norms and $\odot$ scaling are row-wise for right projection, column-wise for left. Second, from step 2, growth is capped against the previous capped residual: $$c_t=\max\left(1,\frac{\\|R_t\\|_F}{1.01(\\|\bar R_{t-1}\\|_F+10^{-8})}\right),\quad \bar R_t=R_t/c_t,\quad t\ge2.$$ Step 1 is uncapped: $\bar R_1=R_1$. Finally, $$W_{t+1}=W_t-\eta_t[\alpha\mathcal B_t(U_t)+\bar R_t].$$ $\eta_t$ includes Adam bias correction; both branches share it, with no separate residual LR. **Exact revisions** (tokenizers/processors use the same repositories): |Repository|Commit| |---|---| |FacebookAI/roberta-base|e2da8e2f811d1448a5b465c236feacd80ffbac7b| |FacebookAI/roberta-large|722cf37b1afa9454edce342e7895e588b6ff1d59| |NousResearch/Meta-Llama-3-8B|315b20096dc791d381d514deb5f8bd9c8d6d3061| |google/gemma-7b|ff6768d9368919a1f025a54f9f5aa0ee591730bb| |google/vit-base-patch16-224-in21k|b4569560a39a0f1af58e3ddaf17facf20ab919b0| |google/vit-large-patch16-224-in21k|6074eaf2211423e928c93b93ef773d5da618aa7e| GLUE: accuracy for SST-2/MRPC/QNLI/RTE; Matthews correlation for CoLA; Pearson for STS-B. MRPC F1 and STS-B Spearman are also computed. **Reasoning prompt:** ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Response: ``` Generation: `do_sample=False`, `max_new_tokens=256`, `use_cache=True`; no beam search; left padding; `pad_token_id=0`. Parse after the final `### Response:`. Numerical tasks remove commas and extract the last number (absolute error ≤0.001). First matching labels: AQuA A–E; BoolQ lowercase `true/false`; PIQA `solution1/solution2`; SIQA/ARC/OBQA `answer1`–`answer5`; HellaSwag `ending1`–`ending4`; WinoGrande `option1`–`option2`. Unmatched outputs are incorrect; averages include all required tasks. **Revisions (appendix):** Selection: L.1 (p.27); LRs: L.2/Table 19 (p.28); Fira/scaling: L.3 (pp.28–29); model revisions/evaluation: L.4/Table 20 (p.29). **_`See Part II.`_**
AuthorsOpenReview ↗

Response to Reviewer wAtp [Part II]

W2 and Q2: "What dtypes are used for the base weights, full-gradient accumulation, QR/projection, residual buffers, and final weight addition?" "Are FP32 master weights or any compensation mechanism used?"

R2: Thank you. Our BF16 reasoning experiments use FP32 gradient recovery, accumulation, QR and projection, followed by BF16 base-weight write-back, without FP32 master weights or cross-step compensation.

Operation/stateRoBERTaLlama-3-8B/Math10K; Gemma-7B/Commonsense170K
Base storage/forwardFP32BF16 weights; mixed-precision forward
Full-gradient recovery/microbatch accumulationFP32FP32
Thin-factor QR/bases/projectionFP32FP32
Normal-residual bufferFP32; cleared after each stepFP32; cleared after each step
Final base updateFP32Cast residual to BF16 before scaling and in-place addition; store in BF16
Additional FP32 master weightsNone; base weights are FP32None
Error feedback/compensated summationNoneNone

Activations/backward signals are cast to FP32 before gradient recovery, which, like QR/projection, runs with autocast disabled. The temporary residual buffer retains no cross-step rounding errors. No base-weight Adam states does not imply zero temporary-buffer memory.

Write-back fidelity. We isolate the normal branch and compare the FP32-residual-based intended increment with before/after stored weights converted to FP64:

\Delta W_t^{\mathrm{actual}}=\operatorname{FP64}(W_{\mathrm{after}})-\operatorname{FP64}(W_{\mathrm{before}}).$$ $$\epsilon_{\mathrm{write}}=\frac{\\|\Delta W_t^{\mathrm{actual}}-\Delta W_t^{\mathrm{intended}}\\|_F}{\\|\Delta W_t^{\mathrm{intended}}\\|_F}.$$ The denominator is the **intended update norm**, not the base-weight norm. On **Llama-3-8B/Math10K, rank 8, seed 42**, we uniformly sample **100 optimizer steps, 6 steps apart across training**. We audit **q/k/v** (`q_proj`, `k_proj`, `v_proj`) in **one-based layers 1/16/32**, not training stages: **9 modules per step, 900 module–step observations**. We report pooled and per-module P95 errors, actual/intended norm ratios, and the fraction of nonzero intended entries rounded to zero. These last entry counts must not be interpreted as lost gradient energy. | Layer | Pooled error (%) | Per-module P95 error (%) | Actual/intended norm ratio | Zero-change entries (%) | |:---:|:---:|:---:|:---:|:---:| | 1 | 0.84 | 1.62 | 0.9993 | 3.8 | | 16 | 1.03 | 1.95 | 0.9988 | 4.6 | | 32 | 1.17 | 2.21 | 0.9985 | 5.3 | Pooled BF16 write-back errors are **0.84–1.17%**, with actual/intended norm ratios of **0.9985–0.9993**. The **3.8–5.3%** zero-change fraction of nonzero intended entries is not lost update energy. These indicate limited aggregate distortion in the sampled layers; R3 separately assesses projection accuracy. **Memory.** Figure 2(c) reports **21.6 GB for GDLoRA versus 22.3 GB for Fira** on Llama-3-8B/Math10K, a reported 0.7 GB (3.1%) difference. The revision reports **peak allocated and peak reserved** memory separately from the existing GPU-memory figures. The stated measurement protocol uses fresh processes under matched hardware, batching, sequence length, accumulation, and training duration. The protocol resets peak counters after model/data loading but before optimizer initialization and the first forward pass, then reads synchronized peaks after the final update. The window includes initial optimizer states, gradient accumulation, QR/SVD workspaces, projection and residual buffers, dtype conversions, every Fira refresh, and GDLoRA write-back; FP64 auditing runs separately. The protocol specifies per-device decimal GB and synchronized first-forward-to-final-update training time, excluding loading, evaluation and checkpointing. | Method | Existing GPU memory (GB) | Peak allocated (GB) | Peak reserved (GB) | |:---|:---:|:---:|:---:| | Fira | 22.3 | 23.4 | 24.0 | | GDLoRA | 21.6 | 21.3 | 21.9 | **The absence of base-weight optimizer moments is structural; the total-memory claim and finite-precision update fidelity require their own measurements.** The rank-wise table in R4 retains its reported GPU-memory and time figures; it is not a rank-by-rank allocated/reserved-memory breakdown. **Changes in the revised manuscript:** The dtype path is in Appendix M.1 (p. 29) and Table 21 (p. 30); write-back definitions and results are in Appendix M.2 (p. 30) and Table 22 (p. 30); memory metrics and the timing protocol are in Appendix M.4 (p. 31) and Table 25 (p. 31). **_`See Part III.`_**
AuthorsOpenReview ↗

Response to Reviewer wAtp [Part III]

W2 and Q3: "How often does the thresholded QR procedure in Appendix H.2 discard directions that belong to the factor column spaces during the reported runs?"

R3: Thank you. Appendix H.2 uses an absolute threshold of 10−610^{-6} for near-zero columns and diagonal filtering after unpivoted QR. The concern is missed numerical column-space directions, not simply retained rank differing from the configured adapter rank. The zero rank of initially zero BB is not underestimation.

We compare same-step B,A⊤B,A^\top against FP64 thin-factor SVD bases, counting singular values above 10−610^{-6} as reference rank. On Llama-3-8B/Math10K, rank 8, seed 42, we uniformly sample 100 optimizer steps, 6 steps apart across training. Auditing q/k/v (q_proj, k_proj, v_proj) in all 32 layers gives 96 modules per step, 9,600 factor-pair observations. With the same full gradient, we measure

\epsilon_{\mathrm{orth}}=\frac{\\|\mathcal P_{\mathcal T_{\mathrm{ref}}}(\widehat G_\perp^{\mathrm{QR}})\\|_F}{\\|\widehat G_\perp^{\mathrm{QR}}\\|_F}.$$ These quantify projection discrepancy and tangent-space leakage, including cases with matching ranks but differing subspaces; zero denominators are recorded separately. | Rank | Reference ranks ($B,A^\top$) | QR retained ranks ($B,A^\top$) | Underestimation rate | Projection error (mean/maximum) | Orthogonality error (mean/maximum) | |:---:|:---:|:---:|:---:|:---:|:---:| | 8 | (8,8) | (8,8) | 0% | 1.8e-7/9.2e-7 | 1.5e-7/7.8e-7 | At **rank 8**, QR and FP64 SVD agree in numerical rank, with **zero observed underestimation** and $10^{-7}$–$10^{-6}$ projection/orthogonality discrepancies. This supports QR for these sampled factors, not arbitrary rank-deficient cases under a fixed threshold. Slash-separated pairs are **mean/maximum**, separated into columns in the revision. A separate Llama-3-8B/Math10K ablation changes **only FP32 thin-factor QR to FP32 thin-factor SVD**, retaining all other settings, including **rank 8, seed 42, and q/k/v**: | Basis | Average accuracy (%) | Training time (min) | Peak GPU memory (GB) | |:---|:---:|:---:|:---:| | Thin-factor QR (FP32) | 70.01 | 21.0 | 21.6 | | Thin-factor SVD (FP32) | 70.24 | 29.3 | 21.6 | SVD gains **0.23 points** for **8.3 extra minutes (39.5%)**, with equal peak memory at displayed precision: QR gives similar accuracy faster. Neither decomposes the full gradient; this ablation complements, not replaces, rank/orthogonality diagnostics. The revision includes these statistics and the rank-8, seed-42, Q/K/V setting. **Changes in the revised manuscript:** The QR/SVD audit and basis-construction comparison are in Appendix M.3 (p. 30), Table 23 (p. 31), and Table 24 (p. 31). Mean and maximum errors are explicitly distinguished. **_`W1 and Q4: "Can you report peak memory and training time for the existing LoRA and GDLoRA rank-8/16/32 configurations in Table 9?"`_** **R4:** Thank you. **Fixed-budget evaluation must compare accuracy, memory, and runtime against higher-rank LoRA, not only matched ranks.** Table 9 accuracies and the Llama-3-8B/Math10K GPU memory and training-time results are: | Method | Rank | Average accuracy (%) | GPU memory (GB) | Time (min) | |:---|:---:|:---:|:---:|:---:| | LoRA | 8 | 67.51 | 20.4 | 19 | | LoRA | 16 | 69.67 | 20.9 | 21 | | LoRA | 32 | 70.80 | 21.9 | 22 | | GDLoRA | 8 | 70.01 | 21.6 | 21 | | GDLoRA | 16 | 71.96 | 21.9 | 24 | | GDLoRA | 32 | 72.81 | 22.4 | 24 | At the same training time of **21 minutes**, **GDLoRA-r8 exceeds LoRA-r16 by 0.34 percentage points**, using **0.7 GB** more GPU memory (21.6 vs. 20.9 GB). LoRA-r32 does exceed GDLoRA-r8 (70.80 vs. 70.01); however, **GDLoRA-r16 exceeds LoRA-r32 by 1.16 percentage points at the same GPU memory usage of 21.9 GB**, at a **2-minute (9.1%)** training-time cost. At matched ranks 8/16/32, GDLoRA improves accuracy by **2.50/2.29/2.01 percentage points**, respectively, with additional GPU memory of **1.2/1.0/0.5 GB** and additional training time of **2/3/2 minutes**. These results show that GDLoRA can improve accuracy at the same memory budget, while the runtime trade-off remains; they do not establish dominance at every time budget. The revision includes this table and distinguishes its reported GPU-memory measurements from the allocated/reserved-memory comparison in R2. **Changes in the revised manuscript:** The rank-wise accuracy–memory–time comparison is in Appendix K.4 (p. 27) and Table 18 (p. 28), with the separate memory-accounting clarification in Appendix M.4 (p. 31). **_`Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns and clarify any doubts. Please let us know of any further questions.`_**
Reviewer_TrrgOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper splits the full weight gradient into directions the current LoRA factors can express to first order and an orthogonal component that their gradients miss. It uses that orthogonal component to update the base weights directly. Multiple experiments were performed to compare the proposed method to other baseline approaches under different benchmarks shows that the proposed performance achieves better resutls.

Critical strengths
  • The decomposition identifies gradient information absent from the current factor gradients, and the proposed update explicitly uses that information. The range identity itself is standard linear algebra, but turning the decomposition into a concrete training method is a meaningful contribution. The authors also acknowledge finite-step normal movement in appendix E, rather than incorrectly claiming that LoRA can never leave its current tangent space.

  • The method preserves standard AdamW optimization of the LoRA factors while applying the token-normalized normal gradient directly to the base weights, without additional base-weight momentum or second-moment buffers. The experimental evaluation covers multiple baselines and benchmarks, and the reporting of standard deviations helps assess the consistency of the results.

Critical weaknesses
  1. The method avoids full-sized optimizer buffers, but the task-specific change is no longer limited to a small adapter. Retaining only AA, BB is therefore insufficient to reproduce the fine-tuned model, which complicates compact checkpoint storage and sharing one frozen backbone across tasks. This is mainly a trade-off depends on the specific application.

  2. The theoretical presentation is unnecessarily difficult to follow. Several symbols are introduced with little explanation of their role or their connection to earlier definitions. For example, the relationship between Φs\Phi_s and JsJ_s are left largely for the reader to reconstruct. The interpretation of ηn\eta_n as the normal-update learning rate was introduced on page 5 but not explained until page 6. More explicit notation and better transitions between the geometric decomposition and the optimization argument would substantially improve readability.

Questions for authors
  1. Clarification needed: in the learning-rate sensitivity ablation, does ηn\eta_n in Figure 2(b) refer to the normal-gradient learning rate ηn,t\eta_{n,t} introduced on line 302, or the base learning rate for its schedule? Which learning rate is varied for LoRA, LoRA+Full, and LoRA+Tangent, and which rates are held fixed?
  2. Please explain the roles and relationships of the theoretical notation more explicitly, especially JsJ_s, Φs\Phi_s, HAH_A, and ηn\eta_n. The current presentation leaves too much for the reader to reconstruct.
  3. Please make the figure labels more descriptive. For example, if “Avg” in Figure 2(c) means accuracy averaged across tasks for the blue bar, “Average accuracy (%)” would be clearer.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Trrg [Part I]

Hi, Reviewer Trrg:

Thank you for recognizing the contribution of turning gradient decomposition into a concrete training method, our careful treatment of finite-step LoRA updates, and the breadth of our evaluation across baselines and benchmarks.

We understand that your main concerns focus on (i) task-specific storage and shared-backbone deployment, (ii) the readability of our theoretical notation and derivations, and (iii) the clarity of learning-rate settings and figure labels. We address these concerns below.

W1: "The method avoids full-sized optimizer buffers, but the task-specific change is no longer limited to a small adapter."

R1: Thank you for highlighting this important distinction. GDLoRA's central contribution is to expand weight adaptation beyond the first-order limits of the current LoRA factors while preserving standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. By combining AdamW optimization of the factors with direct normal-gradient updates, GDLoRA enables richer, potentially full-rank adaptation of the selected base matrices without maintaining full-size base-weight momentum or variance states. The revision makes its positioning as a memory-efficient adaptation method explicit.

This positioning motivates our baseline selection: LoRA and its variants quantify the benefits and additional costs of expanding the update space; Fira provides a direct comparison with memory-efficient full-rank training that also updates the base weights; and FFT provides the full fine-tuning reference.

Compared with standard LoRA, GDLoRA delivers clear performance improvements on Llama-3-8B fine-tuned on Math10K. With both methods using rank 8, GDLoRA improves the average accuracy across six mathematical reasoning tasks from 67.51% to 70.01% (+2.50 percentage points), approaching FFT's 70.63% with a gap of only 0.62 percentage points. At the same time, GDLoRA reduces GPU memory usage from 61.79 GB for FFT to 21.6 GB, a reduction of 65.0% (Table 2 and Figure 2(c)). These results show that normal-gradient compensation can substantially narrow the accuracy gap to FFT while retaining LoRA's optimizer-state budget.

For comparison with Fira, which also updates the base weights, the Llama-3-8B/Math10K results at Fira's highest-accuracy tested refresh interval of 50 steps are:

MethodAverage accuracy (%)Training time (min)GPU memory (GB)
Fira67.354722.3
GDLoRA70.012121.6

GDLoRA achieves 2.66 percentage points higher accuracy and 55.3% lower training time. Thus, even when both methods permit base-weight updates, GDLoRA delivers a more favorable accuracy-resource trade-off, providing direct evidence of the practical value of its update mechanism (Table 2, Figure 2(c), and Appendix A).

Regarding task-specific storage, for inference, each task can store the merged differences between the modified matrices and their pretrained counterparts:

ΔWtask=(Wbase,T−Wpt)+sBTAT\Delta W_{\mathrm{task}} = (W_{\mathrm{base},T} - W_{\mathrm{pt}}) + s B_T A_T

These generally dense differences, together with any other task-specific trained parameters, are sufficient to reconstruct the adapted model. The pretrained checkpoint can remain a common reference, and unchanged parameters can still be shared, avoiding a complete model copy for each task.

GDLoRA therefore makes an explicit trade-off: larger task-specific weight differences in exchange for broader adaptation capacity, while retaining LoRA-level optimizer-state costs. This is valuable for applications that prioritize downstream accuracy and training resources. The revision states this intended use case and the associated checkpoint-storage and task-switching requirements.

Changes in the revised manuscript: See the added positioning statement in Section 1 (p. 2) and the deployment/storage discussion in Appendix I (p. 25), including Eq. 61.

Due to space limitations, please refer to Part II for our responses to other questions.

AuthorsOpenReview ↗

Response to Reviewer Trrg [Part II]

W2 and Q2: "Several symbols are introduced with little explanation of their role or their connection to earlier definitions." "The current presentation leaves too much for the reader to reconstruct."

R2: Thank you. The revision distinguishes a direction, the map generating it, and the space of all such directions, following: factor perturbations → accessible space → orthogonal decomposition → normal update.

At fixed current factors A,BA,B, HA∈Rr×diH_A\in\mathbb{R}^{r\times d_i} and HB∈Rdo×rH_B\in\mathbb{R}^{d_o\times r} are arbitrary perturbation directions with the same shapes as AA and BB, respectively. Under (A+ϵHA,B+ϵHB)(A+\epsilon H_A,B+\epsilon H_B), Js(HA,HB)J_s(H_A,H_B) gives the first-order weight change per unit ϵ\epsilon. These directions are not restricted to gradient or AdamW steps.

NotationRole
Fs(A,B)=sBAF_s(A,B)=sBALoRA weight-update parameterization; ss is its scaling factor.
Js(HA,HB)=s(BHA+HBA)J_s(H_A,H_B)=s(BH_A+H_BA)Maps arbitrary factor perturbations to first-order weight changes; range: tangent space TLoRA\mathcal T_{\mathrm{LoRA}}.
Φs(H)=s2(BB⊤H+HA⊤A)\Phi_s(H)=s^2(BB^\top H+HA^\top A)Maps an arbitrary candidate weight gradient HH to the first-order weight-space direction induced by Euclidean factor gradients; range: GLoRA\mathcal G_{\mathrm{LoRA}}.
GLoRA(G)=Φs(G)G_{\mathrm{LoRA}}(G)=\Phi_s(G)Its value at the current gradient GG: one direction, not the whole accessible space or the actual AdamW update.

Theorem 1 equates the two spaces:

=\operatorname{Range}(\Phi_s) =\operatorname{Range}(J_s) =\mathcal T_{\mathrm{LoRA}}.$$ This does not make $G_{\mathrm{LoRA}}(G)$ the orthogonal tangent projection of $G$: $G-G_{\mathrm{LoRA}}(G)$ is generally not the normal gradient. With orthonormal bases $U,V$ for the column spaces of $B,A^\top$, respectively, we instead compute $$G_\perp=(I-UU^\top)G(I-VV^\top).$$ These relationships now follow their definitions. To connect geometry and optimization, $G=G_{\mathrm{raw}}/N$ is the gradient of the normalized current-batch loss $\overline{\mathcal L}=\mathcal L/N$, and $D$ is an additional base-weight change, not a factor update. With adapters and other weights fixed, $\langle G,D\rangle_F$ is the first-order loss change. Eq. (17) minimizes this approximation plus a magnitude penalty over the normal space, giving $D=-\eta_nG_\perp$ and hence the update rule. We define **$\eta_n$ as the normal-update learning rate at first occurrence**; $\eta_{n,t}$ in Eq. (20) is its step-$t$ value, distinct from the adapter AdamW rate $\eta_{a,t}$. **Changes in the revised manuscript:** Section 3.2 (p. 5) adds explanations beside the definitions and normal-step objective. Appendix J and Table 13 (p. 25) collect the notation and link the maps, spaces, normalized loss, and update, with improved transitions. **_`Q1: "Which learning rate is varied for LoRA, LoRA+Full, and LoRA+Tangent, and which rates are held fixed?"`_** **R3:** Figure 2(b) varies **the additional base-weight update rate, not the adapter AdamW rate**. GDLoRA's rate controls the normal update in Eq. (20); the other branches are: |Method|Additional base-update component|Rate varied in Figure 2(b)| |:--|:--|:--| |LoRA|None|None; fixed reference| |LoRA+Full|Full gradient|Full-gradient base-update rate| |LoRA+Tangent|Tangent gradient|Tangent-gradient base-update rate| |GDLoRA|Normal gradient|Normal-gradient base-update rate| The latter three sweep $\{0.05,0.15,0.5,1.5,5\}$. Each GDLoRA run holds its selected value constant throughout training, $\eta_{n,t}=\eta_n$: it is not a scheduler reference value and has no warmup or decay. Each method's adapter AdamW rate and schedule remain unchanged across the sweep. LoRA's horizontal line is a fixed reference, not a sweep of its adapter rate. **Changes in the revised manuscript:** Figure 2(b)'s horizontal axis now reads **"Base-weight update learning rate"**. Its caption, Section 3.3, and Appendix J.1 (p. 26) clarify the update components, constant normal rate, and unchanged adapter configurations. **_`Q3: "Please make the figure labels more descriptive."`_** **R4:** Thank you. Figure 2(c)'s blue-bar axis and legend replace **"Avg." with "Average accuracy (%)"**, explicitly averaging six mathematical reasoning tasks; its memory axis and legend use **"GPU memory (GB)"**. Figure 2(b) uses **"Average accuracy (%)"** vertically and **"Base-weight update learning rate"** horizontally, with update components and rate configurations in the caption. Figure 2(a) uses **"Average GLUE score"** to match its metric. **Changes in the revised manuscript:** Figure 2(a–c), its caption in Section 4.5, and Appendix J.1 (p. 26) contain these label and metric/rate clarifications. **_`Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns and clarify any doubts. Please let us know of any further questions.`_**

Dear Reviewer Trrg,

We would appreciate your feedback on whether our rebuttal has adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarification and continue the discussion.

Thank you for your time and consideration.

Reviewer_x1TQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GDLoRA, a technique that decomposes the full gradient in the LoRA accessible tangent space and its orthogonal complement and updates LoRA factors A,B based on the former and the base matrix based on the latter. The full gradient is reconstructed from the layer activations and backward signal, without maintaining AdamW first and second moments for the backbone. Across RoBERTa, Llama-3-8B, Gemma-7B, and ViT models, GDLoRA generally comes close to or occasionally exceeds FFT, at a lower training cost than full fine-tuning.

Critical strengths
  • The tangent/normal decomposition gives an intuitive interpretation of information that is inaccessible to the current LoRA parameterization at first order. GDLoRA retains standard AdamW for the adapter factors and adds a direct normal-gradient update to the base weights without introducing full-size momentum and variance states.
  • The paper evaluates several language and vision backbones and compares against standard LoRA, multiple LoRA variants, Fira, and full fine-tuning. The improvements over LoRA are consistent, and the paper includes useful component, learning-rate, rank, memory, and runtime analyses.
Critical weaknesses
  • GD-LoRA sacrifices one of the main properties of PEFT, that the backbone weights remain frozen. This can be a desired property for various reasons, including task or user specific adaptations served over a common backbone. Checkpointing is also affected by this. Therefore, the method can be thought of as a memory-efficient full-rank training method, rather than PEFT, which creates a very different setup and baselines.
  • Full gradient recovery is not free, as GD-LoRA needs to reconstruct G=δTXG=\delta^T X. This can become more noticeable in longer sequences and larger models.
  • The paper does not cleanly separate the benefit of accessing $G_\perp$ from deficiencies in how G∥G_\parallel is optimised. The paper's LoRA+Full and LoRA+Tangent ablations provide useful evidence that the normal component is beneficial. Effectively I would expect a clearer measurement of how much gradient energy actually lies in the normal component.
Questions for authors
  • Comparison to optimizer-aware LoRA methods: Can the authors compare GDLoRA with LoFT [a], or evaluate a combination in which LoFT-like optimization is used for the tangent-space branch while GDLoRA supplies the normal component? This would help separate gains from improved optimization within the accessible space from gains due to accessing (G⊥G_\perp). Additional positioning against [b] would also be helpful for context.
  • Can the authors provide evidence that the reported efficiency tradeoff persists in a more demanding LLM setting—for example, all-linear rather than Q/K/V-only adaptation, a substantially longer context, and/or a larger model? Memory and wall-clock comparisons would be particularly useful.
  • What state must be stored for each downstream task after GDLoRA training?
  • Can GDLoRA support the standard LoRA deployment pattern in which multiple interchangeable task adapters share one immutable pretrained backbone?
  • The appendix already shows that GDLoRA's accuracy advantage persists for ranks {8, 16, 32}. How does the relative normal-gradient magnitude, e.g. (∣G⊥∣F/∣G∣F|G_\perp|_F / |G|_F), evolve with adapter rank, layer, and training step? Such an analysis would more directly connect the proposed geometric mechanism to the observed improvements.

[a] Tastan, N., Laskaridis, S., Takáč, M., Nandakumar, K., & Horváth, S. (2026, April). Loft: Low-rank adaptation that behaves like full fine-tuning. In International Conference on Learning Representations (Vol. 2026, pp. 71294-71324).
[b] Zhang, Y., Yi, L., Li, D., Ding, J., Wang, X., Song, T., & Ning, Z. (2026). DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation. arXiv preprint arXiv:2610.17730.

10 comments
AuthorsOpenReview ↗

Response to Reviewer x1TQ [Part I]

Hi, Reviewer x1TQ:

Thank you for recognizing GDLoRA’s intuitive geometric interpretation, consistent improvements over LoRA, and broad evaluation across language and vision tasks.

We understand that your main concerns focus on (i) method positioning and deployment trade-offs, (ii) training efficiency and scalability, and (iii) distinguishing normal-gradient benefits from improved tangent-space optimization, particularly through optimizer-aware comparisons and gradient-energy analysis. We address these concerns point by point in the following three parts.

W1 and Q3 and Q4: "GD-LoRA sacrifices one of the main properties of PEFT, that the backbone weights remain frozen." "What state must be stored for each downstream task after GDLoRA training?" "Can GDLoRA support the standard LoRA deployment pattern in which multiple interchangeable task adapters share one immutable pretrained backbone?"

R1: Thank you for highlighting this important distinction. GDLoRA's central contribution is to expand weight adaptation beyond the first-order limits of the current LoRA factors while preserving standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. By combining AdamW optimization of the factors with direct normal-gradient updates, GDLoRA enables richer, potentially full-rank adaptation of the selected base matrices without maintaining full-size base-weight momentum or variance states. The revision makes its positioning as a memory-efficient adaptation method explicit.

This positioning motivates our baseline selection: LoRA and its variants quantify the benefits and additional costs of expanding the update space; Fira provides a direct comparison with memory-efficient full-rank training that also updates the base weights; and FFT provides the full fine-tuning reference.

Compared with standard LoRA, GDLoRA delivers clear performance improvements on Llama-3-8B fine-tuned on Math10K. With both methods using rank 8, GDLoRA improves the average accuracy across six mathematical reasoning tasks from 67.51% to 70.01% (+2.50 percentage points), approaching FFT's 70.63% with a gap of only 0.62 percentage points. At the same time, GDLoRA reduces GPU memory usage from 61.79 GB for FFT to 21.6 GB, a reduction of 65.0% (Table 2 and Figure 2(c)). These results show that normal-gradient compensation can substantially narrow the accuracy gap to FFT while retaining LoRA's optimizer-state budget.

For comparison with Fira, which also updates the base weights, the Llama-3-8B/Math10K results at Fira's highest-accuracy tested refresh interval of 50 steps are:

MethodAverage accuracy (%)Training time (min)GPU memory (GB)
Fira67.354722.3
GDLoRA70.012121.6

GDLoRA achieves 2.66 percentage points higher accuracy and 55.3% lower training time. Thus, even when both methods permit base-weight updates, GDLoRA delivers a more favorable accuracy-resource trade-off, providing direct evidence of the practical value of its update mechanism (Table 2, Figure 2(c), and Appendix A).

Regarding task-specific storage, for inference, each task can store the merged differences between the modified matrices and their pretrained counterparts:

ΔWtask=(Wbase,T−Wpt)+sBTAT\Delta W_{\mathrm{task}} = (W_{\mathrm{base},T} - W_{\mathrm{pt}}) + s B_T A_T

These generally dense differences, together with any other task-specific trained parameters, are sufficient to reconstruct the adapted model. The pretrained checkpoint can remain a common reference, and unchanged parameters can still be shared, avoiding a complete model copy for each task.

GDLoRA therefore makes an explicit trade-off: larger task-specific weight differences in exchange for broader adaptation capacity, while retaining LoRA-level optimizer-state costs. This is valuable for applications that prioritize downstream accuracy and training resources. The revision states this intended use case and the associated checkpoint-storage and task-switching requirements.

Changes in the revised manuscript: We added the positioning statement in Section 1 (p. 2) and the task-difference reconstruction and deployment discussion in Appendix I (p. 25), including Eq. 61.

Due to space limitations, please refer to Part II for our responses to other questions.

AuthorsOpenReview ↗

Response to Reviewer x1TQ [Part II]

W2 and Q2: "Full gradient recovery is not free..." "Can the authors provide evidence that the reported efficiency tradeoff persists in a more demanding LLM setting?"

R2: Thank you for highlighting the cost of full-gradient computation and the importance of evaluating efficiency in more demanding LLM settings. For FFT, Fira, and GDLoRA, computing the full gradients of the target weight matrices is a basic operation in their respective update procedures. GDLoRA recovers these gradients from forward activations and backward signals, so this computation represents a shared fundamental cost with methods such as Fira.

GDLoRA offers both memory and computational advantages. Full gradients and residuals are temporary buffers that can be released or reused after each step, without introducing base-weight momentum or variance states. Its projection bases are constructed directly from thin adapter factors, avoiding Fira’s periodic full-gradient SVD and reducing training latency, while preserving LoRA’s optimizer-state budget under matched configurations

The practical efficiency of this design is reflected in the Llama-3-8B/Math10K results. Compared with Fira at its highest-accuracy tested refresh interval of 50 steps:

MethodAverage accuracy (%)Training time (min)GPU memory (GB)
Fira67.354722.3
GDLoRA70.012121.6

GDLoRA reduces training time from 47 to 21 minutes, a reduction of 55.3%, while improving average accuracy by 2.66 percentage points and lowering GPU memory usage from 22.3 to 21.6 GB. Thus, even when both methods compute full weight gradients, GDLoRA achieves higher accuracy with lower training costs (Table 2, Figure 2(c), and Appendix A).

To address your suggestion regarding larger models and all-linear adaptation, we conducted additional all-linear fine-tuning experiments with Qwen2.5-14B on Math10K, increasing both the model size and the scope of adaptation:

MethodAverage accuracy (%)Training time (min)GPU memory (GB)
LoRA81.734841.3
Fira78.5712743.5
FFT84.5594125.7
GDLoRA84.166242.9

In this more demanding setting, GDLoRA improves average accuracy over Fira by 5.59 percentage points and reduces training time from 127 to 62 minutes, a reduction of 51.2%, while also using less GPU memory. Compared with FFT, GDLoRA is only 0.39 percentage points lower in accuracy, while reducing GPU memory usage from 125.7 to 42.9 GB, a reduction of 65.9%. These results demonstrate that, with a larger model and all-linear adaptation, GDLoRA still approaches full fine-tuning performance at substantially lower memory cost, while maintaining higher accuracy and approximately halving training time relative to Fira.

The trade-off relative to standard LoRA is also clear: GDLoRA gains 2.43 percentage points in accuracy with 1.6 GB (3.9%) additional GPU memory and an increase in training time from 48 to 62 minutes. The additional experiments therefore directly support our efficiency claim: GDLoRA retains a favorable accuracy-resource trade-off when extending from Q/K/V adaptation to a larger model with all-linear adaptation, bringing accuracy close to FFT while keeping total GPU memory close to LoRA. These results are included in the revised manuscript.

Changes in the revised manuscript: The new all-linear Qwen2.5-14B/Math10K results are in Appendix K.1 (p. 26) and Table 14 (p. 26). The original Fira refresh-cost analysis in Appendix A is retained.

Due to space limitations, please refer to Part III for our responses to other questions.

AuthorsOpenReview ↗

Response to Reviewer x1TQ [Part III]

W3 and Q5: "I would expect a clearer measurement of how much gradient energy actually lies in the normal component." "How does the relative normal-gradient magnitude ... evolve with adapter rank, layer, and training step?"

R3: Thank you for suggesting direct quantification of the normal gradient. We measured its relative magnitude:

ρ=∥G⊥∥F∥G∥F\rho = \frac{\lVert G_{\perp}\rVert_F}{\lVert G\rVert_F}

In our RoBERTa experiments with LoRA, the relative normal-gradient magnitude is 95.85%, indicating a substantial gradient component outside the adapters' current first-order accessible space. Together with Figure 2's gradient-component ablations using the same adapter optimization rule, this supports directly incorporating normal gradients.

Below are the relative normal-gradient magnitudes and performance across ranks on Llama-3-8B/Math10K; accuracies are from Table 9 of the original submission.

RankRelative magnitude (%)LoRA Avg. (%)GDLoRA Avg. (%)Gain (percentage points)
896.2667.5170.01+2.50
1694.2169.6771.96+2.29
3291.3570.8072.81+2.01

The relative normal-gradient magnitude decreases with rank but remains 91.35% at rank 32, where GDLoRA outperforms LoRA by 2.01 percentage points. Thus, increasing rank does not eliminate the benefit of incorporating normal gradients.

We collected statistics for all 32 layers. Every layer shows rank 8 > rank 16 > rank 32 in relative normal-gradient magnitude, with no monotonic trend across depth. Selected results follow; layers are numbered from 1.

LayerRank 8 (%)Rank 16 (%)Rank 32 (%)
195.3591.7088.52
897.3895.1892.19
1696.8895.0992.50
2496.1894.6091.69
3293.9790.9886.66

Thus, the decrease in relative normal-gradient magnitude with rank holds across all measured layers, rather than being driven by only a few layers.

The revision includes the rank-wise and selected layer-wise measurements reported above.

Changes in the revised manuscript: The norm-ratio definition and rank/layer measurements are in Appendix K.3 (p. 27), Table 16 (p. 27), and Table 17 (p. 27).

Q1: "Can the authors compare GDLoRA with LoFT, or evaluate a combination in which LoFT-like optimization is used for the tangent-space branch while GDLoRA supplies the normal component?" "Additional positioning against would also be helpful for context."

R4: Thank you for this suggestion. LoFT improves low-rank optimization through alternating factor updates and optimizer-state calibration. We evaluated LoFT and LoFT+GDLoRA to test whether normal-gradient compensation adds gains beyond improved tangent-space optimization.

Below are the Llama-3-8B/Math10K results, all at rank 8. LoFT+GDLoRA uses LoFT for adapter optimization and GDLoRA's normal-gradient branch for base-weight updates.

MethodAddSubMultiArithSingleEqSVAMPGSM8KAQuAAvg.
LoRA81.5287.2891.5466.9355.3422.4467.51
LoFT82.3489.5690.3166.7055.7624.8968.26
LoFT+GDLoRA86.4390.2493.7870.3056.9725.9270.61
GDLoRA84.8190.6992.5269.6156.0326.3870.01

GDLoRA outperforms LoFT by 1.75 percentage points on average. Adding normal-gradient compensation improves LoFT from 68.26% to 70.61% (+2.35 percentage points), with gains on all six tasks. The combination exceeds GDLoRA by 0.60 percentage points. These results indicate that normal-gradient compensation benefits extend beyond standard LoRA optimization, complementing improved tangent-space optimization.

DT-LoRA focuses on incorporating new residual directions within a fixed low-rank adapter budget, using periodic matrix-free randomized SVD followed by recompression. Its explicit use of residual information is therefore constrained by the refresh interval and rank truncation. GDLoRA instead performs direct weight-space compensation, applying the entire normal component to the base weights at every step without waiting for periodic refreshes or compressing it back into a fixed adapter budget. Its projection bases come from the current low-rank factors, avoiding residual SVD and recompression while retaining standard factor AdamW and LoRA's optimizer-state budget under matched configurations. The associated storage and deployment trade-offs are discussed in R1.

Changes in the revised manuscript: The LoFT and LoFT+GDLoRA comparison is in Appendix K.2 (p. 26) and Table 15 (p. 26); the same subsection adds the discussion of DT-LoRA and its distinction from direct normal-gradient updates.

Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns and clarify any doubts. Please let us know of any further questions.

Comments on the paper

2 comments

Dear reviewers,

The authors have submitted their rebuttal. Please read it carefully, along with the other reviews, and respond whether it adequately addresses your concerns and whether you plan to update your score.

If concerns remain, please state the open points clearly and specifically so the authors can follow up during the discussion period.

Thank you for your engagement!

Best, AC

AuthorsOpenReview ↗

Summary of Review and Discussion

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback. We briefly summarize the discussion, GDLoRA's contributions, and our responses to the reviews.

I. Review and Discussion

ReviewerInitial ratingCurrent rating
x1TQ23 (Weak acceptance)
Trrg33 (Weak acceptance)
wAtp23 (Weak acceptance)

We submitted our point-by-point rebuttal on November 12. Following additional comparisons with GaLore, APOLLO, and SCALE, Reviewer x1TQ raised their rating from 2 to 3 on November 19, stating: “Thank you for running the extra comparisons. I will increase my score.”

On the same day, Reviewer wAtp also raised their rating from 2 to 3, stating: “Thanks for running the extra comparisons. This resolves my concerns, and I will update my score.” All three reviewers now assign positive ratings (3-3-3), with two reviewers explicitly increasing their scores following the rebuttal.

II. Contributions of This Paper

GDLoRA identifies gradient directions unavailable to the current LoRA factors at first order and directly uses the complementary normal gradient to update selected base weights, while retaining standard AdamW optimization of the adapters.

  • Geometric foundation: We characterize the adapter-accessible space and its orthogonal complement, while explicitly accounting for the normal displacement possible through finite LoRA steps.
  • Memory-efficient adaptation: GDLoRA introduces no base-weight momentum or variance states, preserving LoRA's optimizer-state budget under matched configurations. Its base-weight updates entail an explicit task-storage trade-off.
  • Consistent performance: GDLoRA improves over same-rank LoRA across all 42 model-task pairs, spanning language understanding, mathematical reasoning, commonsense reasoning, and image classification.
  • Scalability and complementarity: On Qwen2.5-14B with all-linear adaptation, GDLoRA reaches 84.16% versus FFT's 84.55%, with 65.9% less reported GPU memory. Adding its normal branch to LoFT improves accuracy from 68.26% to 70.61% on Llama-3-8B/Math10K.

III. Reviewers' Evaluations and Our Responses

Recognized strengths:

  1. A meaningful, implementable use of complementary gradient information [x1TQ, Trrg, wAtp].
  2. Careful treatment of finite-step LoRA updates and theoretical scope [Trrg, wAtp].
  3. Broad evaluation, consistent improvements, and informative ablations [x1TQ, Trrg, wAtp].

Main concerns and corresponding revisions:

  1. Positioning and deployment [x1TQ, Trrg]. We explicitly position GDLoRA as memory-efficient adaptation and clarify dense task-specific weight storage and task-switching requirements (Section 1; Appendix I).
  2. Baseline coverage, scalability, and mechanism [x1TQ]. We added LoFT/LoFT+GDLoRA, Qwen2.5-14B all-linear experiments, and rank/layer gradient diagnostics (Appendix K). Further discussion added GaLore, APOLLO, and SCALE comparisons: GDLoRA achieves higher accuracy, lower reported GPU memory, and shorter training time than these baselines on Llama-3-8B/Math10K.
  3. Presentation [Trrg]. We clarified notation, the geometric-to-optimization argument, learning-rate conventions, and figure labels (Section 3; Appendix J).
  4. Reproducibility, numerical validation, and resource reporting [wAtp]. We supplied exact baseline settings, selection/evaluation protocols, dtype specifications, BF16 update measurements, QR/SVD diagnostics, and rank-wise accuracy-memory-time comparisons (Appendices K.4, L, M). The sampled audit found no QR rank underestimation and pooled BF16 write-back errors of 0.84-1.17%. Our implementation is available through the anonymous repository.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.