- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GDLoRA, a fine-tuning method that supplements standard LoRA optimization with updates outside the current adapter-induced first-order weight-space directions. It characterizes the range of the Euclidean-gradient-induced equivalent-gradient operator and shows that this range coincides with the tangent space of the current LoRA parameterization. This characterization yields a two-sided orthogonal projection of the full weight gradient onto the corresponding normal space. GDLoRA reconstructs the full weight gradient from forward activations and backward signals, applies an additional input-count normalization to the normal branch, and directly updates the base weights using the resulting normal component. The LoRA factors retain standard AdamW optimization, and the base-weight branch introduces no additional first- or second-moment optimizer buffers. The appendices analyze finite-step normal displacements of LoRA updates, their reachable set under a factor-step budget, and local properties of direct normal compensation.
Critical strengths
-
The paper connects a precise description of the current adapter-accessible space to an implementable normal-gradient update that leaves the adapter optimizer unchanged (Sec. 3, pp. 3–6). The construction uses thin adapter factors for basis computation and avoids additional base-weight momentum and variance states. Importantly, Appendix E explicitly accounts for the normal displacement already available through the bilinear remainder of a finite LoRA step, and characterizes its rank and nuclear-norm constraints. The authors also distinguish local gradient matching from guarantees about complete optimization trajectories. This combination of an actionable method and careful theoretical scope is consequential to my positive assessment.
-
Tables 1–4 report improvements over same-rank LoRA across all 42 model–task pairs, with average gains of 1.75–2.50 points across the six backbone settings. The experiments include FFT, Fira, and several relevant LoRA variants, with three-seed reporting stated in Appendix G. The Full/Tangent/Normal comparison and the learning-rate sweep in Figure 2 provide meaningful evidence that the result is not explained solely by adding an arbitrary full-gradient branch or selecting one favorable base-update learning rate. Table 9 further shows gains at ranks 8, 16, and 32. The paper also reports the actual cost: 21.6 GB versus 20.4 GB for LoRA in Figure 2(c), together with runtime measurements in Table 6. These controls make the empirical contribution more convincing than a comparison of final task scores alone.
Critical weaknesses
-
Important comparison details remain unspecified, and the practical advantage at a matched total-memory budget is not yet established. Appendix G reports independent learning-rate searches for the language experiments, which is a positive feature, but does not fully specify the selected baseline configurations or the validation and checkpoint-selection protocol. For vision, Table 12 provides GDLoRA's adapter and head learning rates while leaving FFT's base-weight learning rate and its selection unclear. Fira's projection rank, target modules, and residual-scaling configuration are also insufficiently documented. Several evaluation details, including exact checkpoints, task metrics, and reasoning prompts and answer extraction, are needed to make the comparisons reproducible. These omissions matter because the central empirical claim depends on the strength of the FFT and memory-efficient full-rank baselines; they do not, by themselves, demonstrate an unfair comparison. The resource comparison also has a narrower scope than a full practical trade-off analysis. Table 9 reports 70.80 for rank-32 LoRA versus 70.01 for rank-8 GDLoRA, whereas Figure 2(c) compares memory at rank 8. Without memory and runtime measurements across the existing rank sweep, it remains unclear how GDLoRA compares with allocating the additional memory to larger LoRA adapters. This limits the assessment of practical significance beyond the demonstrated same-rank optimizer-state comparison.
-
The fidelity of the numerical normal update to the exact construction needs validation. Tables 10–11 report BF16 training, while Appendix H.1 accumulates full weight gradients and applies an input-count-scaled normal update directly to the base weights. The manuscript does not adequately specify the storage and computation dtypes of the base weights, accumulated gradients, projection buffers, and final weight addition, or whether master weights or another mechanism preserve small increments. BF16 training alone does not determine these choices, so I am not asserting that the updates are rounded away. Nevertheless, the choices matter both for the implemented mechanism and for the memory result in Figure 2(c). There is also an acknowledged limitation in Appendix H.2: unpivoted QR followed by diagonal thresholding need not recover the complete column space of nonzero rank-deficient factors. Consequently, the numerical residual need not be orthogonal to the full adapter-accessible space in such cases, although the exact theorems remain valid. The paper does not report how often this occurs or whether it affects the results. Representative measurements of actual base-weight changes, retained ranks, and projection error against reliable column-space bases would resolve an important gap between the mathematical construction and the implementation.
Questions for authors
-
What exact configurations were used for FFT and Fira, particularly the vision FFT base-weight learning rates and Fira's projection rank, target modules, and residual-scaling settings? Please also specify the validation and checkpoint-selection procedure, exact model checkpoints, GLUE metrics, and reasoning evaluation protocol. Were hyperparameters and checkpoints selected independently of the final evaluation data?
-
What dtypes are used for the base weights, full-gradient accumulation, QR/projection, residual buffers, and final weight addition? Are FP32 master weights or any compensation mechanism used? For a representative reasoning run, can you report the discrepancy between the intended normal step and the actual base-weight change, and explain how all relevant buffers are included in the memory measurement?
-
How often does the thresholded QR procedure in Appendix H.2 discard directions that belong to the factor column spaces during the reported runs? Can you provide retained-rank and orthogonality diagnostics against bases obtained using a reliable thin-factor decomposition, together with a representative check of whether using that decomposition changes accuracy or runtime?
-
Can you report peak memory and training time for the existing LoRA and GDLoRA rank-8/16/32 configurations in Table 9? Given that rank-32 LoRA scores 70.80 while rank-8 GDLoRA scores 70.01, these measurements would materially clarify which configuration is preferable under a fixed total-memory or training-time budget.
8 comments
Response to Reviewer wAtp [Part II]
W2 and Q2: "What dtypes are used for the base weights, full-gradient accumulation, QR/projection, residual buffers, and final weight addition?" "Are FP32 master weights or any compensation mechanism used?"
R2: Thank you. Our BF16 reasoning experiments use FP32 gradient recovery, accumulation, QR and projection, followed by BF16 base-weight write-back, without FP32 master weights or cross-step compensation.
| Operation/state | RoBERTa | Llama-3-8B/Math10K; Gemma-7B/Commonsense170K |
|---|---|---|
| Base storage/forward | FP32 | BF16 weights; mixed-precision forward |
| Full-gradient recovery/microbatch accumulation | FP32 | FP32 |
| Thin-factor QR/bases/projection | FP32 | FP32 |
| Normal-residual buffer | FP32; cleared after each step | FP32; cleared after each step |
| Final base update | FP32 | Cast residual to BF16 before scaling and in-place addition; store in BF16 |
| Additional FP32 master weights | None; base weights are FP32 | None |
| Error feedback/compensated summation | None | None |
Activations/backward signals are cast to FP32 before gradient recovery, which, like QR/projection, runs with autocast disabled. The temporary residual buffer retains no cross-step rounding errors. No base-weight Adam states does not imply zero temporary-buffer memory.
Write-back fidelity. We isolate the normal branch and compare the FP32-residual-based intended increment with before/after stored weights converted to FP64:
\Delta W_t^{\mathrm{actual}}=\operatorname{FP64}(W_{\mathrm{after}})-\operatorname{FP64}(W_{\mathrm{before}}).$$ $$\epsilon_{\mathrm{write}}=\frac{\\|\Delta W_t^{\mathrm{actual}}-\Delta W_t^{\mathrm{intended}}\\|_F}{\\|\Delta W_t^{\mathrm{intended}}\\|_F}.$$ The denominator is the **intended update norm**, not the base-weight norm. On **Llama-3-8B/Math10K, rank 8, seed 42**, we uniformly sample **100 optimizer steps, 6 steps apart across training**. We audit **q/k/v** (`q_proj`, `k_proj`, `v_proj`) in **one-based layers 1/16/32**, not training stages: **9 modules per step, 900 module–step observations**. We report pooled and per-module P95 errors, actual/intended norm ratios, and the fraction of nonzero intended entries rounded to zero. These last entry counts must not be interpreted as lost gradient energy. | Layer | Pooled error (%) | Per-module P95 error (%) | Actual/intended norm ratio | Zero-change entries (%) | |:---:|:---:|:---:|:---:|:---:| | 1 | 0.84 | 1.62 | 0.9993 | 3.8 | | 16 | 1.03 | 1.95 | 0.9988 | 4.6 | | 32 | 1.17 | 2.21 | 0.9985 | 5.3 | Pooled BF16 write-back errors are **0.84–1.17%**, with actual/intended norm ratios of **0.9985–0.9993**. The **3.8–5.3%** zero-change fraction of nonzero intended entries is not lost update energy. These indicate limited aggregate distortion in the sampled layers; R3 separately assesses projection accuracy. **Memory.** Figure 2(c) reports **21.6 GB for GDLoRA versus 22.3 GB for Fira** on Llama-3-8B/Math10K, a reported 0.7 GB (3.1%) difference. The revision reports **peak allocated and peak reserved** memory separately from the existing GPU-memory figures. The stated measurement protocol uses fresh processes under matched hardware, batching, sequence length, accumulation, and training duration. The protocol resets peak counters after model/data loading but before optimizer initialization and the first forward pass, then reads synchronized peaks after the final update. The window includes initial optimizer states, gradient accumulation, QR/SVD workspaces, projection and residual buffers, dtype conversions, every Fira refresh, and GDLoRA write-back; FP64 auditing runs separately. The protocol specifies per-device decimal GB and synchronized first-forward-to-final-update training time, excluding loading, evaluation and checkpointing. | Method | Existing GPU memory (GB) | Peak allocated (GB) | Peak reserved (GB) | |:---|:---:|:---:|:---:| | Fira | 22.3 | 23.4 | 24.0 | | GDLoRA | 21.6 | 21.3 | 21.9 | **The absence of base-weight optimizer moments is structural; the total-memory claim and finite-precision update fidelity require their own measurements.** The rank-wise table in R4 retains its reported GPU-memory and time figures; it is not a rank-by-rank allocated/reserved-memory breakdown. **Changes in the revised manuscript:** The dtype path is in Appendix M.1 (p. 29) and Table 21 (p. 30); write-back definitions and results are in Appendix M.2 (p. 30) and Table 22 (p. 30); memory metrics and the timing protocol are in Appendix M.4 (p. 31) and Table 25 (p. 31). **_`See Part III.`_**Response to Reviewer wAtp [Part III]
W2 and Q3: "How often does the thresholded QR procedure in Appendix H.2 discard directions that belong to the factor column spaces during the reported runs?"
R3: Thank you. Appendix H.2 uses an absolute threshold of for near-zero columns and diagonal filtering after unpivoted QR. The concern is missed numerical column-space directions, not simply retained rank differing from the configured adapter rank. The zero rank of initially zero is not underestimation.
We compare same-step against FP64 thin-factor SVD bases, counting singular values above as reference rank. On Llama-3-8B/Math10K, rank 8, seed 42, we uniformly sample 100 optimizer steps, 6 steps apart across training. Auditing q/k/v (q_proj, k_proj, v_proj) in all 32 layers gives 96 modules per step, 9,600 factor-pair observations. With the same full gradient, we measure
Response to Reviewer wAtp [Part I]
Hi, Reviewer wAtp:
Thank you for recognizing our practical method, finite-step analysis, and broad evaluation.
Your concerns are (i) reproducibility, (ii) numerical fidelity, and (iii) fixed-budget efficiency. We address these concerns below.
W1/Q1: Exact FFT/Fira settings; hyperparameter/checkpoint selection independent of final evaluation?R1: Thank you for requesting these reproducibility details. Our code is public; Tables 7, 10–12 give training settings; Appendix G covers LR searches, three-seed reporting, and Fira refresh intervals.
Selection. All methods select LRs on independent validation data, never final evaluation results. Each language baseline, including FFT/Fira, independently searches 1,3,5,7,9,10×10−5. Vision jointly searches each method's main/head LRs:
DTD fits/selects/evaluates on official train/validation/test splits, never validating on test. Validation: seeded 90/10 train splits for CIFAR-10/100; 10% train holdouts for OxfordPets/StanfordCars; supplied FGVC-Aircraft/RESISC45/EuroSAT splits. Multi-epoch candidates choose and compare checkpoints by peak primary validation metric. GLUE fits official training data with separate LR/checkpoint validation. Math/commonsense: one epoch per candidate; LR selected by independent validation accuracy, never test accuracy.
Final LRs. FFT/Fira use 1e-4 on RoBERTa-Base/Large's six GLUE tasks and Llama-3-8B/Math10K; 3e-5 on Gemma-7B/Commonsense170K. Vision cells give (FFT/Fira main LR, FFT head LR); Fira head optimization/LRs follow GDLoRA (Table 12).
Fira, Llama-3-8B/Math10K. Gradient-projection rank 8 is not adapter rank. Fira directly updates
q_proj.weight,k_proj.weight,v_proj.weightin every Transformer layer; all other parameters are frozen, with no LoRA adapters. One-sided SVD (proj_type="std") projects right for an m×n gradient with m≥n, left otherwise. Refresh: 50 optimizer steps; 100/200-step runs are also saved. Optimizer:betas=(0.9,0.999),eps=1e-6,correct_bias=True, LR=1e-4, weight decay=0.Residual scaling has two stages, not just
R_t=s_t\odot[G_t-\alpha\mathcal B_t(g_t)],\quad\alpha=1.$$ Axis norms and $\odot$ scaling are row-wise for right projection, column-wise for left. Second, from step 2, growth is capped against the previous capped residual: $$c_t=\max\left(1,\frac{\\|R_t\\|_F}{1.01(\\|\bar R_{t-1}\\|_F+10^{-8})}\right),\quad \bar R_t=R_t/c_t,\quad t\ge2.$$ Step 1 is uncapped: $\bar R_1=R_1$. Finally, $$W_{t+1}=W_t-\eta_t[\alpha\mathcal B_t(U_t)+\bar R_t].$$ $\eta_t$ includes Adam bias correction; both branches share it, with no separate residual LR. **Exact revisions** (tokenizers/processors use the same repositories): |Repository|Commit| |---|---| |FacebookAI/roberta-base|e2da8e2f811d1448a5b465c236feacd80ffbac7b| |FacebookAI/roberta-large|722cf37b1afa9454edce342e7895e588b6ff1d59| |NousResearch/Meta-Llama-3-8B|315b20096dc791d381d514deb5f8bd9c8d6d3061| |google/gemma-7b|ff6768d9368919a1f025a54f9f5aa0ee591730bb| |google/vit-base-patch16-224-in21k|b4569560a39a0f1af58e3ddaf17facf20ab919b0| |google/vit-large-patch16-224-in21k|6074eaf2211423e928c93b93ef773d5da618aa7e| GLUE: accuracy for SST-2/MRPC/QNLI/RTE; Matthews correlation for CoLA; Pearson for STS-B. MRPC F1 and STS-B Spearman are also computed. **Reasoning prompt:** ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Response: ``` Generation: `do_sample=False`, `max_new_tokens=256`, `use_cache=True`; no beam search; left padding; `pad_token_id=0`. Parse after the final `### Response:`. Numerical tasks remove commas and extract the last number (absolute error ≤0.001). First matching labels: AQuA A–E; BoolQ lowercase `true/false`; PIQA `solution1/solution2`; SIQA/ARC/OBQA `answer1`–`answer5`; HellaSwag `ending1`–`ending4`; WinoGrande `option1`–`option2`. Unmatched outputs are incorrect; averages include all required tasks. **Revisions (appendix):** Selection: L.1 (p.27); LRs: L.2/Table 19 (p.28); Fira/scaling: L.3 (pp.28–29); model revisions/evaluation: L.4/Table 20 (p.29). **_`See Part II.`_**alpha=1(unrelated tolora_alpha). Let Gt,gt,Ut,Bt denote the full gradient, projected gradient, low-rank Adam direction, and back-projection. First,