acceptodds
Desk rejected at ICLR 2027

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at https://anonymous.4open.science/r/ReGRA-8D6C.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LoRA, residual-gradient adaptation, low-rank optimization, large language models.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mVyMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReGRA freezes a trained rank-8 adapter and adds a second one initialized from residual task gradients. It removes the gradient component jointly aligned with the first update's left/right subspaces and regularizes the Frobenius alignment of the two updates. The aim is better adaptation at a total adapter rank budget of 16.

Critical strengths
  1. The method improves on rank-matched LoRA across encoder and decoder evaluations. Relative to rank-16 LoRA, the GLUE averages improve by 0.77 points on RoBERTa-base and 0.67 on RoBERTa-large. Math and code averages also improve, although the margins are smaller. Both stage-wise updates remain mergeable, so the gains require no additional inference branch.

  2. The math ablation supports a benefit beyond extra training. In Figure 4, gains over rank-8 LoRA are 0.00 points for ordinary two-stage LoRA, 1.00 for residual initialization alone, and 1.31 for full ReGRA in the tested math setting. Under the shared second-stage optimization settings, the extra training stage alone does not explain the result.

Critical weaknesses
  1. The direct control for residual subtraction is missing. Figure 4 compares residual-gradient SVD initialization with random initialization, so it does not distinguish generic gradient guidance from removing the component jointly aligned with the first-stage left and right subspaces. Initializing from the unprojected gradient's leading right singular vectors under the same schedule and scaling would test the specific contribution of ReGRA's subtraction step.

  2. Single-seed results do not establish reliable small gains. Appendix B.1 reports one training seed. Gains over rank-16 LoRA are 0.50 and 0.24 points for the two math averages, 0.27 for MBPP+, and 0.02 for commonsense. Calibration also uses just one example in the Llama math setting (Table 5). Repeated runs and different calibration subsets are needed to judge whether these improvements are reliable.

Questions for authors
  1. Does residual-gradient initialization beat unprojected-gradient SVD initialization across paired seeds and calibration subsets? How much gradient energy does the subtraction remove?
  2. What total training schedule is used for rank-16 LoRA, and does its comparison with ReGRA change when total data exposure is matched?
5 comments
AuthorsOpenReview ↗

Response to Reviewer mVyM

We sincerely appreciate your suggestions, which can greatly help us to refine our paper. We respond to your questions as follows:

W1&Q1: Contribution of residual subtraction.

Following the reviewer’s suggestion, we added an unprojected-gradient SVD control to evaluate the contribution of residual subtraction separately from that of gradient-guided initialization. It uses the leading right singular vectors of the calibration gradient to initialize Stage 2, without subtracting the component aligned with the Stage 1 subspaces. All other settings, including the Stage 1 checkpoint, training seed, calibration subset, adapter scaling, and Stage 2 training schedule, are held fixed.

We repeated the comparison across multiple training seeds and calibration subsets. As shown in Table 1, ReGRA achieved higher mean Math accuracy than unprojected-gradient SVD, with gains of 1.631.63 and 0.290.29 percentage points on Llama-3-8B and Qwen3-8B-Base, respectively. Although residual subtraction removed only 0.02770.0277% and 0.03210.0321% of the calibration gradient energy, it still improved performance on both models. These results suggest that even a small removed component can matter for initialization.

Table 1. Effect of residual subtraction on Math.

ModelUnprojected SVD (%)ReGRA (%)Removed gradient energy (%)
Llama-3-8B70.42 ± 0.1772.05 ± 0.450.0277 ± 0.0017
Qwen3-8B-Base84.48 ± 0.1684.77 ± 0.430.0321 ± 0.0012

W2: Reliability across random seeds and calibration subsets.

We have taken the reviewer’s suggestion into account. We repeated the experiments using three independent training seeds and evaluated different Stage 2 calibration subsets. All results have been updated in the revised submission. Table 2 summarizes the results. ReGRA achieves the highest mean score across all evaluated settings, showing that its strong performance persists across repeated runs.

Table 2. Results across three independent training runs. Values are reported as mean ±\pm sample standard deviation (SD), in percent. GLUE and Math scores are unweighted task averages; code scores are pass@1.

MethodGLUE — RoBERTa-baseGLUE — RoBERTa-largeMath — Llama-3-8BMath — Qwen3-8B-BaseCode — MBPPCode — MBPP+
LoRA (r=8r=8)83.47 ± 0.2487.83 ± 0.1970.79 ± 0.2184.27 ± 0.0866.75 ± 0.7755.73 ± 0.61
LoRA (r=16r=16)84.26 ± 0.2687.96 ± 0.1871.38 ± 0.2484.28 ± 0.7667.98 ± 0.0057.06 ± 0.07
Bi-LoRA84.39 ± 0.0388.09 ± 0.0771.00 ± 0.4281.91 ± 0.5367.28 ± 0.1556.61 ± 0.46
DoRA83.44 ± 0.2387.91 ± 0.2570.95 ± 0.8582.79 ± 1.7266.84 ± 0.5555.73 ± 0.81
KeepLoRA83.45 ± 0.7886.61 ± 0.5270.40 ± 0.4083.21 ± 0.3866.57 ± 0.7755.64 ± 0.67
LoRA-One83.76 ± 0.0587.99 ± 0.1971.35 ± 1.7582.47 ± 1.9266.75 ± 0.5555.73 ± 1.07
PiSSA82.53 ± 0.2586.87 ± 0.1369.24 ± 0.8982.35 ± 1.5567.11 ± 0.3155.37 ± 0.81
ReGRA84.74 ± 0.2888.21 ± 0.2672.05 ± 0.4584.77 ± 0.4368.60 ± 0.3157.22 ± 0.17

Q2: Rank-16 LoRA schedule and matched data exposure.

We thank the reviewer for this question. The original rank-16 LoRA baseline is trained for 2 epochs. ReGRA uses 2 epochs for each stage, totaling 4 training epochs, with gradient calibration between the two stages. We therefore also trained rank-16 LoRA for 4 epochs to match the training data exposure. As shown in Table 3, ReGRA still achieves higher Math accuracy, by 0.38 and 0.86 percentage points on Llama-3-8B and Qwen3-8B-Base.

Table 3. Math accuracy (%) with different training schedules. ReGRA uses 2 epochs per stage.

ModelLoRA (r=16r=16) 2 epochsLoRA (r=16r=16) 4 epochsReGRA 2+2 epochs
Llama-3-8B71.6471.7672.14
Qwen3-8B-Base84.9984.3785.23

Thank you for the additional experiments. The unprojected-gradient control and the multi-seed results are useful and directly address parts of my original concerns. However, several details remain unclear. The response does not specify how many calibration subsets were evaluated, how they were paired across methods, or whether the reported variation reflects training seeds, calibration subsets, or both. The compute-matched comparison is limited to Math and is reported without uncertainty estimates; its values also differ from the corresponding multi-seed results without explanation. Moreover, matching the number of epochs does not account for the additional gradient-calibration and SVD costs, and the lower four-epoch Qwen LoRA result suggests that the longer baseline schedule may require retuning. The commonsense setting is absent from the multi-seed evaluation, while the 0.29-point Qwen improvement over unprojected SVD remains difficult to assess without paired seed-level results.

These additions reduce my concerns but do not fully resolve them, so I will keep my rating unchanged. That said, I would not object to acceptance if the other reviewers and the AC consider the remaining issues sufficiently minor.

AuthorsOpenReview ↗

Response to Reviewer mVyM

We sincerely thank you for the prompt follow-up comments and for recognizing the value of our additional experiments. We would like to provide further details regarding the experimental settings and computational overhead.

1. Calibration subset and paired evaluation. Calibration data are randomly sampled from the training set by shuffling with each seed and taking the first few batches. For the same seed, both methods use exactly the same sampled training data and calibration subset. We evaluate three different random seeds, and the reported standard deviations reflect the combined effects of training randomness and calibration sampling. Further experimental details are provided in the appendix B.2.

2. Computational overhead. Under matched total optimization steps, our theoretical FLOPs estimate for Llama-3-8B on Math is slightly lower for ReGRA than for rank-16 LoRA(16.4% reduction excluding the shared backbone cost), including calibration, SVD, and regularization costs. We provide the detailed calculations and assumptions in the appendix B.7. The original matched-schedule Math comparison used a single seed. We have now repeated it across three random seeds, with rank-16 LoRA trained for four epochs and ReGRA for two epochs per stage. Table 1 reports the updated mean accuracy.

Table 1. Math accuracy (%) with different training schedules. ReGRA uses 2 epochs per stage.

ModelLoRA (r=16r=16) 2 epochsLoRA (r=16r=16) 4 epochsReGRA 2+2 epochs
Llama-3-8B71.3871.7872.05
Qwen3-8B-Base84.2984.2884.77

3.Multi-seed experiments for commonsense setting We initially prioritized GLUE because it also evaluates language understanding. We have now completed the multi-seed commonsense evaluation. As shown in Table 2, ReGRA performs well, with experimental details provided in the appendix C.1.

Table 2. Commonsense average accuracy (%), reported as mean ± sample SD across three seeds.

MethodAverage accuracy (%)
LoRA (r=8r=8)85.11 ± 0.13
LoRA (r=16r=16)85.99 ± 0.10
Bi-LoRA84.12 ± 0.70
DoRA85.37 ± 0.43
KeepLoRA81.86 ± 0.12
LoRA-One85.39 ± 0.47
PiSSA83.37 ± 0.53
ReGRA86.06 ± 0.22

4.Paired seed-level results on Qwen. We provide the paired results across three training seeds in Table 3. ReGRA achieves higher accuracy in all three runs.

Table 3. Paired Qwen Math accuracy (%) and differences across three training seeds. SD denotes sample standard deviation.

Training seedUnprojected SVD (%)ReGRA (%)ReGRA − SVD (%)
4284.6485.23+0.59
8984.3384.38+0.05
15207284.4784.69+0.22
Mean ± SD84.48 ± 0.1684.77 ± 0.43+0.29 ± 0.28

All the details and additional results described above are included in the revised manuscript and appendix. We hope these clarifications are helpful. Please let us know if you have any further questions or concerns that we can address.

Reviewer_zNJyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReGRA, a two-stage LoRA fine-tuning method that aims to make better use of a fixed rank budget. The method first trains a standard LoRA adapter and then uses residual task gradients to initialize a second adapter. An orthogonality regularizer encourages the two updates to learn complementary directions. The authors evaluate ReGRA on natural language understanding, mathematical reasoning, and code generation tasks. The results show improvements over several LoRA baselines, supported by ablation studies and additional analyses of singular components.

Critical strengths

The paper addresses an interesting problem of improving rank utilization in LoRA fine-tuning. The idea of using residual gradients to guide a second-stage adapter is intuitive and well motivated. The experiments cover several tasks and model families, and ReGRA shows improvements over the tested baselines. I also appreciate the ablation studies and the analysis of individual singular components, which provide useful insights into the proposed method.

Critical weaknesses

1. Unclear fairness of the training budget comparison. ReGRA uses two training stages, each with a full task-specific training schedule. It also requires additional gradient calibration and SVD computation. Although the paper compares ReGRA with rank-16 LoRA under the same total rank budget, it does not provide a clear compute-matched comparison. It is therefore difficult to determine how much of the improvement comes from the proposed method rather than additional training.

2. Limited statistical validation. The experiments use only one random seed, and several improvements over rank-16 LoRA are relatively small. For example, the gains on mathematical reasoning and commonsense reasoning are modest. Without multiple seeds or uncertainty estimates, it is difficult to assess whether these improvements are consistent and statistically reliable.

3. Insufficient evidence for learned subspace complementarity. The residual-gradient projection removes only the component jointly aligned with the two Stage-1 subspaces. The orthogonality regularizer also penalizes only the inner product between the two updates. These operations do not guarantee that the learned subspaces remain complementary. The paper would benefit from more direct measurements of subspace overlap and comparisons with related sequential LoRA methods.

Questions for authors
  1. Can the authors provide comparisons under matched training steps or wall-clock budgets? In particular, how does ReGRA compare with rank-16 LoRA when both methods receive the same total training computation?

  2. Can the authors report results across multiple random seeds? This would help establish whether the relatively small improvements over rank-16 LoRA are robust.

  3. Can the authors directly measure the overlap between the learned Stage-1 and Stage-2 subspaces? It would also be helpful to compare ReGRA with sequential adaptation methods such as ReLoRA and Chain of LoRA.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zNJy

We sincerely thank the reviewer for the constructive and insightful comments.These comments have helped us identify several aspects that require further investigation and clarification.We address each concern point by point below.

W1&Q1: Fairness of the training budget comparison. We thank the reviewer for this helpful suggestion. Following this suggestion, we additionally evaluated rank-16 LoRA under a training budget matched to ReGRA. As shown in Table 1, ReGRA achieves better performance than rank-16 LoRA under the same training budget.

Table 1. Performance on math.

ModelLoRA (r=16r=16)ReGRA
llama3-8b71.7672.14
qwen3-8b-base84.3785.23

W2&Q2: Statistical validation. We thank the reviewer for highlighting this concern. To assess the robustness of our results, we repeated the mathematical reasoning, code generation, and GLUE experiments using three independent random seeds for each setting. All results have been updated in the revised submission. Table 2 reports the mean and standard deviation across these runs, showing that ReGRA maintains strong performance across different seeds.

Table 2. Results across three independent training runs. Values are reported as mean ±\pm sample standard deviation (SD), in percent. GLUE and Math scores are unweighted task averages; code scores are pass@1.

MethodGLUE — RoBERTa-baseGLUE — RoBERTa-largeMath — Llama-3-8BMath — Qwen3-8B-BaseCode — MBPPCode — MBPP+
LoRA (r=8r=8)83.47 ± 0.2487.83 ± 0.1970.79 ± 0.2184.27 ± 0.0866.75 ± 0.7755.73 ± 0.61
LoRA (r=16r=16)84.26 ± 0.2687.96 ± 0.1871.38 ± 0.2484.28 ± 0.7667.98 ± 0.0057.06 ± 0.07
Bi-LoRA84.39 ± 0.0388.09 ± 0.0771.00 ± 0.4281.91 ± 0.5367.28 ± 0.1556.61 ± 0.46
DoRA83.44 ± 0.2387.91 ± 0.2570.95 ± 0.8582.79 ± 1.7266.84 ± 0.5555.73 ± 0.81
KeepLoRA83.45 ± 0.7886.61 ± 0.5270.40 ± 0.4083.21 ± 0.3866.57 ± 0.7755.64 ± 0.67
LoRA-One83.76 ± 0.0587.99 ± 0.1971.35 ± 1.7582.47 ± 1.9266.75 ± 0.5555.73 ± 1.07
PiSSA82.53 ± 0.2586.87 ± 0.1369.24 ± 0.8982.35 ± 1.5567.11 ± 0.3155.37 ± 0.81
ReGRA84.74 ± 0.2888.21 ± 0.2672.05 ± 0.4584.77 ± 0.4368.60 ± 0.3157.22 ± 0.17

W3&Q3: Subspace complementarity.

We thank the reviewer for the suggestion. Table 3 compares ReGRA with ReLoRA and Chain of LoRA on Math across three training seeds per model. ReGRA achieves the highest mean accuracy on both models.

To quantify subspace complementarity, we compute a compact SVD of each stage's update, ΔWi=UiΣiVi⊤\Delta W_i = U_i \Sigma_i V_i^\top. Let rir_i be the rank of ΔWi\Delta W_i and let k=min⁡(r1,r2)k = \min(r_1, r_2). The left and right subspace overlaps are OL=∥U1⊤U2∥F2/kO_L = \lVert U_1^\top U_2 \rVert_F^2 / k and OR=∥V1⊤V2∥F2/kO_R = \lVert V_1^\top V_2 \rVert_F^2 / k, respectively. The two-sided overlap is Otwo=∥PLΔW2PR∥F2/∥ΔW2∥F2O_{\mathrm{two}} = \lVert P_L \Delta W_2 P_R \rVert_F^2 / \lVert \Delta W_2 \rVert_F^2, where PL=U1U1⊤P_L = U_1 U_1^\top and PR=V1V1⊤P_R = V_1 V_1^\top. It measures the fraction of Stage-2 update energy that lies jointly in the Stage-1 left and right subspaces.

Table 4 reports overlap averaged over the Q/K/V update matrices within each run and summarized across three training seeds. In our experiments, ReLoRA and Chain of LoRA randomly initialize their Stage-2 adapter directions, which may contribute to their lower overlap. ReGRA uses the calibration gradient to guide Stage-2 initialization. Its residual subtraction removes the gradient component that lies jointly in both Stage-1 subspaces, allowing the model to use useful components aligned with either side. After training, ReGRA's mean two-sided overlap remains below 0.30.3% on both models. These results show that ReGRA can achieve higher Math accuracy in this comparison while keeping the fraction of Stage-2 update energy in the joint Stage-1 subspace small.

Table 3. Math accuracy (%) for sequential low-rank adaptation methods. Each run's accuracy is averaged equally over six datasets. Values are mean ±\pm sample SD across three training seeds.

ModelReLoRAChain of LoRAReGRA
Llama-3-8B69.27 ± 0.6164.73 ± 5.3972.05 ± 0.45
Qwen3-8B-Base82.30 ± 0.5183.09 ± 0.1584.77 ± 0.43

Table 4. Subspace overlap (%) between the first two stages on Math. For each run, overlap is averaged equally over the Q/K/V update matrices. Values are mean ±\pm sample SD across three training seeds.

MethodLlama-3-8B — LeftLlama-3-8B — RightLlama-3-8B — Two-sidedQwen3-8B-Base — LeftQwen3-8B-Base — RightQwen3-8B-Base — Two-sided
ReLoRA3.1340.3090.01912.5230.3750.0265
Chain of LoRA2.1470.1940.00451.8840.1950.0036
ReGRA9.9511.8770.296011.2621.5370.2010

Dear Reviewer,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

AuthorsOpenReview ↗

Response to Reviewer zNJy

Dear Reviewer zNJy,

As the rebuttal discussion period is approaching its end, we would greatly appreciate your final feedback on our responses. If our clarifications and additional experiments have adequately addressed your concerns, we kindly invite you to reconsider and update your scores accordingly.

If any concerns or questions remain, please let us know. We will do our best to address them promptly during the remaining discussion period to resolve any outstanding issues.

Thank you again for your time, consideration, and valuable feedback.

Reviewer_ZwyDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that LoRA rank components contribute unevenly and that increasing rank does not monotonically improve accuracy. It proposes ReGRA: train a standard LoRA (Stage 1), compute task gradients on a small calibration set at the Stage-1 solution, project out the two-sided subspace already captured by Stage 1, and use the residual gradient to initialize a second adapter. Stage 2 freezes Stage 1 and trains only the second adapter with an orthogonality regularizer. Both updates are merged after training, so inference adds no cost. Experiments cover GLUE, mathematical reasoning, code generation, and commonsense reasoning. ReGRA generally beats rank-8 LoRA and is comparable to or slightly better than rank-16 LoRA under a similar total rank budget.

Critical strengths
  1. Well-motivated problem. The observation that LoRA singular components are functionally unequal, and that more rank is not always better, is interesting and practically relevant.
  2. Simple, mergeable method. The residual-gradient projection idea is clean, and the merged model has no inference overhead.
  3. Broad evaluation. Four task families, encoder and decoder backbones, multiple baselines.
  4. Honest ablation. Figure 4 separates residual initialization from orthogonality regularization and shows the former is the main driver.
Critical weaknesses
  1. The benefit of two-stage training is not isolated. The paper’s own ablation shows that plain two-stage LoRA (no residual init, no orthogonal reg.) gives 0.00 gain over rank-8 LoRA. The missing key baseline is a single-stage rank-16 LoRA initialized with the same residual directions and trained jointly. Without it, the paper cannot show that the two-stage design—rather than residual initialization or extra training—is responsible for the gains.
  2. Compute is not matched. Both stages use the same epoch count, so ReGRA likely receives roughly twice the training compute of rank-16 LoRA. A compute-matched comparison is needed to rule out extra training as an explanation.
  3. Gains are modest and often within noise. Over rank-16 LoRA: GLUE base +0.77, GLUE large +0.67, Math Llama +0.50, Math Qwen +0.24, MBPP +0.80, MBPP+ +0.27, Commonsense +0.02. Some GLUE variances are large (CoLA ±3.56), and the ablation uses one seed. Multi-seed significance testing is needed.
Questions for authors
  1. Can you provide a single-stage rank-16 LoRA with the same residual initialization, jointly trained? How does it compare with ReGRA under matched compute?
  2. Can you provide a compute-matched comparison (same total steps/FLOPs) against rank-16 LoRA?
  3. Are the Qwen math (+0.24) and commonsense (+0.02) gains significant across multiple seeds?
  4. What happens if Stage 1 is allowed to fine-tune slightly during Stage 2, or both stages are jointly optimized after residual init?
  5. Could the authors compare with Bi-LoRA [1]? It also shares a dual lora module design.

[1] Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models. ICLR'26

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZwyD

We sincerely thank the reviewer for the valuable time and constructive suggestions, which have helped us improve the clarity and completeness of our paper. Below we provide point-by-point responses and clarify the improvements made in the revised version.

W1&Q1: The benefit of two-stage training.

We thank the reviewer for this suggestion. Our two-stage design is motivated by the fact that Stage 2 relies on the adaptation ranks learned in Stage 1 to compute its residual initialization. If both adapters are trained jointly in a single stage after this initialization, The resulting approach is similar to LoRA-One, differing only in the initialization strategy. We also conducted the experiment suggested by the reviewer and added the results to the ablation study, as shown in Table 1. ReGRA’s two-stage training helps identify more effective adaptation rank, leading to better model performance.

Table 1. Performance on math.

ModelLoRA (r=16r=16)ReGRA
llama3-8b71.6272.14
qwen3-8b-base83.9785.23

W2&Q2: Training cost and compute-matched comparison.

We appreciate the reviewer’s recommendation. To compare performance under the same training budget, we additionally evaluated rank-16 LoRA with a training budget matched to ReGRA. As shown in Table 2, ReGRA achieves better performance under this setting.

Table 2. Performance on math under the same training budget.

ModelLoRA (r=16r=16)ReGRA
llama3-8b71.7672.14
qwen3-8b-base84.3785.23

W3&Q3: Reliability of the gains across random seeds.

We thank the reviewer for this helpful suggestion. To check the reliability of the results, we repeated the experiments on mathematical reasoning, code generation, and GLUE using three independent training seeds. All results have been updated in the revised submission. Table 3 reports the mean and standard deviation across three independent runs. ReGRA achieves the highest mean score in all six evaluated settings, showing that its strong performance persists across different training seeds.

Table 3. Results across three independent training runs. Values are reported as mean ±\pm sample standard deviation (SD), in percent. GLUE and Math scores are unweighted task averages; code scores are pass@1.

MethodGLUE — RoBERTa-baseGLUE — RoBERTa-largeMath — Llama-3-8BMath — Qwen3-8B-BaseCode — MBPPCode — MBPP+
LoRA (r=8r=8)83.47 ± 0.2487.83 ± 0.1970.79 ± 0.2184.27 ± 0.0866.75 ± 0.7755.73 ± 0.61
LoRA (r=16r=16)84.26 ± 0.2687.96 ± 0.1871.38 ± 0.2484.28 ± 0.7667.98 ± 0.0057.06 ± 0.07
Bi-LoRA84.39 ± 0.0388.09 ± 0.0771.00 ± 0.4281.91 ± 0.5367.28 ± 0.1556.61 ± 0.46
DoRA83.44 ± 0.2387.91 ± 0.2570.95 ± 0.8582.79 ± 1.7266.84 ± 0.5555.73 ± 0.81
KeepLoRA83.45 ± 0.7886.61 ± 0.5270.40 ± 0.4083.21 ± 0.3866.57 ± 0.7755.64 ± 0.67
LoRA-One83.76 ± 0.0587.99 ± 0.1971.35 ± 1.7582.47 ± 1.9266.75 ± 0.5555.73 ± 1.07
PiSSA82.53 ± 0.2586.87 ± 0.1369.24 ± 0.8982.35 ± 1.5567.11 ± 0.3155.37 ± 0.81
ReGRA84.74 ± 0.2888.21 ± 0.2672.05 ± 0.4584.77 ± 0.4368.60 ± 0.3157.22 ± 0.17

Q4: Train together during stage2. Thank you for this suggestion. We examine a joint optimization variant using the same rank-8 Stage 1 checkpoint as ReGRA. After residual initialization of Stage 2, we allow both adapters to update during Stage 2 and evaluate them using the same protocol. The joint-training variant achieves 70.51% average accuracy, below both ReGRA (72.14%) and the Stage 1 checkpoint alone (70.83%). This suggests that updating both adapters during Stage 2 may disturb the useful rank learned in Stage 1, rather than consistently improving upon them.

Q5: Comparison with Bi-LoRA. We thank the reviewer for this valuable suggestion. We agree that Bi-LoRA is a relevant baseline given its dual-LoRA architecture. Following the reviewer's suggestion, we reproduced Bi-LoRA and incorporated it into our comparative experiments. The results are reported in Table 4, providing a direct comparison between ReGRA and Bi-LoRA.

Table 4. Comparison of LoRA (r=16r=16), Bi-LoRA, and ReGRA. Scores are mean ±\pm sample SD (%). GLUE and Math use unweighted averages, and Code reports Pass@1.

MethodGLUE — RoBERTa-baseGLUE — RoBERTa-largeMath — Llama-3-8BMath — Qwen3-8B-BaseCode — MBPP+ BaseCode — MBPP+ Plus
LoRA (r=16r=16)84.26 ± 0.2687.96 ± 0.1871.38 ± 0.2484.28 ± 0.7667.98 ± 0.0057.06 ± 0.07
Bi-LoRA84.39 ± 0.0388.09 ± 0.0771.00 ± 0.4281.91 ± 0.5367.28 ± 0.1556.61 ± 0.46
ReGRA84.74 ± 0.2888.21 ± 0.2672.05 ± 0.4584.77 ± 0.4368.60 ± 0.3157.22 ± 0.17

Dear Reviewer,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

AuthorsOpenReview ↗

Response to Reviewer ZwyD

Dear Reviewer ZwyD,

As the rebuttal discussion period is approaching its end, we would greatly appreciate your final feedback on our responses. If our clarifications and additional experiments have adequately addressed your concerns, we kindly invite you to reconsider and update your scores accordingly.

If any concerns or questions remain, please let us know. We will do our best to address them promptly during the remaining discussion period to resolve any outstanding issues.

Thank you again for your time, consideration, and valuable feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comments from Authors

Dear PCs, SACs, ACs, and Reviewers,

We sincerely thank all PCs, SACs, ACs, and reviewers for their valuable time and constructive feedback. We are encouraged by the positive recognition from all three reviewers. Specifically, Reviewer ZwyD highlighted the well-motivated problem, clean and mergeable design, and informative ablations. Reviewer zNJy recognized the importance of rank utilization and described our residual-gradient initialization as intuitive and well motivated. Reviewer mVyM acknowledged the empirical improvements over rank-matched LoRA and the absence of additional inference overhead.

Meanwhile, the reviewers raised concerns regarding compute-budget fairness, statistical reliability, and subspace complementarity. We carefully addressed these concerns through additional experiments and analyses. Notably, Reviewer mVyM subsequently confirmed that our additional experiments and clarifications addressed the remaining questions and stated that they had no further questions.

The major improvements in our revised manuscript include:

  • Multi-seed Validation: Three-seed experiments across GLUE, Math, and Code, together with additional commonsense evaluation and paired seed-level comparisons.
  • Compute-Budget Comparison: Matched-schedule experiments and theoretical FLOPs analysis, demonstrating better Math performance with lower estimated non-shared computational cost.
  • Additional Ablations and Baselines: Joint-training and unprojected-gradient controls, comparisons with Bi-LoRA, ReLoRA, and Chain of LoRA, and quantitative subspace-overlap analysis.

Overall, ReGRA achieves strong performance across GLUE, Math, and Code. Although improvements on commonsense reasoning are more modest, this task relies more heavily on pretrained knowledge, leaving less room for task-specific adaptation.

With the growing use of large language models, parameter-efficient fine-tuning has become increasingly important, and LoRA is widely adopted. ReGRA uses residual gradients in a two-stage process to learn complementary low-rank directions. We believe this highlights that effective LoRA adaptation depends not only on rank size, but also on whether its rank components capture useful and complementary directions.

We sincerely appreciate your time, feedback, and consideration.

Sincerely,

The Authors

AuthorsOpenReview ↗

Confidential Response

Dear ACs, PCs and SACs,

We sincerely appreciate the reviewers' time, thoughtful feedback, and constructive discussions throughout the rebuttal process. We are encouraged that most of the original concerns have been addressed through our additional experiments and clarifications. We understand that the remaining concerns are relatively minor and mainly relate to two aspects: empirical performance and computational efficiency. We would like to further clarify these points to avoid potential misunderstandings and ensure that our contributions are fully understood and fairly evaluated.

  • Empirical Performance: The concern about modest gains mainly applies to commonsense reasoning, rather than our overall results. ReGRA achieves the highest mean performance across all six main evaluation settings. Even on commonsense, it slightly outperforms rank-16 LoRA. The smaller gain is due to the task's stronger reliance on pretrained knowledge, rather than a general limitation of our method.

  • Computational Efficiency: We believe the concern regarding computational efficiency may stem from focusing on the two-stage structure rather than the overall computational cost. In fact, under matched training schedules, ReGRA achieves better Math performance than rank-16 LoRA. Moreover, our theoretical FLOPs analysis demonstrates a 16.4% reduction in non-shared computation, including calibration, SVD, and regularization costs. These results demonstrate that ReGRA achieves better adaptation performance with improved computational efficiency.

We have incorporated all these clarifications and additional results into the revised manuscript.

Importantly, the reviewers' follow-up comments suggest that these are limited remaining concerns rather than fundamental issues with our method. Reviewer zNJy acknowledged that our additional analyses addressed much of the original concerns and maintained a supportive recommendation, while Reviewer mVyM confirmed that the remaining questions had been addressed and stated that they had no further questions.

We sincerely hope these additional results and clarifications will be taken into consideration in the final assessment of our work.

Thank you for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.