- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReGRA freezes a trained rank-8 adapter and adds a second one initialized from residual task gradients. It removes the gradient component jointly aligned with the first update's left/right subspaces and regularizes the Frobenius alignment of the two updates. The aim is better adaptation at a total adapter rank budget of 16.
Critical strengths
-
The method improves on rank-matched LoRA across encoder and decoder evaluations. Relative to rank-16 LoRA, the GLUE averages improve by 0.77 points on RoBERTa-base and 0.67 on RoBERTa-large. Math and code averages also improve, although the margins are smaller. Both stage-wise updates remain mergeable, so the gains require no additional inference branch.
-
The math ablation supports a benefit beyond extra training. In Figure 4, gains over rank-8 LoRA are 0.00 points for ordinary two-stage LoRA, 1.00 for residual initialization alone, and 1.31 for full ReGRA in the tested math setting. Under the shared second-stage optimization settings, the extra training stage alone does not explain the result.
Critical weaknesses
-
The direct control for residual subtraction is missing. Figure 4 compares residual-gradient SVD initialization with random initialization, so it does not distinguish generic gradient guidance from removing the component jointly aligned with the first-stage left and right subspaces. Initializing from the unprojected gradient's leading right singular vectors under the same schedule and scaling would test the specific contribution of ReGRA's subtraction step.
-
Single-seed results do not establish reliable small gains. Appendix B.1 reports one training seed. Gains over rank-16 LoRA are 0.50 and 0.24 points for the two math averages, 0.27 for MBPP+, and 0.02 for commonsense. Calibration also uses just one example in the Llama math setting (Table 5). Repeated runs and different calibration subsets are needed to judge whether these improvements are reliable.
Questions for authors
- Does residual-gradient initialization beat unprojected-gradient SVD initialization across paired seeds and calibration subsets? How much gradient energy does the subtraction remove?
- What total training schedule is used for rank-16 LoRA, and does its comparison with ReGRA change when total data exposure is matched?
5 comments
Thank you for the additional experiments. The unprojected-gradient control and the multi-seed results are useful and directly address parts of my original concerns. However, several details remain unclear. The response does not specify how many calibration subsets were evaluated, how they were paired across methods, or whether the reported variation reflects training seeds, calibration subsets, or both. The compute-matched comparison is limited to Math and is reported without uncertainty estimates; its values also differ from the corresponding multi-seed results without explanation. Moreover, matching the number of epochs does not account for the additional gradient-calibration and SVD costs, and the lower four-epoch Qwen LoRA result suggests that the longer baseline schedule may require retuning. The commonsense setting is absent from the multi-seed evaluation, while the 0.29-point Qwen improvement over unprojected SVD remains difficult to assess without paired seed-level results.
These additions reduce my concerns but do not fully resolve them, so I will keep my rating unchanged. That said, I would not object to acceptance if the other reviewers and the AC consider the remaining issues sufficiently minor.
Response to Reviewer mVyM
We sincerely thank you for the prompt follow-up comments and for recognizing the value of our additional experiments. We would like to provide further details regarding the experimental settings and computational overhead.
1. Calibration subset and paired evaluation. Calibration data are randomly sampled from the training set by shuffling with each seed and taking the first few batches. For the same seed, both methods use exactly the same sampled training data and calibration subset. We evaluate three different random seeds, and the reported standard deviations reflect the combined effects of training randomness and calibration sampling. Further experimental details are provided in the appendix B.2.
2. Computational overhead. Under matched total optimization steps, our theoretical FLOPs estimate for Llama-3-8B on Math is slightly lower for ReGRA than for rank-16 LoRA(16.4% reduction excluding the shared backbone cost), including calibration, SVD, and regularization costs. We provide the detailed calculations and assumptions in the appendix B.7. The original matched-schedule Math comparison used a single seed. We have now repeated it across three random seeds, with rank-16 LoRA trained for four epochs and ReGRA for two epochs per stage. Table 1 reports the updated mean accuracy.
Table 1. Math accuracy (%) with different training schedules. ReGRA uses 2 epochs per stage.
| Model | LoRA () 2 epochs | LoRA () 4 epochs | ReGRA 2+2 epochs |
|---|---|---|---|
| Llama-3-8B | 71.38 | 71.78 | 72.05 |
| Qwen3-8B-Base | 84.29 | 84.28 | 84.77 |
3.Multi-seed experiments for commonsense setting We initially prioritized GLUE because it also evaluates language understanding. We have now completed the multi-seed commonsense evaluation. As shown in Table 2, ReGRA performs well, with experimental details provided in the appendix C.1.
Table 2. Commonsense average accuracy (%), reported as mean ± sample SD across three seeds.
| Method | Average accuracy (%) |
|---|---|
| LoRA () | 85.11 ± 0.13 |
| LoRA () | 85.99 ± 0.10 |
| Bi-LoRA | 84.12 ± 0.70 |
| DoRA | 85.37 ± 0.43 |
| KeepLoRA | 81.86 ± 0.12 |
| LoRA-One | 85.39 ± 0.47 |
| PiSSA | 83.37 ± 0.53 |
| ReGRA | 86.06 ± 0.22 |
4.Paired seed-level results on Qwen. We provide the paired results across three training seeds in Table 3. ReGRA achieves higher accuracy in all three runs.
Table 3. Paired Qwen Math accuracy (%) and differences across three training seeds. SD denotes sample standard deviation.
| Training seed | Unprojected SVD (%) | ReGRA (%) | ReGRA − SVD (%) |
|---|---|---|---|
| 42 | 84.64 | 85.23 | +0.59 |
| 89 | 84.33 | 84.38 | +0.05 |
| 152072 | 84.47 | 84.69 | +0.22 |
| Mean ± SD | 84.48 ± 0.16 | 84.77 ± 0.43 | +0.29 ± 0.28 |
All the details and additional results described above are included in the revised manuscript and appendix. We hope these clarifications are helpful. Please let us know if you have any further questions or concerns that we can address.
Response to Reviewer mVyM
We sincerely appreciate your suggestions, which can greatly help us to refine our paper. We respond to your questions as follows:
W1&Q1: Contribution of residual subtraction.
Following the reviewer’s suggestion, we added an unprojected-gradient SVD control to evaluate the contribution of residual subtraction separately from that of gradient-guided initialization. It uses the leading right singular vectors of the calibration gradient to initialize Stage 2, without subtracting the component aligned with the Stage 1 subspaces. All other settings, including the Stage 1 checkpoint, training seed, calibration subset, adapter scaling, and Stage 2 training schedule, are held fixed.
We repeated the comparison across multiple training seeds and calibration subsets. As shown in Table 1, ReGRA achieved higher mean Math accuracy than unprojected-gradient SVD, with gains of 1.63 and 0.29 percentage points on Llama-3-8B and Qwen3-8B-Base, respectively. Although residual subtraction removed only 0.0277 and 0.0321 of the calibration gradient energy, it still improved performance on both models. These results suggest that even a small removed component can matter for initialization.
Table 1. Effect of residual subtraction on Math.
W2: Reliability across random seeds and calibration subsets.
We have taken the reviewer’s suggestion into account. We repeated the experiments using three independent training seeds and evaluated different Stage 2 calibration subsets. All results have been updated in the revised submission. Table 2 summarizes the results. ReGRA achieves the highest mean score across all evaluated settings, showing that its strong performance persists across repeated runs.
Table 2. Results across three independent training runs. Values are reported as mean ± sample standard deviation (SD), in percent. GLUE and Math scores are unweighted task averages; code scores are pass@1.
Q2: Rank-16 LoRA schedule and matched data exposure.
We thank the reviewer for this question. The original rank-16 LoRA baseline is trained for 2 epochs. ReGRA uses 2 epochs for each stage, totaling 4 training epochs, with gradient calibration between the two stages. We therefore also trained rank-16 LoRA for 4 epochs to match the training data exposure. As shown in Table 3, ReGRA still achieves higher Math accuracy, by 0.38 and 0.86 percentage points on Llama-3-8B and Qwen3-8B-Base.
Table 3. Math accuracy (%) with different training schedules. ReGRA uses 2 epochs per stage.