- Rating
- 2(+1)
- Confidence
- 2(+1)
- Informativeness
- 2(+1)
- Presentation
- 3(+2)
- Soundness
- 2(+1)
Read review
Paper summary
RSLoRA allocates adapter capacity using the representational dissimilarity between each LoRA branch and its corresponding pretrained branch, measured by linear CKA. It combines the dissimilarity with its temporal variation, periodically removes low-sensitivity component from low-scoring modules, and adds components to high-scoring modules. The paper evaluates the method on GLUE, commonsense reasoning and few-shot image classification.
Critical strengths
I would like to thank the authors. The paper introduces an interesting allocation signal that explicitly compares each adapter with the pretrained mapping it modifies. This provides a useful perspective beyond measuring properties of the adapter alone.
The empirical results cover three backbones and different task families, with means and standard deviations. RSLoRA improves the GLUE average from 88.8 for AdaLoRA to 89.6, and commonsense reasoning accuracy from 86.9 to 87.5 at rank 8 and from 86.5 to 88.1 at rank 32. The visual experiments provide additional evidence of generality
The revised paper includes substantial ablations. Comparisons with random allocation, frozen scores, alternative metrics, and shuffled pretrained references help establish the contribution of the proposed signal. The controlled rank-addition experiment in Appendix H.3 provides more direct evidence that high-scoring modules benefit from additional capacity.
Critical weaknesses
Incomplete specification of the controlled intervention. Appendix H.3 does not report the number and identities of selected modules, the checkpoint’s training stage, or the precise composition of the held-out QA set. Adding the same number of rank components does not necessarily add the same number of parameters when module dimensions differ. Actual added parameter counts should be reported before describing the branches as having matched capacity.
Incomplete isolation of the allocation benefit. Compared with standard LoRA, RSLoRA also introduces an SVD-inspired parameterization and orthogonality regularization. The random-allocation controls are useful, but a uniform-rank baseline using the identical parameterization, initialization, regularization, and optimizer settings would clarify how much of the improvement comes specifically from redistribution.
Understated computational overhead. The efficiency discussion emphasizes comparisons with AdaLoRA while Table 13 reports approximately 1.9 times the training runtime on CoLA and 1.35 times on LLaMA-3-8B relative to standard LoRA. This is a meaningful practical cost. The FLOPs accounting should explicitly state whether calibration passes, similarity computation, and allocation operations are included.
Missing relevant baseline. The experiments omit some newer/better baselines, (PRILoRA for example), which combines nonuniform rank allocation with pruning and evaluates on the same DeBERTaV3-base and GLUE benchmarks. A comparison under matched parameter budgets, evaluation metrics, and tuning effort would strengthen the claim that RSLoRA improves over existing PEFT methods.
Limited verification of parameter-budget comparability. Appendix C.2 correctly acknowledges that preserving total rank does not preserve the parameter count. However, Table 6 audits only sampled allocation events from two rank-8 runs. Full-run minimum, maximum, and final parameter counts, including rank 32, would better substantiate the resource comparisons.
Questions for authors
See weaknesses please .
2 comments
W4. Comparison with PRILoRA.
We added PRILoRA to the GLUE comparison with a matched initial parameter budget of approximately 1.9M trainable parameters, using the same task-specific training hyperparameters and evaluation metrics as the other methods. No additional PRILoRA-specific hyperparameter search was performed.
As reported in the table, RSLoRA achieves an average score of 89.6, compared with 87.9 for PRILoRA, an improvement of 1.7 points. These results further support RSLoRA's effectiveness against a recent baseline.
| Method | Param. | MNLI (Acc.) | SST-2 (Acc.) | CoLA (Mcc.) | QQP (Acc.-F1) | QNLI (Acc.) | RTE (Acc.) | MRPC (Acc.-F1) | STS-B (Corr.) | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| PRILoRA | 1.9M | 89.6 ± 0.2 | 94.4 ± 0.3 | 67.7 ± 1.4 | 90.7 ± 0.1 | 93.9 ± 0.1 | 85.6 ± 1.9 | 91.5 ± 1.0 | 90.0 ± 0.8 | 87.9 |
| RSLoRA | 1.9M | 90.7 ± 0.1 | 96.4 ± 0.1 | 71.2 ± 0.6 | 90.8 ± 0.1 | 94.6 ± 0.2 | 88.5 ± 0.3 | 92.7 ± 0.2 | 91.7 ± 0.1 | 89.6 |
Revisions to the manuscript.
The PRILoRA comparison is included in Table 1, with per-task means and standard deviations reported in Appendix I.1 (Table 18). The shared task-level training settings and evaluation metrics are described in Appendix D.1.
W5. Full-run parameter audit.
We audited two full training runs of RSLoRA on LLaMA-3-8B at each of r = 8 and r = 32, recording the initial, minimum, maximum, and final active parameter counts. Across the audited runs, the maximum absolute deviations from same-rank fixed LoRA were 2.0056% and 0.5733%, respectively, while the total active rank remained constant.
These full-run audits show that active parameter counts remained close to those of same-rank fixed LoRA throughout the reported runs, although RSLoRA does not enforce exact parameter conservation.
| Rank | Configuration | Initial (M) | Full-run min. (M) | Full-run max. (M) | Final (M) | Max. abs. deviation |
|---|---|---|---|---|---|---|
| r = 8 | Fixed-rank LoRA | 14.156 | 14.156 | 14.156 | 14.156 | 0.00% |
| r = 8 | RSLoRA, Run 1 | 14.157 | 14.020 | 14.157 | 14.126 | 0.96% |
| r = 8 | RSLoRA, Run 2 | 14.157 | 14.157 | 14.440 | 14.440 | 2.01% |
| r = 32 | Fixed-rank LoRA | 56.623 | 56.623 | 56.623 | 56.623 | 0.00% |
| r = 32 | RSLoRA, Run 1 | 56.628 | 56.628 | 56.948 | 56.948 | 0.57% |
| r = 32 | RSLoRA, Run 2 | 56.628 | 56.522 | 56.628 | 56.531 | 0.18% |
Revisions to the manuscript.
We have expanded Appendix C.2 and replaced the sampled-event audit with full-run statistics in Table 6, covering two runs at each of r = 8 and r = 32.
Dear Reviewer oQs7,
We sincerely thank you for your time and effort in reviewing our manuscript and for your thoughtful and constructive feedback. Your comments have helped us improve the clarity and rigor of our work. Below, we respond to your comments and questions point by point, present the additional evidence, and explain the corresponding revisions to the manuscript.
W1. Controlled intervention details.
In our experiment, the three rank-addition branches match both module-type composition and added active parameter count. We clarify the module selection, checkpoint stage, and held-out evaluation set below.
Revisions to the manuscript.
We have expanded Appendix H.3 to specify the module selection, matched added parameter counts, checkpoint stage, and held-out validation-set composition. Table 16 reports the intervention results, and Table 17 lists the exact selected module layer indices for each comparison group and rank-addition branch.
W2. Contribution of rank redistribution.
To distinguish the contribution of rank redistribution from the SVD-style parameterization and orthogonal regularization, we added RSLoRA-fixed, a uniform-rank control. RSLoRA-fixed retains the same parameterization, initialization, orthogonal regularization, optimizer settings, and training protocol as RSLoRA, but keeps the initial uniform ranks fixed throughout training.
RSLoRA outperforms this control by 2.0 points on NLU and 2.4 percentage points on CR. These results support the additional contribution of rank redistribution beyond those shared components.
Revisions to the manuscript.
We have added this comparison and the corresponding analysis to Section 4.5 (Table 4) and Appendix H.2 (Table 15).
W3. Runtime overhead and FLOPs accounting.
We acknowledge that RSLoRA incurs additional runtime cost relative to standard LoRA. Table 13 reports training times of 1.87×, 1.97×, and 1.35× those of LoRA on CoLA, RTE, and LLaMA-3-8B, respectively. These timings already include calibration, CKA computation, rank redistribution, and periodic validation.
The original FLOPs estimates covered standard training batches but did not separately account for regularization, optimizer updates, calibration, similarity computation, rank-allocation overhead, or periodic validation. We have now supplemented these estimates with the additional costs and periodic validation, as reported below.
The expanded estimates show that RSLoRA increases total FLOPs over LoRA by 0.95%, 2.31%, and 0.97%, respectively, calculated before rounding.
Our results illustrate a trade-off between accuracy and training cost. Adaptive rank-allocation methods, including RSLoRA and AdaLoRA, achieve higher average scores than standard LoRA while requiring additional training time.
Revisions to the manuscript.
We have updated Appendix E and Table 13 to clarify the accounting scope, and discussed the additional runtime cost relative to LoRA.