acceptodds
Under review as a conference paper at ICLR 2027

RSLoRA: Dynamic Rank Allocation via Representational Similarity

13 views

Abstract

Low-Rank Adaptation (LoRA) freezes the pretrained parameters and introduces additional trainable low-rank modules. Determining an appropriate rank for each LoRA module remains a nontrivial problem. Prior approaches typically assign module-wise ranks by estimating module importance using information derived from the module itself or from interactions among modules, but largely neglect the relationship between each module and its associated pretrained weights. As noted by Hu et al. (2021), LoRA modules effectively amplify task-relevant directions that are already present in the pretrained model but are insufficiently emphasized. Hence, rank allocation should more appropriately be guided by characterizing the relationship between each module and its associated pretrained weights. Motivated by this observation, we propose RSLoRA, which introduces a new evaluation mechanism for rank allocation based on representational similarity. Specifically, RSLoRA measures the similarity between representations produced by the LoRA branch and the pretrained branch using Centered Kernel Alignment (CKA). Under a fixed rank budget, we devise a rank allocation criterion based on the principle that modules exhibiting greater representational dissimilarity (i.e., lower CKA) should be assigned larger rank capacity, as they must model structures that are not adequately captured by the pretrained weights. Overall, our method offers a representation-centric perspective on rank allocation. Experiments across multiple pretrained language models and downstream tasks demonstrate that RSLoRA consistently outperforms state-of-the-art parameter-efficient fine-tuning (PEFT) baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, dynamic rank allocation, representational similarity, centered kernel alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oQs7OpenReview ↗
Rating
2(+1)
Confidence
2(+1)
Informativeness
2(+1)
Presentation
3(+2)
Soundness
2(+1)
Read review
Paper summary

RSLoRA allocates adapter capacity using the representational dissimilarity between each LoRA branch and its corresponding pretrained branch, measured by linear CKA. It combines the dissimilarity with its temporal variation, periodically removes low-sensitivity component from low-scoring modules, and adds components to high-scoring modules. The paper evaluates the method on GLUE, commonsense reasoning and few-shot image classification.

Critical strengths

I would like to thank the authors. The paper introduces an interesting allocation signal that explicitly compares each adapter with the pretrained mapping it modifies. This provides a useful perspective beyond measuring properties of the adapter alone.

The empirical results cover three backbones and different task families, with means and standard deviations. RSLoRA improves the GLUE average from 88.8 for AdaLoRA to 89.6, and commonsense reasoning accuracy from 86.9 to 87.5 at rank 8 and from 86.5 to 88.1 at rank 32. The visual experiments provide additional evidence of generality

The revised paper includes substantial ablations. Comparisons with random allocation, frozen scores, alternative metrics, and shuffled pretrained references help establish the contribution of the proposed signal. The controlled rank-addition experiment in Appendix H.3 provides more direct evidence that high-scoring modules benefit from additional capacity.

Critical weaknesses

Incomplete specification of the controlled intervention. Appendix H.3 does not report the number and identities of selected modules, the checkpoint’s training stage, or the precise composition of the held-out QA set. Adding the same number of rank components does not necessarily add the same number of parameters when module dimensions differ. Actual added parameter counts should be reported before describing the branches as having matched capacity.

Incomplete isolation of the allocation benefit. Compared with standard LoRA, RSLoRA also introduces an SVD-inspired parameterization and orthogonality regularization. The random-allocation controls are useful, but a uniform-rank baseline using the identical parameterization, initialization, regularization, and optimizer settings would clarify how much of the improvement comes specifically from redistribution.

Understated computational overhead. The efficiency discussion emphasizes comparisons with AdaLoRA while Table 13 reports approximately 1.9 times the training runtime on CoLA and 1.35 times on LLaMA-3-8B relative to standard LoRA. This is a meaningful practical cost. The FLOPs accounting should explicitly state whether calibration passes, similarity computation, and allocation operations are included.

Missing relevant baseline. The experiments omit some newer/better baselines, (PRILoRA for example), which combines nonuniform rank allocation with pruning and evaluates on the same DeBERTaV3-base and GLUE benchmarks. A comparison under matched parameter budgets, evaluation metrics, and tuning effort would strengthen the claim that RSLoRA improves over existing PEFT methods.

Limited verification of parameter-budget comparability. Appendix C.2 correctly acknowledges that preserving total rank does not preserve the parameter count. However, Table 6 audits only sampled allocation events from two rank-8 runs. Full-run minimum, maximum, and final parameter counts, including rank 32, would better substantiate the resource comparisons.

Questions for authors

See weaknesses please .

2 comments

Dear Reviewer oQs7,

We sincerely thank you for your time and effort in reviewing our manuscript and for your thoughtful and constructive feedback. Your comments have helped us improve the clarity and rigor of our work. Below, we respond to your comments and questions point by point, present the additional evidence, and explain the corresponding revisions to the manuscript.

W1. Controlled intervention details.

In our experiment, the three rank-addition branches match both module-type composition and added active parameter count. We clarify the module selection, checkpoint stage, and held-out evaluation set below.

  1. Modules and capacity: The three rank-addition branches match added active parameter count. Each branch selects two modules per type (q/k/v/up/down), totaling 10 modules, and increases each from rank 8 to 13. Within each type, the high-score and low-score branches select the two highest- and lowest-scoring eligible modules, respectively, while the random branch samples two modules from the same candidate set. Thus, all three branches have identical module-type composition and add exactly 553,010 active adapter parameters. The exact selected modules for each branch and checkpoint are provided in Appendix H.3.
  2. Checkpoint: Within each comparison group, branching follows 2,000 shared fixed-rank preparation steps from the step-5,000 snapshot, with scoring at steps 6,000 and 7,000. Each branch then trains for 200 steps without further rank redistribution.
  3. Held-out set: We randomly hold out 1,024 examples from Commonsense170K using a fixed data-split seed, without task stratification, and exclude them from both training and calibration. The set contains 59 BoolQ, 112 PIQA, 203 SIQA, 245 HellaSwag, 364 WinoGrande, 10 ARC-e, 4 ARC-c, and 27 OBQA examples.

Revisions to the manuscript.

We have expanded Appendix H.3 to specify the module selection, matched added parameter counts, checkpoint stage, and held-out validation-set composition. Table 16 reports the intervention results, and Table 17 lists the exact selected module layer indices for each comparison group and rank-addition branch.

W2. Contribution of rank redistribution.

To distinguish the contribution of rank redistribution from the SVD-style parameterization and orthogonal regularization, we added RSLoRA-fixed, a uniform-rank control. RSLoRA-fixed retains the same parameterization, initialization, orthogonal regularization, optimizer settings, and training protocol as RSLoRA, but keeps the initial uniform ranks fixed throughout training.

RSLoRA outperforms this control by 2.0 points on NLU and 2.4 percentage points on CR. These results support the additional contribution of rank redistribution beyond those shared components.

VariantNLU: CoLANLU: RTENLU: SST-2NLU: Avg.CR: BoolQCR: PIQACR: ARC-cCR: Avg.
RSLoRA-fixed69.785.295.483.475.988.979.181.3
RSLoRA71.288.596.485.476.290.784.383.7

Revisions to the manuscript.

We have added this comparison and the corresponding analysis to Section 4.5 (Table 4) and Appendix H.2 (Table 15).

W3. Runtime overhead and FLOPs accounting.

We acknowledge that RSLoRA incurs additional runtime cost relative to standard LoRA. Table 13 reports training times of 1.87×1.87\times, 1.97×1.97\times, and 1.35×1.35\times those of LoRA on CoLA, RTE, and LLaMA-3-8B, respectively. These timings already include calibration, CKA computation, rank redistribution, and periodic validation.

The original FLOPs estimates covered standard training batches but did not separately account for regularization, optimizer updates, calibration, similarity computation, rank-allocation overhead, or periodic validation. We have now supplemented these estimates with the additional costs and periodic validation, as reported below.

SettingLoRAAdaLoRARSLoRAIncrease over LoRA
CoLA2.606×10152.606\times10^{15}2.627×10152.627\times10^{15}2.631×10152.631\times10^{15}0.95%
RTE1.207×10161.207\times10^{16}1.216×10161.216\times10^{16}1.235×10161.235\times10^{16}2.31%
LLaMA-3-8B2.763×10182.763\times10^{18}2.766×10182.766\times10^{18}2.789×10182.789\times10^{18}0.97%

The expanded estimates show that RSLoRA increases total FLOPs over LoRA by 0.95%, 2.31%, and 0.97%, respectively, calculated before rounding.

Our results illustrate a trade-off between accuracy and training cost. Adaptive rank-allocation methods, including RSLoRA and AdaLoRA, achieve higher average scores than standard LoRA while requiring additional training time.

Revisions to the manuscript.

We have updated Appendix E and Table 13 to clarify the accounting scope, and discussed the additional runtime cost relative to LoRA.

W4. Comparison with PRILoRA.

We added PRILoRA to the GLUE comparison with a matched initial parameter budget of approximately 1.9M trainable parameters, using the same task-specific training hyperparameters and evaluation metrics as the other methods. No additional PRILoRA-specific hyperparameter search was performed.

As reported in the table, RSLoRA achieves an average score of 89.6, compared with 87.9 for PRILoRA, an improvement of 1.7 points. These results further support RSLoRA's effectiveness against a recent baseline.

MethodParam.MNLI (Acc.)SST-2 (Acc.)CoLA (Mcc.)QQP (Acc.-F1)QNLI (Acc.)RTE (Acc.)MRPC (Acc.-F1)STS-B (Corr.)Avg.
PRILoRA1.9M89.6 ± 0.294.4 ± 0.367.7 ± 1.490.7 ± 0.193.9 ± 0.185.6 ± 1.991.5 ± 1.090.0 ± 0.887.9
RSLoRA1.9M90.7 ± 0.196.4 ± 0.171.2 ± 0.690.8 ± 0.194.6 ± 0.288.5 ± 0.392.7 ± 0.291.7 ± 0.189.6

Revisions to the manuscript.

The PRILoRA comparison is included in Table 1, with per-task means and standard deviations reported in Appendix I.1 (Table 18). The shared task-level training settings and evaluation metrics are described in Appendix D.1.

W5. Full-run parameter audit.

We audited two full training runs of RSLoRA on LLaMA-3-8B at each of r = 8 and r = 32, recording the initial, minimum, maximum, and final active parameter counts. Across the audited runs, the maximum absolute deviations from same-rank fixed LoRA were 2.0056% and 0.5733%, respectively, while the total active rank remained constant.

These full-run audits show that active parameter counts remained close to those of same-rank fixed LoRA throughout the reported runs, although RSLoRA does not enforce exact parameter conservation.

RankConfigurationInitial (M)Full-run min. (M)Full-run max. (M)Final (M)Max. abs. deviation
r = 8Fixed-rank LoRA14.15614.15614.15614.1560.00%
r = 8RSLoRA, Run 114.15714.02014.15714.1260.96%
r = 8RSLoRA, Run 214.15714.15714.44014.4402.01%
r = 32Fixed-rank LoRA56.62356.62356.62356.6230.00%
r = 32RSLoRA, Run 156.62856.62856.94856.9480.57%
r = 32RSLoRA, Run 256.62856.52256.62856.5310.18%

Revisions to the manuscript.

We have expanded Appendix C.2 and replaced the sampled-event audit with full-run statistics in Table 6, covering two runs at each of r = 8 and r = 32.

Reviewer_6KANOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RSLoRA, which dynamically allocates LoRA ranks by leveraging representation similarity between LoRA branches and their pre-trained counterparts. It computes CKA over a fixed calibration subset of training data, combines representation discrepancies and their temporal variations into module importance scores, and redistributes ranks via sensitivity pruning and zero-output initialization expansion. The method achieves promising results on GLUE with DeBERTaV3-base and commonsense reasoning tasks with LLaMA-3-8B. The authors also conduct ablations over different scoring functions, reference representations, and dynamic update strategies.

Critical strengths
  1. Unified rank allocation cannot reflect the adaptation requirements of different modules. How to allocate LoRA capacity under limited resources is an important problem. This paper incorporates the corresponding pre-trained mapping into the distribution signal rather than merely analyzing the update matrix itself. This perspective is reasonable and somewhat innovative.

  2. The approach is internally coherent, with clear logical connections across CKA differences, temporal smoothing, periodic allocation and low-sensitivity pruning.

Critical weaknesses
  1. While the experiments across two models and two task types serve as reasonable preliminary validation, the mechanism analysis is limited to small-scale NLU tasks, with no accompanying explanations for the strong LLaMA results.
Questions for authors

See Weakness

1 comment

Dear Reviewer 6KAN,

We sincerely thank you for your time and effort in reviewing our manuscript and for your thoughtful and constructive feedback. Your comments have helped us improve the clarity and rigor of our work.

W1

We added LLaMA-specific ablations and controlled rank-addition experiments to provide a more direct explanation of RSLoRA’s performance. The ablations support the contribution of RSLoRA’s allocation scores and their associated design choices, while the controlled intervention shows greater accuracy gains from adding matched capacity to high-scoring module groups than to low-scoring or randomly selected groups. We also provide few-shot visual results as complementary evidence for generality beyond language tasks and effectiveness with limited labeled data.

Complete LLaMA ablation results and analyses are provided in Appendix H.2 (Table 15), while the controlled rank-addition experiment is detailed in Appendix H.3 (Tables 16-17). Rank-allocation dynamics are analyzed in Section 5.2 (Figure 2(c–d)). The visual experiments are discussed in Section 4.4, with detailed results in Appendix I.3 (Table 20).

1. Ablation analysis on LLaMA-3-8B.

We extended the ablations to LLaMA-3-8B, examining allocation metrics, score aggregation, component pruning, module selection, score updates, and reference representations.

RSLoRA achieves the highest average accuracy of 83.7 among the evaluated ablation variants. The results support the effectiveness of score-guided rank redistribution, the proposed aggregation and pruning criteria, online score updates, and matched pretrained references.

We present the key ablation results below, with complete results and detailed analyses provided in Appendix H.2 (Table 15).

VariantBoolQPIQAARC-cAvg.
RSLoRA76.290.784.383.7
Fixed-rank control
RSLoRA-fixed75.988.979.181.3
Alternative importance metrics
Sensitivity75.889.278.881.3
Frobenius norm75.788.779.181.2
Spectral entropy76.088.979.581.5
Representation similarity metrics
Centered cosine75.688.178.380.7
RBF-CKA76.189.179.681.6
RSA-Spearman75.888.579.181.1
Score aggregation
Sᵢ=Īᵢ75.190.183.082.7
Sᵢ=Ūᵢ75.089.882.182.3
Sᵢ=(Īᵢ+Ūᵢ)/275.189.079.181.1
Sᵢ=ĪᵢŪᵢ76.089.178.581.2
Module allocation
Random-score75.889.079.281.3
Random-allocation75.988.679.581.3

2. Controlled rank-addition experiments.

Adding rank to high-scoring modules yields higher mean accuracy than adding matched capacity to low-scoring or randomly selected groups.

From a common LLaMA-3-8B checkpoint with uniform rank 8, we compared no addition with adding five rank components to each selected high-score, low-score, or random module. Each rank-addition branch selected the same number of modules and added five rank components to each selected module. Further redistribution was disabled, and each branch trained for 200 steps. The table reports mean and standard deviation over 3 random seeds.

BranchAnswer accuracy (%)Δ
No rank addition91.96 ± 0.34–
High-score modules +592.71 ± 0.31+0.75
Low-score modules +591.83 ± 0.46-0.13
Random modules +591.80 ± 0.76-0.16

High-score addition improves over the control by 0.75 percentage points. This supports using RSLoRA’s allocation scores to identify modules that benefit more from additional rank in this controlled continuation setting.

3. Complementary evidence from few-shot visual tasks.

RSLoRA also improves performance on visual tasks with limited labeled data. Complete results and analysis are provided in Section 4.4.

MethodParam.Food101Oxford PetsDTDUCF101Stanford CarsAvg.
4-shot
LoRA0.7M80.8 ± 0.788.7 ± 0.665.1 ± 0.779.8 ± 1.876.2 ± 0.678.12
AdaLoRA1.1M83.4 ± 0.491.7 ± 0.365.4 ± 1.182.0 ± 1.077.0 ± 0.379.90
RSLoRA0.7M84.4 ± 0.492.5 ± 0.365.8 ± 0.283.0 ± 1.477.8 ± 0.380.70
16-shot
LoRA0.7M83.1 ± 0.691.0 ± 0.772.5 ± 0.286.1 ± 0.287.3 ± 0.184.00
AdaLoRA1.1M84.8 ± 0.392.8 ± 0.673.0 ± 0.286.7 ± 0.386.9 ± 0.284.84
RSLoRA0.7M85.1 ± 0.293.5 ± 0.572.9 ± 0.687.0 ± 0.487.0 ± 0.285.10

4. Rank-allocation dynamics on LLaMA-3-8B.

Section 5.2 and Figure 2(c–d) complement the ablations and controlled intervention by illustrating how RSLoRA redistributes adaptation capacity under different initial rank budgets.

Reviewer_3eahOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RSLoRA, a dynamic rank-allocation method for parameter-efficient fine-tuning. It measures the representational similarity between each LoRA branch and its corresponding pretrained branch using linear CKA. Representational dissimilarity and its temporal variation are combined into a module-level allocation score. Under a fixed total rank budget, low-scoring modules lose their least-sensitive rank component, while high-scoring modules receive a new component initialized to preserve the current output. Experiments evaluate DeBERTaV3-base on GLUE and LLaMA-3-8B on eight commonsense reasoning benchmarks.

Critical strengths
  1. The method considers the relationship between LoRA updates and pretrained weights, rather than only the LoRA parameters. This is an interesting perspective for rank allocation.

  2. The results are encouraging. RSLoRA improves the GLUE average from 88.8 to 89.6 over AdaLoRA, with results reported over five random seeds. On commonsense reasoning at r=32r=32, the average accuracy improves from 86.8 to 88.0.

  3. The paper provides useful ablation studies. The comparisons with frozen scores, weight-space metrics, and shuffled pretrained references help explain why the proposed design works

Critical weaknesses
  1. The parameter-budget comparison needs clarification. Section 3.3 keeps the total rank fixed, but rank components in modules with different dimensions have different parameter costs. Table 2 also reports 21.2M parameters for RSLoRA and 14.2M for LoRA at r=8r=8. Please clarify whether these numbers refer to active or preallocated parameters, and how parameter costs are controlled during rank transfer. This is important for interpreting the comparison with uniform-rank LoRA.

  2. The allocation score has indirect support, but its connection to additional rank benefit remains unclear. The alternative-metric and shuffled-reference ablations provide useful evidence. However, they do not directly show whether modules with higher scores benefit more from additional rank. A controlled rank-addition experiment would help support this interpretation.

  3. The reliability of the commonsense reasoning gains needs further evidence. Table 2 evaluates one backbone and reports only a 0.1-point improvement over AdaLoRA at r=8r=8, without training-seed variation. The 1.2-point gain at r=32r=32 is more encouraging. Repeated runs and key allocation ablations on this backbone would strengthen the results.

Questions for authors
  1. How are active and preallocated parameters counted? When rank is moved between modules with different dimensions, how is the parameter budget controlled?

  2. Do modules with higher allocation scores obtain larger improvements from additional rank? How does the method compare with random module allocation using the same pruning strategy?

  3. Can the authors provide results over multiple training seeds and the main allocation ablations for commonsense reasoning?

3 comments

Dear Reviewer 3eah,

We sincerely thank you for your time and effort in reviewing our manuscript and for your thoughtful and constructive feedback. Your comments have helped us improve the clarity and rigor of our work. Below, we respond to your comments and questions point by point, present the additional evidence, and explain the corresponding revisions to the manuscript.

W1 & Q1

Param. in Tables 1 and 2 reports the initial active trainable parameter count. We compare RSLoRA with uniform-rank LoRA to evaluate the benefit of rank redistribution under a fixed total active-rank budget and comparable initial parameter counts. During rank redistribution, RSLoRA preserves the total active rank, but does not impose an additional parameter-cost constraint. We agree that rank components in modules with different dimensions have different parameter costs. However, our theoretical analysis bounds this variation, and our full-run audits show maximum absolute deviations from same-rank fixed LoRA of 2.0056% at rank 8 and 0.5733% at rank 32 across the audited runs. We clarify the parameter accounting, theoretical bounds, and empirical evidence below. Detailed derivations and the empirical audit are provided in Appendix C.2.

1. Parameter accounting and correction.

For an SVD-style rank component, let ci=diin+diout+1c_i=d_i^{\mathrm{in}}+d_i^{\mathrm{out}}+1. The active and allocated low-rank parameter counts are Pactive=∑iriciP_{\mathrm{active}}=\sum_i r_i c_i and Pallocated=∑iRiciP_{\mathrm{allocated}}=\sum_i R_i c_i. Here rir_i is active rank and RiR_i is allocated rank capacity. In RSLoRA’s dynamic tensor-resizing implementation, the active and allocated parameter counts are equal, since Ri(t)=ri(t)R_i(t)=r_i(t). With uniform rank 8 on the same LLaMA-3-8B modules, RSLoRA and LoRA start with 14,157,056 and 14,155,776 parameters, respectively; the difference is RSLoRA’s 1,280 scalar coefficients. Both round to 14.2M. We apologize for the incorrect 21.2M entry and have corrected it. The accounting correction does not change the recorded accuracies.

2. Bounds on parameter variation.

RSLoRA removes rank components from low-scoring modules and adds the same number to high-scoring modules, without a separate parameter-cost constraint. Nevertheless, the resulting parameter variation is bounded. If b(t)b^{(t)} components are scheduled for redistribution, then

R c_{\min}\le P_{\mathrm{active}}(t)\le R c_{\max}.$$ Here $R$ is the fixed total active rank, $c_{\min}$ and $c_{\max}$ are the smallest and largest per-component costs, and $b^{(t)}\le b^{(0)}$ follows the decreasing adjustment schedule. For the adapted LLaMA-3-8B modules, $c_{\min}=5{,}121$ and $c_{\max}=18{,}433$. Thus, redistributing one component changes the active parameter count by at most approximately 0.094% of the initial rank-8 parameter count. These bounds describe the worst case. The actual parameter changes are reported below. **3. Observed parameter variation.** We audited two full training runs of RSLoRA on LLaMA-3-8B at each of $r=8$ and $r=32$, recording the initial, minimum, maximum, and final active parameter counts. Across the audited runs, the maximum absolute deviations from same-rank fixed LoRA were 2.0056% and 0.5733%, respectively, while the total active rank remained constant. These full-run audits show that active parameter counts remained close to those of same-rank fixed LoRA throughout the reported runs, although RSLoRA does not enforce exact parameter conservation. | Rank | Configuration | Initial (M) | Full-run min. (M) | Full-run max. (M) | Final (M) | Max. abs. deviation (%) | |---|---|---:|---:|---:|---:|---:| | 8 | Fixed-rank LoRA | 14.156 | 14.156 | 14.156 | 14.156 | 0.00 | | 8 | RSLoRA, Run 1 | 14.157 | 14.020 | 14.157 | 14.126 | 0.96 | | 8 | RSLoRA, Run 2 | 14.157 | 14.157 | 14.440 | 14.440 | 2.01 | | 32 | Fixed-rank LoRA | 56.623 | 56.623 | 56.623 | 56.623 | 0.00 | | 32 | RSLoRA, Run 1 | 56.628 | 56.628 | 56.948 | 56.948 | 0.57 | | 32 | RSLoRA, Run 2 | 56.628 | 56.522 | 56.628 | 56.531 | 0.18 | **Revisions to the manuscript.** We corrected Tables 1–2 and added the accounting, bounds, and audit to Appendix C.2, Eqs. (17)–(20), and Table 6.

W2 & Q2

To provide more direct evidence on whether allocation scores identify where additional rank is more beneficial, we added controlled rank-addition experiments and random allocation comparisons.

  • The controlled intervention experiment shows that adding rank to high-scoring modules yields greater accuracy gains than adding matched capacity to low-scoring or randomly selected modules.

  • The random-allocation ablations show that RSLoRA outperforms random allocation through score-guided rank redistribution under the same total rank budget and pruning criterion.

Detailed analysis of the controlled rank-addition experiment is provided in Appendix H.3, while detailed results and analysis of the random allocation ablations are presented in Appendix H.2.

1. Assess the benefit of additional rank in high-scoring modules.

From a common LLaMA-3-8B checkpoint with uniform rank 8, we compared no addition with adding five rank components to each selected high-score, low-score, or random module. Each rank-addition branch selected the same number of modules and added five rank components to each selected module. Further redistribution was disabled, and each branch trained for 200 steps. The table reports mean and standard deviation over 3 random seeds. Detailed experimental settings and selected modules are provided in Appendix H.3 (Tables 16--17).

BranchAnswer accuracy (%)Δ
No rank addition91.96 ± 0.34–
High-score modules +592.71 ± 0.31+0.75
Low-score modules +591.83 ± 0.46-0.13
Random modules +591.80 ± 0.76-0.16

High-score addition improves over the control by 0.75 percentage points. This supports that our representational score usefully identifies the modules that benefit more from additional rank.

2. Comparison with random allocation under the same pruning strategy.

Random-score replaces allocation scores with random values; random-allocation directly samples eligible modules for rank increases and decreases. Both retain RSLoRA’s component-pruning criterion, total active-rank budget, and training settings.

VariantNLU CoLANLU RTENLU SST-2NLU Avg.CR BoolQCR PIQACR ARC-cCR Avg.
random-score70.087.395.484.275.889.079.281.3
random-allocation68.987.895.684.175.988.679.581.3
RSLoRA71.288.596.485.476.290.784.383.7

RSLoRA outperforms both random variants on all three NLU and all three CR tasks, supporting score-guided module allocation and our allocation scores beyond the shared pruning criterion.

W3 & Q3

We added multi-seed evaluations and allocation ablations on LLaMA-3-8B, together with complementary few-shot visual experiments.

The repeated-run results show higher mean accuracy than LoRA and AdaLoRA at both rank settings, the LLaMA ablations support the effectiveness of the key allocation choices, and the visual results provide complementary evidence for generality beyond language tasks and effectiveness with limited labeled data. We summarize the main results below.

Detailed multi-seed results are provided in Appendix I.2 (Table 19). Complete ablation results and analyses are presented in Appendix H.2 (Table 15). The visual experiments are discussed in Section 4.4, with detailed results in Appendix I.3 (Table 20).

1. Results across multiple training seeds.

We have updated the commonsense reasoning results to report the mean and standard deviation over 5 random seeds.

MethodBoolQPIQASIQAHellaS.WinoG.ARC-eARC-cOBQAAvg.
LoRA (r=8)72.7 ± 0.584.4 ± 1.079.9 ± 0.493.8 ± 0.285.0 ± 0.387.9 ± 0.475.6 ± 1.583.6 ± 0.882.9
AdaLoRA (r=8)75.8 ± 0.588.7 ± 0.882.1 ± 0.796.2 ± 0.588.4 ± 0.493.1 ± 0.183.1 ± 0.787.9 ± 1.086.9
RSLoRA (r=8)75.1 ± 0.790.1 ± 0.482.4 ± 0.794.5 ± 0.289.3 ± 0.593.6 ± 0.383.8 ± 0.690.8 ± 0.387.5
LoRA (r=32)70.8 ± 1.185.8 ± 0.479.3 ± 0.394.3 ± 0.386.8 ± 0.390.4 ± 0.380.3 ± 0.685.8 ± 1.484.2
AdaLoRA (r=32)73.2 ± 0.489.4 ± 0.581.7 ± 0.895.7 ± 0.188.1 ± 0.493.0 ± 0.181.7 ± 0.389.4 ± 0.286.5
RSLoRA (r=32)76.2 ± 0.290.7 ± 0.183.3 ± 0.296.8 ± 0.289.5 ± 0.293.8 ± 0.184.3 ± 0.290.0 ± 0.588.1

RSLoRA exceeds AdaLoRA by 0.6 and 1.6 percentage points at r=8 and r=32, respectively; the corresponding gains over LoRA are 4.6 and 3.9 points.

2. Ablations on LLaMA-3-8B.

We extended the ablations to LLaMA-3-8B, examining allocation metrics, score aggregation, component pruning, module selection, score updates, and reference representations.

RSLoRA achieves the highest average accuracy of 83.7 among the evaluated ablation variants. The results support the effectiveness of score-guided rank redistribution, the proposed aggregation and pruning criteria, online score updates, and matched pretrained references. Detailed results and analyses are provided in Appendix H.2 (Table 15).

VariantBoolQPIQAARC-cAvg.
RSLoRA76.290.784.383.7
Fixed-rank control
RSLoRA-fixed75.988.979.181.3
Alternative importance metrics
Sensitivity75.889.278.881.3
Frobenius norm75.788.779.181.2
Spectral entropy76.088.979.581.5
Representation similarity metrics
Centered cosine75.688.178.380.7
RBF-CKA76.189.179.681.6
RSA-Spearman75.888.579.181.1
Score aggregation
Sᵢ=Īᵢ75.190.183.082.7
Sᵢ=Ūᵢ75.089.882.182.3
Sᵢ=(Īᵢ+Ūᵢ)/275.189.079.181.1
Sᵢ=ĪᵢŪᵢ76.089.178.581.2

3. Complementary few-shot visual tasks.

RSLoRA also improves performance on visual tasks with limited labeled data. Complete results and analysis are provided in Section 4.4.

MethodParam.Food101Oxford PetsDTDUCF101Stanford CarsAvg.
4-shot
LoRA0.7M80.8 ± 0.788.7 ± 0.665.1 ± 0.779.8 ± 1.876.2 ± 0.678.12
AdaLoRA1.1M83.4 ± 0.491.7 ± 0.365.4 ± 1.182.0 ± 1.077.0 ± 0.379.90
RSLoRA0.7M84.4 ± 0.492.5 ± 0.365.8 ± 0.283.0 ± 1.477.8 ± 0.380.70
16-shot
LoRA0.7M83.1 ± 0.691.0 ± 0.772.5 ± 0.286.1 ± 0.287.3 ± 0.184.00
AdaLoRA1.1M84.8 ± 0.392.8 ± 0.673.0 ± 0.286.7 ± 0.386.9 ± 0.284.84
RSLoRA0.7M85.1 ± 0.293.5 ± 0.572.9 ± 0.687.0 ± 0.487.0 ± 0.285.10

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank you for your constructive feedback. We have revised the manuscript as follows:

  1. Parameter accounting (Reviewer 3eah, W1/Q1; Reviewer oQs7, W5). We corrected the affected parameter-count entries in Tables 1–2 in the main text. Appendix C.2 clarifies active versus allocated parameters, uniform-rank initialization, and parameter variation during redistribution, including theoretical bounds and an empirical audit (Table 6).

  2. Allocation-score validation (Reviewer 3eah, W2/Q2; Reviewer oQs7, W1). We added a controlled rank-addition experiment on LLaMA-3-8B (Appendix H.3, Tables 16-17) and random allocation comparisons with unchanged component pruning on NLU and CR tasks (Appendix H.2, Table 15).

  3. Repeated-run results (Reviewer 3eah, W3/Q3). We expanded the multi-seed evaluation on CR tasks and updated Table 2 in the main text with the repeated-run means. Per-task means and standard deviations are reported in Appendix I.2 (Table 19).

  4. LLaMA mechanism analysis (Reviewer 3eah, W3/Q3; Reviewer 6KAN; Reviewer oQs7, W2). We extended the allocation ablations to LLaMA-3-8B and added a fixed-rank control in Section 4.5 (Tables 4–5) and Appendix H.2 (Table 15). The controlled intervention in Appendix H.3 (Table 16) further examines the connection between allocation scores and additional-rank benefit.

  5. Additional visual evaluation. We added CLIP ViT-B/16 experiments on five image classification datasets under 4-shot and 16-shot settings to examine generality beyond language tasks and effectiveness with limited labeled data. Results appear in Section 4.4 (Table 3), with experimental settings in Appendix D.3 and detailed results in Appendix I.3 (Table 20).

  6. Computational cost (Reviewer oQs7, W3). We updated Appendix E and Table 13 with expanded FLOPs accounting and a discussion of runtime overhead relative to LoRA.

  7. PRILoRA comparison (Reviewer oQs7, W4). We added PRILoRA results to Table 1 and Appendix I.1 (Table 18).

We appreciate your suggestions and welcome further discussion.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.