acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Layer Selection for Low-Rank Adaptation

Abstract

Layer selection concentrates a low-rank adaptation budget on layers judged important by a pretrained model. We show that internal coordinates can change these recommendations while leaving the available adaptation functions unchanged. Our capacity-preserving audit preserves both the network function and the set of LoRA functions at every fixed placement. Gradient-based selections change across the tested architectures and calibration tasks, whereas contraction and block-ablation controls retain their predicted invariances. Controlled calibration experiments and norm balancing connect this sensitivity to internal scale. Training every selected set on the original backbone then measures its adaptation value: random-placement controls improve on the canonical gradient selection, and fixed sets exhibit different utility across tasks. Finally, comparisons with static placements, AdaLoRA, and FoRa match final effective adapter parameters, training steps, and learning-rate search. Full-depth, uniform low-rank LoRA achieves the highest accuracy in these comparisons, including against adaptive rank allocation that retains every layer. These results connect score interpretation to practical allocation: test coordinate sensitivity, measure the utility of the selected layers, and compare against distributed adaptation under a common parameter budget.

Primary area:
optimization
Keywords:
parameter-efficient fine-tuning, LoRA, layer selection, function-preserving transformations, neural network symmetries, model reparameterization, transfer learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HeMiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper examines the coordinate dependence of LoRA layer-selection metrics through function-preserving transformations and compares different allocation strategies. It establishes that the transformations preserve the attainable LoRA function set at each fixed placement and reports that full-depth uniform low-rank allocation outperforms the tested alternatives under matched final adapter budgets.

Critical strengths

The paper combines theoretical analysis with empirical evaluation. Section 3 tests the invariance properties derived in Section 2, while Section 4 examines the training utility of the resulting layer selections. Section 5 compares allocation strategies under matched adapter budgets.

Critical weaknesses
  1. The manuscript contains multiple undefined terms, including “internal coordinate”, “network function” and “adaptation function”, which substantially impair readability.

  2. Likewise, the contributions are not clearly stated, and the manuscript lacks a thorough comparison against existing PEFT approaches, such as [1].

  3. It is challenging to identify novel technical contributions in this work; most experimental findings have already been covered in prior PEFT studies. For instance, layer placement is already considered within the search space of [1], which may render gradient-based layer selection redundant under its specific search formulation.

References: [1] Neural Prompt Search, TPAMI, 2025

Questions for authors

Please see the concerns listed in the critical weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer HeMi (1/2)

We sincerely thank the reviewer for the careful reading, and we appreciate your recognition that the paper combines theoretical analysis with empirical evaluation. W2 and W3 each contain two points, which we label W2-1, W2-2 and W3-1, W3-2, and we answer related points together. We have uploaded a revised PDF in which all changes are marked in blue.

W1. Definitions of the terms

We have added a Terminology paragraph at the beginning of Section 2 and explained each of the three terms where it first appears in the Introduction. Their meanings are as follows.

  • Network function: the input-output map of the model, that is, the function that the model actually computes (Definition 1).
  • Internal coordinates: one particular set of weight values. Different sets of weight values can correspond to the same network function (Theorem 1).
  • Adaptation functions: all functions that the model can represent after LoRA of a given rank is added to a chosen set of layers (Theorem 2).

Equation 2 gives a concrete example. If the value matrix of an attention head is multiplied by a constant c and its output matrix is divided by c, the network function does not change, and by Theorem 2 the adaptation functions at any set of layers do not change either. Only the internal coordinates change. However, on Llama-3.2-3B, when the same kind of rescaling is applied to the up and down projections of each MLP with factors of at most 8, 5 of the 7 layers selected by the gradient-norm score (called Gradient below) are replaced (Section 3). In other words, the model stays the same while the layer scores change, which is exactly the problem studied in the paper.

W2-1 & W3-1. Statement of contributions and technical novelty

The central contribution of the paper is a test of whether layer scores stay the same under an equivalent reparameterization, which no prior PEFT study has carried out. For readability, the revised Section 1 states the three contributions more directly. An equivalent reparameterization changes only the weight values and leaves the outputs of the model unchanged, and the rescaling of Equation 2 in W1 is one example. The contributions are those of the submission, and they now read as follows.

  1. A capacity-preserving audit. We change the weights of a model without changing its outputs or what LoRA can express, and find that gradient-based layer scores then select different layers.
  2. Consequences for training. The layers selected before and after this change reach different accuracy when trained on the same model, so these scores should not be read as layer importance.
  3. A strong baseline. At a matched adapter budget, full-depth LoRA, which adapts every layer at a low rank, outperforms layer selection, AdaLoRA, and FoRA in every tested setting.

The supporting evidence is in Sections 3 to 5. For example, the Gradient selection changes in all 6 tested combinations of model and task, and in a crossed 5 × 5 study on Llama the reparameterization explains 82.4% of the variation in the scores, whereas the calibration batch explains only 4.1%.

Regarding novelty, making this test valid required the following three pieces of technical work, which are also the technical contributions of the paper.

  • We construct output-preserving weight transformations for the blocks of modern LLMs, covering RoPE attention with grouped key-value heads and SwiGLU MLPs, without assuming that SiLU is homogeneous (Theorem 1).
  • We prove that these transformations also preserve what LoRA can express at every set of layers and every rank (Theorem 2). Without this result, a change in the selection could simply be caused by a change in the capacity of LoRA.
  • We predict in advance which scores change and which do not (Proposition 1), and a certified audit over 140 transformed models confirms these predictions (Table 1).

Regarding the comment that most experimental findings are covered by prior PEFT studies, findings 1 and 2 above do not appear in prior work. None of NOAH, AdaLoRA, and FoRA tests whether the criterion used for selection or allocation depends on the particular weight values, and W2-2 & W3-2 compares with them one by one.

AuthorsOpenReview ↗

Response to Reviewer HeMi (2/2)

W2-2 & W3-2. Comparison with existing PEFT methods such as NOAH [1], and whether the search of NOAH makes gradient-based selection redundant

Our paper differs clearly from NOAH, AdaLoRA, and FoRA. We have also added a NOAH-style search baseline, and the searched placement still stays below full-depth LoRA.

These three methods study how to select layers or allocate ranks better, whereas our paper studies what such selection criteria actually reflect, and whether selecting layers is better than adding LoRA to every layer at the same budget. The specific differences are as follows, and the discussion of NOAH has been added to Related Work.

  • NOAH trains a supernet on each target dataset and then searches for the best PEFT configuration. It targets ViT-B/16 and vision tasks, does not assign a score to each layer, and does not test whether the searched configuration depends on the particular weight values.
  • AdaLoRA measures importance by weight-gradient products and prunes gradually during training. Our paper compares it under the same final adapter budget, training steps, and learning-rate search, and full-depth LoRA is more accurate (Tables 3 and 4).
  • FoRA selects layers by Fisher information and adds an orthogonality constraint. Our paper also compares it at the same budget (Tables 3 and 4) and shows that scores of the Fisher type change under equivalent reparameterizations (Proposition 1, Table 1).

The core of NOAH is to search for a configuration on the target data instead of scoring layers. In our LoRA setting, the counterpart is a search over which layers receive the adapter budget, and we have added this search as a baseline. On Llama-3.2-3B at budget 2L, where the total adapter rank is twice the number of layers, we generate 8 random quarter-depth placements (7 of the 28 layers at rank 8), train each of them in full, and keep the one with the lowest validation loss, with the other settings of Table 3. Table R1 lists the 8 candidates at seed 42, where CS-Avg is the average accuracy over seven commonsense reasoning tasks.

Table R1. Validation loss and accuracy of the 8 candidate placements. The search keeps candidate 1, which has the lowest validation loss (Table 27, App. W of the revised PDF).

CandidateSelected layersValidation lossCS-Avg (%)
11, 3, 6, 9, 11, 18, 241.376567.68
20, 1, 3, 11, 15, 16, 271.376666.27
30, 2, 4, 9, 16, 25, 261.378866.49
43, 5, 11, 17, 21, 22, 231.380165.70
50, 1, 5, 7, 19, 21, 231.380464.40
63, 6, 14, 17, 18, 22, 231.380965.59
73, 7, 9, 15, 20, 21, 221.381166.94
81, 3, 5, 20, 21, 23, 251.384761.31

Candidate 1, which the search keeps, is also the most accurate of the 8. The 8 candidates reach between 61.31% and 67.68%, all below the 68.32% of full-depth LoRA (the three-seed mean in Table 3), so no way of choosing among these candidates can exceed full-depth LoRA. We then evaluate candidate 1 over seeds 42 to 44 and repeat the same search on Qwen3-4B (9 of the 36 layers), and the results are given in Table R2.

Table R2. CS-Avg (%) of the searched placement and of three allocations from Table 3 (Table 28, App. W of the revised PDF).

AllocationLlama-3.2-3BQwen3-4B
Gradient top-k63.83 ± 0.9782.10 ± 0.08
Uniform quarter-depth66.22 ± 0.7482.16 ± 0.07
Searched quarter-depth67.19 ± 0.8882.23 ± 0.09
Full-depth LoRA68.32 ± 1.0782.37 ± 0.12

On both models, the searched placement is better than the uniform one but still below full-depth LoRA, and the search requires 8 additional training runs. The other 14 allocations that Table 10 of the paper compares on three models are also all below full-depth LoRA.

Gradient-based selection is indeed redundant, and this is also the conclusion of our paper. The paper does not propose a gradient-based selection method. Instead, it shows that such scores are unreliable. On Llama-3.2-3B, the layers selected by Gradient are less accurate not only than full-depth LoRA but also than a uniform quarter-depth placement that uses no score at all (Table R2). Four random placements within the same depth range are also more accurate than Gradient (65.42% to 65.81%, Section 4). The reason is that Gradient prefers early layers (0, 1, 2, 4, 5, 7, and 10), and this preference comes from the differences in weight scale across layers, not from these layers being more important. Once the scales are balanced without changing the outputs of the model, Gradient selects exactly the evenly spaced layers 0, 4, 8, …, 24 (Section 4).

Dear Reviewer HeMi,

Thanks for your time and effort in reviewing this paper. In the rebuttal and the revised paper, we have added definitions of the three terms (W1), stated the three contributions more directly and explained how they differ from prior PEFT studies (W2, W3), and added a NOAH-style search baseline, whose searched placement remains below full-depth LoRA (W2, W3).

As the discussion period will end in about 3 days, we gently request that you check whether these address your concerns. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_RJ7UOpenReview ↗
Rating
4(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper rethinks the layer selection strategy in Low-Rank Adaptation (LoRA) and studies whether layer importance scores are influenced by the model’s internal parameterization. The authors propose an audit method that preserves both model functionality and LoRA adaptability. The authors demonstrate that equivalent parameterizations can lead to different layer selection outcomes. Finally, comparisons with static placements, AdaLoRA, and FoRa match final effective adapter parameters, training steps, and learning-rate search. Full-depth, uniform low-rank LoRA achieves the highest accuracy in these comparisons, including against adaptive rank allocation that retains every layer.

Critical strengths

S1: This paper rethinks the problem of layer selection in LoRA from the perspective of parametric invariance. The authors not only provide a theoretical analysis but also use experiments to verify the impact of equivalent parameterization on layer importance scores.

S2: The experimental design is relatively systematic.

Critical weaknesses

W1: Although this paper highlights the limitations of existing layer importance scoring, I think it remains unclear how these findings can guide the practical selection of LoRA layers.

W2: I think the current experiments are insufficient, and I suggest that the authors further explore tasks such as code generation and open-ended question-answering to verify whether the advantages of Full-depth LoRA can be generalized to different types of downstream tasks.

W3: In this paper, the final effective adapter parameter budgets are matched, but I think it has not yet fully compared the trade-offs between training computational cost and performance across different methods.

W4: I recommend the authors provide extra experiments,e.g., directly comparing the LoRA training performance on the original model and the equivalent parameterized model, to verify the impact of parameterization changes on actual optimization behavior.

Questions for authors

Please refer to W1-W4. Some explanation and experiments need to be provided.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RJ7U (1/2)

We sincerely thank the reviewer for the encouraging comments and suggestions, and we appreciate your recognition of the parametric-invariance perspective and of the systematic experimental design. We have uploaded a revised PDF in which all changes are marked in blue.

W1. How the findings guide practical layer selection

The findings can be summarized as three practical rules, which we have added to Section 6 as a checklist ("A checklist for practitioners").

  1. Before a score is interpreted as layer importance, check whether it is invariant under an equivalent reparameterization. An equivalent reparameterization changes only the weight values and leaves the outputs of the model unchanged. For example, multiply the value matrix of one attention head by a constant and divide its output matrix by the same constant (Equation 2). If the score changes while the model does not, the score reflects the scale of the weights rather than the role of the layer. This check requires only one additional scoring pass.
  2. At a given adapter budget, start with full-depth low-rank LoRA, that is, low-rank LoRA on every layer. It is the most accurate allocation on every model and budget that we tested (Tables 3 and 4, App. U).
  3. If LoRA is still to be applied to a subset of layers, compare the subset with a uniform placement of the same size and report the selected layer IDs. On Llama-3.2-3B, the layers selected by the gradient-norm score (called Gradient below) are 2.39 points below the uniform quarter-depth placement (63.83% vs. 66.22%), and once the layer scales are balanced, Gradient selects exactly the evenly spaced layers (Section 4).

W2. Code generation and open-ended question answering

Following your suggestion, we have added both types of tasks on two models. Full-depth LoRA has the highest mean on every metric, in the same order as in Table 3. The setup is as follows, and the results are given in Table R1.

  • Models and budget: Llama-3.2-3B and Qwen3-4B at budget 2L, where the total adapter rank is twice the number of layers.
  • Allocations: full-depth LoRA (rank 2 on all layers), and the uniform quarter-depth placement and Gradient top-k (rank 8 on a quarter of the layers).
  • Open-ended question answering: fine-tuning on SQuAD v1.1, with exact match (EM) and F1 on 2,000 validation questions.
  • Code generation: fine-tuning on CodeAlpaca-20k, with pass@1 on the 664 problems of HumanEval and MBPP.
  • Training: the number of steps of Table 3, seeds 42 to 44, and greedy decoding at evaluation. The learning rate is 4 × 10⁻⁴, which all six methods select in Table 4, and on both tasks on Llama-3.2-3B it also gives a lower validation loss than 2 × 10⁻⁴.

Table R1. Three allocations on question answering and code generation, mean ± SD (Table 26, App. V of the revised PDF).

ModelAllocationSQuAD EMSQuAD F1Code pass@1
Llama-3.2-3BFull-depth85.20 ± 0.4491.99 ± 0.3333.28 ± 0.26
Uniform quarter-depth84.88 ± 0.2891.73 ± 0.2033.13 ± 0.45
Gradient top-k83.73 ± 0.7091.03 ± 0.4232.58 ± 0.09
Qwen3-4BFull-depth87.42 ± 0.3193.56 ± 0.2446.84 ± 0.30
Uniform quarter-depth87.15 ± 0.2893.38 ± 0.2146.59 ± 0.40
Gradient top-k86.63 ± 0.4593.02 ± 0.3246.13 ± 0.23

On both models, full-depth LoRA is the best on all three metrics, in the same order as in Table 3. On Llama-3.2-3B, the Gradient scores were recomputed on 32 training examples of each task, and the selected layers are again concentrated at the beginning of the network. Both tasks select layers 0 to 5. At each of the three seeds, Gradient is the lowest of the three allocations on every metric, and on the QA task its EM is at least 1.15 points lower than that of the other two allocations.

AuthorsOpenReview ↗

Response to Reviewer RJ7U (2/2)

W3. Trade-off between training cost and performance

We have measured the training cost of all six methods. Full-depth LoRA takes 1.3 to 1.6 times the training time of a quarter-depth placement, has the same inference cost, and is more accurate.

The measurement is carried out on Llama-3.2-3B at budget 2L, on a single dedicated H100-class GPU with one run at a time, with seed 42 and the other settings of Table 3, and the results are given in Table R2. CS-Avg is the average accuracy over seven commonsense reasoning tasks.

Table R2. Training cost of the six methods on Llama-3.2-3B (Table 29, App. X of the revised PDF).

AllocationFinal parametersScoring before training (s)Training (s)Total (s)Peak memory (GB)CS-Avg (%)
Full-depth3.04M027127124.968.32
Uniform quarter-depth3.04M020220220.166.22
Gradient top-k3.04M6.819119820.063.83
Contraction top-k3.04M6.820521219.865.55
AdaLoRA3.04M043543525.266.71
FoRA3.04M1224325521.165.12

All six methods have the same final adapter parameter count. Compared with the three quarter-depth methods, full-depth LoRA takes 1.3 to 1.4 times the training time and about 5 GB more memory, while it is 2.10 points more accurate than the uniform placement and 4.49 points more accurate than Gradient. The higher accuracy thus costs about 1.3 to 1.4 times the training time. AdaLoRA is the most expensive method, because it starts from rank 8 on every projection and prunes only later. Computing the layer scores before training is cheap. Gradient and Contraction share one backward pass over the 32 calibration examples. Full-depth LoRA and the uniform placement use no score, and AdaLoRA computes its importance scores during training, which is included in its training time, so the scoring time before training is 0 for these three methods. After LoRA is merged into the weights, all six methods run the original model at inference, so their inference cost is the same.

The measurement on Qwen2.5-0.5B in App. K is consistent with this. There, full-depth LoRA trains in 274 s and Gradient top-k in 171 s including its 4 s scoring pass, a ratio of 1.6. The revised Abstract and Conclusion also state this trade-off.

W4. LoRA training on the original model and on an equivalently reparameterized model

This comparison is already reported in Table 14 of App. L. Full-depth LoRA stays above the layers selected by Gradient whether the original or the reparameterized model is trained.

Table 14 is a 2 × 2 experiment. Training is carried out on the original model and on an equivalently reparameterized model, the Gradient selection is also computed on each of the two models, and full-depth LoRA is trained on both models as a reference. On Llama-3.2-3B, full-depth LoRA reaches 68.03% and 68.82% on the two models, and the four Gradient-selected combinations are between 62.64% and 64.54%. On Qwen2.5-0.5B, full-depth LoRA reaches 54.04% and 54.44%, and the four Gradient-selected combinations are between 51.54% and 52.76%. When the same placement is trained on the two models, the accuracy differs by at most 1.90 points, and full-depth LoRA is above all four Gradient-selected sets on both models. The conclusion about allocation therefore does not depend on which model is trained. Table 14 follows its own initialization protocol, so its full-depth mean on the original model (68.03%) differs slightly from the 68.32% of Table 3.

Regarding the optimization behavior that you ask about, the training results on the two models are not identical, which is expected. The reparameterization does not change what LoRA can express (Theorem 2), but it rescales the gradients of the LoRA parameters. AdamW normalizes the update of each parameter separately, so the optimization path is different, although the set of reachable functions is the same in the two cases. This is also why Section 4 trains all selected layer sets on the original model, so that the comparison isolates the choice of layers. We have added two sentences to Section 4 of the revised PDF that state this and point to Table 14.

Dear Reviewer RJ7U,

Thanks for your time and effort in reviewing this paper. In the rebuttal and the revised paper, we have added a checklist for practitioners (W1), experiments on code generation and open-ended question answering, where full-depth LoRA has the highest mean on every metric (W2), and the training cost of all six methods (W3). The comparison requested in W4 is reported in Table 14.

As the discussion period will end in about 3 days, we gently request that you check whether these address your concerns. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_U9UiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines whether LoRA layer-selection scores reflect useful adaptation locations or depend on the model's internal parameter coordinates. It constructs compensated transformations that preserve both the pretrained function and the available LoRA functions at each fixed placement. Gradient-based selections change under these transformations, while contraction and block-ablation controls exhibit their predicted invariances. To assess the selected layers separately from changes in optimization coordinates, every placement is trained on the original backbone. The experiments include random placements, crossed calibration, scale balancing, and comparisons with AdaLoRA and FoRA under matched final adapter-parameter budgets, training steps, and learning-rate searches. Full-depth uniform low-rank LoRA achieves the highest accuracy in the reported allocation comparisons, motivating its use as a baseline for layer selection and adaptive rank allocation.

Critical strengths

The fixed-placement argument and the original-backbone evaluation work well together. They make it possible to ask whether a score changes for reasons unrelated to which layers are useful, without also changing the optimizer's training coordinates.

The experiments are unusually detailed about layer IDs, budgets, paired runs, and learning-rate searches. The random placements, crossed calibration, five-seed tests, and 8B extension give useful checks. The paper also distinguishes score invariance from selection quality: the effect on downstream accuracy differs between Llama and Qwen.

The main contribution is the evidence that selection scores can change without changing adaptation capacity. Together with the fixed-backbone results, this makes a useful case for including full-depth low-rank LoRA as a baseline within the tested task and budget scope.

Critical weaknesses

The allocation recommendation needs a resource-budget qualification. Matching final adapter size does not match training time or memory. Appendix K already shows a runtime trade-off, so full-depth low-rank adaptation may not be the preferred choice under every practical budget. The conclusion should keep this distinction clear.

The adaptive-method comparison also depends on tuning. The equal learning-rate search is a good control, but the allocation schedules tried for AdaLoRA and FoRA are unclear. This is worth clarifying before interpreting their ordering as evidence about adaptive allocation in general.

Questions for authors

Which allocation schedules were fixed or tuned for AdaLoRA and FoRA in the reported experiments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer U9Ui (1/2)

We sincerely thank the reviewer for the careful and supportive comments, and we appreciate your recognition of the fixed-placement argument, the original-backbone evaluation, and the level of experimental detail. We have uploaded a revised PDF in which all changes are marked in blue.

W1. Stating which resource is held fixed

The revised Abstract and Conclusion now state that the advantage of full-depth LoRA is obtained under matched adapter parameters and training steps, and they report its additional cost in training time and memory. The revised Conclusion reads as follows.

Under matched final adapter budgets and common training protocols, full-depth uniform low-rank LoRA outperforms the tested static and adaptive alternatives. In our measurements, it takes about 1.3 to 1.6 times the training time of a quarter-depth placement and up to about a quarter more peak memory, with the same inference cost after merging; a quarter-depth placement is cheaper to train when time or memory is the main constraint.

The factor of 1.6 comes from the measurement on Qwen2.5-0.5B in App. K, where full-depth LoRA (low-rank LoRA on every layer) trains in 274 s and Gradient top-k, which selects layers by gradient norm, in 171 s. We also measured all six methods on Llama-3.2-3B at budget 2L, where the total adapter rank is twice the number of layers, on a single dedicated H100-class GPU with one run at a time and seed 42. The results are given in Table R1, where CS-Avg is the average accuracy over seven commonsense reasoning tasks.

Table R1. Training cost of the six methods on Llama-3.2-3B (Table 29, App. X of the revised PDF).

AllocationFinal parametersScoring before training (s)Training (s)Total (s)Peak memory (GB)CS-Avg (%)
Full-depth3.04M027127124.968.32
Uniform quarter-depth3.04M020220220.166.22
Gradient top-k3.04M6.819119820.063.83
Contraction top-k3.04M6.820521219.865.55
AdaLoRA3.04M043543525.266.71
FoRA3.04M1224325521.165.12

The six methods hold the final adapter parameter count and the training steps fixed, and they differ in training time and memory. The three methods with a scoring time of 0 compute no score before training, and AdaLoRA computes its importance scores during training, which is included in its training time. Full-depth LoRA takes 1.3 to 1.4 times the training time of the three quarter-depth methods and about 5 GB more peak memory. The advantage therefore comes at the cost of longer training and more memory. When these are the main constraint, a quarter-depth placement is cheaper to train, and the revised Conclusion also states this. Section 6 already contains a paragraph on this issue as well ("State which resource is held fixed").

AuthorsOpenReview ↗

Response to Reviewer U9Ui (2/2)

W2 & Q. Whether the AdaLoRA and FoRA schedules were fixed or tuned

The schedules of both methods were fixed and were not tuned. The learning rate was the only hyperparameter that we searched, with the same grid and validation rule for all six methods. Each method used the single schedule listed in App. R for both models and all seeds, with the following settings.

  • AdaLoRA: at budget 2L, it starts from rank 8 on every projection and prunes toward an average rank of 2 between steps 200 and 800 of the 1,000 training steps. At budget 8L, it prunes from rank 16 to rank 8.
  • FoRA: it selects 7 of the 28 layers of Llama-3.2-3B by Fisher information, keeps the B factor orthonormal with Cayley-Adam, and re-orthogonalizes it every 100 steps.
  • Learning rate: each method selects from {1, 2, 4} × 10⁻⁴ by validation loss, and all six methods select 4 × 10⁻⁴ (App. R and Table 4). When the learning rate is raised to 8 × 10⁻⁴ on Llama at budget 2L, the validation loss of all six methods increases.

To check that the ordering does not depend on the schedule, we ran an additional experiment on Llama-3.2-3B at budget 2L with different AdaLoRA and FoRA settings, and the results are given in Table R2. The training data, the 1,000 training steps, the learning rate (2 × 10⁻⁴), and seeds 42 to 44 are the same as in Table 3.

Table R2. CS-Avg under different AdaLoRA and FoRA settings. The first row of each method is the setting used in the paper (Table 30, App. Y of the revised PDF).

MethodSettingCS-Avg (%)
AdaLoRAInitial rank 8, pruning steps 200 to 80066.71
AdaLoRAInitial rank 8, pruning steps 100 to 70066.38
AdaLoRAInitial rank 8, pruning steps 300 to 90066.95
AdaLoRAInitial rank 12, pruning steps 200 to 80066.84
FoRA7 layers at rank 865.12
FoRA14 layers at rank 465.86
Full-depth LoRAAll 28 layers at rank 268.32

The ordering does not change under any of these settings. The best AdaLoRA variant reaches 66.95%, which is still 1.37 points below full-depth LoRA, and both FoRA settings are also below it. On Qwen3-4B, the best AdaLoRA variant (82.26%) also stays below full-depth LoRA (82.37%).

In addition, after pruning AdaLoRA still keeps nonzero ranks in all 28 layers of Llama-3.2-3B and all 36 layers of Qwen3-4B. It therefore adapts the same layers as full-depth LoRA and differs only in how the ranks are distributed across layers, and the results show that the uniform ranks of full-depth LoRA are more accurate.

Dear Reviewer U9Ui,

Thanks for your time and effort in reviewing this paper. The revised Abstract and Conclusion now state the extra training time and memory of full-depth LoRA (W1). The AdaLoRA and FoRA schedules were fixed, as listed in App. R, and full-depth LoRA stays above both methods at both budgets on both models (W2, Q).

As the discussion period will end in about 3 days, we gently request that you check whether these address your concerns. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the Area Chair and all reviewers for their time and effort in providing careful and constructive feedback on our submission.

During the discussion, Reviewer RJ7U raised the rating from 3 to 4 (Clear acceptance) and Reviewer HeMi from 2 to 3 (Weak acceptance), and the rating of Reviewer U9Ui is 3 (Weak acceptance). All three reviewers currently recommend acceptance.


We are encouraged to see that our reviewers recognize the combination of theoretical analysis and experiments in this study of LoRA layer selection.

  • Reviewer RJ7U notes that the paper "rethinks the problem of layer selection in LoRA from the perspective of parametric invariance", and Reviewer U9Ui finds that the fixed-placement argument and the original-backbone evaluation "work well together".
  • Reviewer HeMi notes that the paper "combines theoretical analysis with empirical evaluation", and Reviewer U9Ui finds the experiments "unusually detailed about layer IDs, budgets, paired runs, and learning-rate searches".
  • Reviewer U9Ui finds that the results make "a useful case for including full-depth low-rank LoRA as a baseline within the tested task and budget scope".

We highlight our contributions as follows:

  • Our Capacity-Preserving Audit changes the weights of a model without changing its outputs or what LoRA can express, and shows that Gradient-Based Layer Scores then select different layers.
  • By combining Fixed-Backbone Training of every selected layer set with allocation comparisons at matched final adapter parameters, training steps, and learning-rate search, we show that these scores should not be read as layer importance and that Full-Depth Uniform Low-Rank LoRA achieves the highest accuracy among the tested static and adaptive allocations, including AdaLoRA and FoRA.

As suggested by the reviewers, we revised our manuscript and added new appendix sections starting at Appendix Section V. Each table in our responses is labeled with its number in the revised PDF, and new and revised text is marked in blue.

Additional Results & Experiments

  • Added code generation and open-ended question answering, comparing full-depth LoRA with the uniform quarter-depth placement and Gradient top-k, in Table 26 and Appendix Section V, as suggested by Reviewer RJ7U.
  • Added a validation-searched placement baseline, which selects among quarter-depth placements by validation loss, with the eight candidate placements in Table 27 and the searched placement in Table 28, both in Appendix Section W, in response to the comment of Reviewer HeMi on the comparison with NOAH.
  • Added the training time, peak memory, and scoring time of all six methods on Llama-3.2-3B in Table 29 and Appendix Section X, in response to Reviewers RJ7U and U9Ui.

Other Changes & Modifications

  • We have added a Terminology paragraph at the beginning of Section 2 and explained network function, internal coordinates, and adaptation functions where they first appear in Section 1, in response to Reviewer HeMi.
  • We have restated the three contributions at the end of Section 1 and added NOAH to the related work in Section 7, in response to Reviewer HeMi.
  • We have added a checklist for practitioners to Section 6, in response to Reviewer RJ7U.
  • We have stated the training-time and memory cost of full-depth LoRA in the Abstract, Section 6, and Section 8, in response to Reviewers U9Ui and RJ7U.
  • We have revised Section 4 to explain why all selected layer sets are trained on the original model and to point to the comparison on the original and reparameterized models in Table 14, in response to Reviewer RJ7U.
  • We have stated in Appendix Section R that each method uses a single schedule for both models and all seeds, in response to Reviewer U9Ui.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.