- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper examines the coordinate dependence of LoRA layer-selection metrics through function-preserving transformations and compares different allocation strategies. It establishes that the transformations preserve the attainable LoRA function set at each fixed placement and reports that full-depth uniform low-rank allocation outperforms the tested alternatives under matched final adapter budgets.
Critical strengths
The paper combines theoretical analysis with empirical evaluation. Section 3 tests the invariance properties derived in Section 2, while Section 4 examines the training utility of the resulting layer selections. Section 5 compares allocation strategies under matched adapter budgets.
Critical weaknesses
-
The manuscript contains multiple undefined terms, including “internal coordinate”, “network function” and “adaptation function”, which substantially impair readability.
-
Likewise, the contributions are not clearly stated, and the manuscript lacks a thorough comparison against existing PEFT approaches, such as [1].
-
It is challenging to identify novel technical contributions in this work; most experimental findings have already been covered in prior PEFT studies. For instance, layer placement is already considered within the search space of [1], which may render gradient-based layer selection redundant under its specific search formulation.
References: [1] Neural Prompt Search, TPAMI, 2025
Questions for authors
Please see the concerns listed in the critical weaknesses section.
5 comments
Response to Reviewer HeMi (2/2)
W2-2 & W3-2. Comparison with existing PEFT methods such as NOAH [1], and whether the search of NOAH makes gradient-based selection redundant
Our paper differs clearly from NOAH, AdaLoRA, and FoRA. We have also added a NOAH-style search baseline, and the searched placement still stays below full-depth LoRA.
These three methods study how to select layers or allocate ranks better, whereas our paper studies what such selection criteria actually reflect, and whether selecting layers is better than adding LoRA to every layer at the same budget. The specific differences are as follows, and the discussion of NOAH has been added to Related Work.
- NOAH trains a supernet on each target dataset and then searches for the best PEFT configuration. It targets ViT-B/16 and vision tasks, does not assign a score to each layer, and does not test whether the searched configuration depends on the particular weight values.
- AdaLoRA measures importance by weight-gradient products and prunes gradually during training. Our paper compares it under the same final adapter budget, training steps, and learning-rate search, and full-depth LoRA is more accurate (Tables 3 and 4).
- FoRA selects layers by Fisher information and adds an orthogonality constraint. Our paper also compares it at the same budget (Tables 3 and 4) and shows that scores of the Fisher type change under equivalent reparameterizations (Proposition 1, Table 1).
The core of NOAH is to search for a configuration on the target data instead of scoring layers. In our LoRA setting, the counterpart is a search over which layers receive the adapter budget, and we have added this search as a baseline. On Llama-3.2-3B at budget 2L, where the total adapter rank is twice the number of layers, we generate 8 random quarter-depth placements (7 of the 28 layers at rank 8), train each of them in full, and keep the one with the lowest validation loss, with the other settings of Table 3. Table R1 lists the 8 candidates at seed 42, where CS-Avg is the average accuracy over seven commonsense reasoning tasks.
Table R1. Validation loss and accuracy of the 8 candidate placements. The search keeps candidate 1, which has the lowest validation loss (Table 27, App. W of the revised PDF).
| Candidate | Selected layers | Validation loss | CS-Avg (%) |
|---|---|---|---|
| 1 | 1, 3, 6, 9, 11, 18, 24 | 1.3765 | 67.68 |
| 2 | 0, 1, 3, 11, 15, 16, 27 | 1.3766 | 66.27 |
| 3 | 0, 2, 4, 9, 16, 25, 26 | 1.3788 | 66.49 |
| 4 | 3, 5, 11, 17, 21, 22, 23 | 1.3801 | 65.70 |
| 5 | 0, 1, 5, 7, 19, 21, 23 | 1.3804 | 64.40 |
| 6 | 3, 6, 14, 17, 18, 22, 23 | 1.3809 | 65.59 |
| 7 | 3, 7, 9, 15, 20, 21, 22 | 1.3811 | 66.94 |
| 8 | 1, 3, 5, 20, 21, 23, 25 | 1.3847 | 61.31 |
Candidate 1, which the search keeps, is also the most accurate of the 8. The 8 candidates reach between 61.31% and 67.68%, all below the 68.32% of full-depth LoRA (the three-seed mean in Table 3), so no way of choosing among these candidates can exceed full-depth LoRA. We then evaluate candidate 1 over seeds 42 to 44 and repeat the same search on Qwen3-4B (9 of the 36 layers), and the results are given in Table R2.
Table R2. CS-Avg (%) of the searched placement and of three allocations from Table 3 (Table 28, App. W of the revised PDF).
| Allocation | Llama-3.2-3B | Qwen3-4B |
|---|---|---|
| Gradient top-k | 63.83 ± 0.97 | 82.10 ± 0.08 |
| Uniform quarter-depth | 66.22 ± 0.74 | 82.16 ± 0.07 |
| Searched quarter-depth | 67.19 ± 0.88 | 82.23 ± 0.09 |
| Full-depth LoRA | 68.32 ± 1.07 | 82.37 ± 0.12 |
On both models, the searched placement is better than the uniform one but still below full-depth LoRA, and the search requires 8 additional training runs. The other 14 allocations that Table 10 of the paper compares on three models are also all below full-depth LoRA.
Gradient-based selection is indeed redundant, and this is also the conclusion of our paper. The paper does not propose a gradient-based selection method. Instead, it shows that such scores are unreliable. On Llama-3.2-3B, the layers selected by Gradient are less accurate not only than full-depth LoRA but also than a uniform quarter-depth placement that uses no score at all (Table R2). Four random placements within the same depth range are also more accurate than Gradient (65.42% to 65.81%, Section 4). The reason is that Gradient prefers early layers (0, 1, 2, 4, 5, 7, and 10), and this preference comes from the differences in weight scale across layers, not from these layers being more important. Once the scales are balanced without changing the outputs of the model, Gradient selects exactly the evenly spaced layers 0, 4, 8, …, 24 (Section 4).
Dear Reviewer HeMi,
Thanks for your time and effort in reviewing this paper. In the rebuttal and the revised paper, we have added definitions of the three terms (W1), stated the three contributions more directly and explained how they differ from prior PEFT studies (W2, W3), and added a NOAH-style search baseline, whose searched placement remains below full-depth LoRA (W2, W3).
As the discussion period will end in about 3 days, we gently request that you check whether these address your concerns. Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
Response to Reviewer HeMi (1/2)
We sincerely thank the reviewer for the careful reading, and we appreciate your recognition that the paper combines theoretical analysis with empirical evaluation. W2 and W3 each contain two points, which we label W2-1, W2-2 and W3-1, W3-2, and we answer related points together. We have uploaded a revised PDF in which all changes are marked in blue.
We have added a Terminology paragraph at the beginning of Section 2 and explained each of the three terms where it first appears in the Introduction. Their meanings are as follows.
Equation 2 gives a concrete example. If the value matrix of an attention head is multiplied by a constant c and its output matrix is divided by c, the network function does not change, and by Theorem 2 the adaptation functions at any set of layers do not change either. Only the internal coordinates change. However, on Llama-3.2-3B, when the same kind of rescaling is applied to the up and down projections of each MLP with factors of at most 8, 5 of the 7 layers selected by the gradient-norm score (called Gradient below) are replaced (Section 3). In other words, the model stays the same while the layer scores change, which is exactly the problem studied in the paper.
The central contribution of the paper is a test of whether layer scores stay the same under an equivalent reparameterization, which no prior PEFT study has carried out. For readability, the revised Section 1 states the three contributions more directly. An equivalent reparameterization changes only the weight values and leaves the outputs of the model unchanged, and the rescaling of Equation 2 in W1 is one example. The contributions are those of the submission, and they now read as follows.
The supporting evidence is in Sections 3 to 5. For example, the Gradient selection changes in all 6 tested combinations of model and task, and in a crossed 5 × 5 study on Llama the reparameterization explains 82.4% of the variation in the scores, whereas the calibration batch explains only 4.1%.
Regarding novelty, making this test valid required the following three pieces of technical work, which are also the technical contributions of the paper.
Regarding the comment that most experimental findings are covered by prior PEFT studies, findings 1 and 2 above do not appear in prior work. None of NOAH, AdaLoRA, and FoRA tests whether the criterion used for selection or allocation depends on the particular weight values, and W2-2 & W3-2 compares with them one by one.