- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that where LoRA adapters are placed affects both target-task gains and retention of general capabilities. It measures layer-wise target-task sensitivity with empirical Fisher scores and proposes input–output cosine similarity as a forward-only proxy, reporting Spearman correlations between and across four models and two tasks, together with a theorem that lower-bounds the Fisher score by . LS-LoRA trains adapters only in layers with similarity below . On math (MATH-10K) and code (Magicoder-Evol) fine-tuning of four models, it matches or slightly improves target-task averages over all-layer LoRA ( to points on math) while retaining more commonsense accuracy ( to points after math fine-tuning, to after code fine-tuning), and it also reduces memory and training time.
Critical strengths
S1: The method is simple, cheap and consistently helpful on the reported metrics. Layer selection needs one forward pass over unlabeled inputs ( seconds on Llama-3.2-3B), and LS-LoRA has the best target-task and commonsense averages among fine-tuned methods on all four models in both tasks. It also reduces peak memory by –% and training time by –%, and on Llama-3.2-3B the same selection also improves DoRA and PiSSA.
S2: The proxy is checked carefully. Its correlation with empirical Fisher is strong on all eight model–task pairs, layer rankings are stable with very few calibration samples ( with a single sample) and with last-token versus token-averaged scoring (–), and on math, similarity-based selection beats random selection of the same number of layers on all four models.
Critical weaknesses
W1: The retention evaluation, which carries the main claim, is narrow and its reliability is unclear. Retention is measured only on three commonsense benchmarks, and the evaluation protocol (prompt format, answer extraction, number of shots) is not described. Several results suggest the scores may reflect output format rather than retained capability: Llama-3-8B Base scores on average, below Llama-3.2-3B Base () and close to Llama-3.2-1B Base (); code fine-tuning raises Llama-3-8B to –; and Llama-3-8B Base reaches on AQuA while all fine-tuned variants score lower (–). No variance over seeds is reported, and the target-task gains over LoRA are small.
W2: The link between the selection criterion and retention is not established, and key comparisons are missing. Empirical Fisher is computed on the target task only, so it explains target-task sensitivity but not why adapting these layers should reduce forgetting. Theorem 1 supports ranking only for fixed , while depends on layer-specific quantities that are not measured. The selection criterion is the same input–output similarity used by ShortGPT, yet the paper does not compare with selecting layers directly by empirical Fisher, nor with the layer-selection methods it cites (e.g., Ogawa et al. (2026), Pan et al. (2024), Liu et al. (2025a)). The threshold is fixed without sensitivity analysis, and the number of selected layers is not reported.
Questions for authors
Q1: How are the commonsense benchmarks evaluated (prompt format, answer extraction, zero- or few-shot)? Why is Llama-3-8B Base below Llama-3.2-3B Base, and why does code fine-tuning nearly double its commonsense accuracy?
Q2: How many and which layers are selected for each model and task at ? How was chosen, and how sensitive are the results to it?
Q3: How does LS-LoRA compare with selecting layers by empirical Fisher directly, and with the similarity-based layer-wise LoRA of Ogawa et al. (2026)?
Q4: Are the results from a single run? Can you report variance over seeds, and over several random draws for Rand-LoRA?
Q5: Figure 1 shows per-layer target gains and retention for Llama-3.2-1B. Does input–output similarity predict which single layers cause the least degradation there?
10 comments
Official Comment by Authors
3. Broader coverage of the retention evaluation.
We have extended the retention evaluation beyond ARC-C, OB-QA, and SI-QA by adding BoolQ, HellaSwag, and PIQA on Llama-3.2-1B. These benchmarks broaden the evaluation to include yes/no reading comprehension, commonsense inference, and physical commonsense reasoning. The results are shown below as accuracy percentages:
| Method | BoolQ | HellaSwag | PIQA | Avg. |
|---|---|---|---|---|
| Base | 57.6 | 29.4 | 51.1 | 46.0 |
| LoRA | 41.4 | 22.5 | 23.6 | 29.2 |
| CKA-LoRA | 45.4 | 25.3 | 43.6 | 38.1 |
| LS-LoRA | 50.1 | 27.1 | 45.5 | 40.9 |
Bold indicates the best result among the fine-tuned methods.
LS-LoRA achieves higher accuracy than both LoRA and CKA-LoRA on all three additional benchmarks. Its average accuracy is 40.9%, compared with 29.2% for LoRA and 38.1% for CKA-LoRA. Although it remains below the Base model, it preserves more performance across each of these additional evaluations.
Official Comment by Authors
Response to W2, Q2, and Q3: Selection rationale, comparative evidence, and threshold sensitivity
We have added direct CKA- and Fisher-based comparisons, complete selected-layer lists, and a threshold sensitivity analysis. Below, we clarify the role of the Fisher analysis and provide the supporting results directly.
1. What connects the selection criterion to capability retention?
The Fisher analysis characterizes target-task sensitivity; the retention benefit is supported by controlled fine-tuning comparisons. Empirical Fisher scores are computed on the target task and do not directly measure a layer’s importance for non-target capabilities.
To examine whether the choice of layers contributes to retention beyond reducing the adaptation budget, we compare LS-LoRA with Rand-LoRA using the same number of adapted layers. The mathematical reasoning experiments report five runs for Rand-LoRA, with a newly sampled layer subset in each run, and three random-seed runs for LS-LoRA:
| Model | Rand-LoRA: Math | LS-LoRA: Math | Rand-LoRA: Commonsense | LS-LoRA: Commonsense |
|---|---|---|---|---|
| Llama-3.2-1B | 51.2 ± 0.7 | 52.1 ± 0.2 | 26.2 ± 2.6 | 31.6 ± 0.6 |
| Llama-3.2-3B | 65.9 ± 0.6 | 67.0 ± 0.4 | 56.8 ± 1.0 | 58.4 ± 0.3 |
| Llama-3-8B | 69.9 ± 0.7 | 71.6 ± 0.3 | 51.6 ± 0.3 | 56.3 ± 0.2 |
| Mistral-7B | 55.8 ± 1.0 | 56.3 ± 0.4 | 39.1 ± 2.5 | 46.3 ± 0.5 |
Values are mean accuracy (%) ± standard deviation, averaged within each benchmark category. LS-LoRA achieves higher mean mathematical and commonsense accuracy across all four models. These results support the value of selecting particular layers at a fixed adaptation budget.
The selected-layer patterns provide a possible interpretation. As listed below, LS-LoRA predominantly selects early Transformer layers and, in several settings, a small number of layers near the output. Concentrating adaptation in task-sensitive layers may support downstream learning while leaving the remaining layers without trainable adapters, potentially reducing interference with pretrained capabilities. This remains a hypothesis about the mechanism; the experiments establish the observed retention benefit without establishing a functional separation between “adaptation layers” and “knowledge-storage layers.”
Theorem 1 provides theoretical motivation, not a guarantee of cross-layer Fisher rankings. It establishes the conditional lower bound , where contains layer-dependent quantities that we have not measured across layers. Moreover, ordering lower bounds would not necessarily order the actual Fisher scores even with a common factor. The ranking interpretation instead rests on the observed Spearman correlations of −0.76 to −0.91 and the direct Fisher-based selection results reported below.
Official Comment by Authors
Response to W1, Q1, and Q4: Evaluation reliability, protocol, and variability
1. Llama-3-8B evaluation anomalies.
We traced the unexpectedly low Llama-3-8B Base scores to an answer-format issue. The base model did not consistently produce answers in the format expected by the extraction procedure, causing extraction failures and underestimating its accuracy.
We evaluate the benchmarks in a zero-shot setting, following the regular-expression-based answer extraction procedure of Liu et al. (2025b), LIFT. The instruction–response prompt is:
Here, {instruction} contains the evaluation question and its answer choices. No few-shot demonstrations are included. For Llama-3-8B evaluation, we added the following answer-format instructions to the commonsense questions:
These templates and constraints are documented in Appendix F.
With the added constraint, Llama-3-8B Base achieves 77.2% on ARC-C, 69.8% on OB-QA, and 68.6% on SI-QA, averaging 71.9%, compared with the originally reported 32.0%. This corrects the anomalously low Base reference underlying the apparent increase after code fine-tuning, and the corrected Base average exceeds Llama-3.2-3B Base’s 59.1%.
We also tested the revised prompt on the code-fine-tuned models and observed no clear accuracy increase.
The AQuA anomaly had the same cause. After adding the answer-format constraint, Llama-3-8B achieves 52.3% accuracy before fine-tuning, compared with 63.1% for LoRA, 59.7% for L2-SP-LoRA, 60.5% for EWC-LoRA, and 66.1% for LS-LoRA. All four fine-tuned variants now outperform Base, resolving the anomalous pattern in the original evaluation. These updated results are included in Table 2.
2. Variability across training runs and random layer selections.
We have expanded the mathematical reasoning layer-selection experiments to report means and standard deviations across all four models. Rand-LoRA uses five runs, with a newly sampled layer subset in each run; LoRA and LS-LoRA each use three random-seed runs. For each model, Rand-LoRA and LS-LoRA adapt the same number of layers.
The complete per-benchmark results for these three methods are shown below. Values are accuracy percentages, reported as means ± standard deviations. Math Avg. and Commonsense Avg. denote averages across their respective three benchmarks.
LS-LoRA achieves higher mean mathematical and commonsense accuracy than both all-layer LoRA and Rand-LoRA across all four models. The target-task gains are relatively small; the principal benefit is competitive target-task performance together with higher retained commonsense accuracy. The comparisons with Rand-LoRA also support the benefit of the selection criterion when the number of adapted layers is held fixed.