acceptodds
Under review as a conference paper at ICLR 2027

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention

Abstract

Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining. Existing approaches primarily mitigate this trade-off through data replay or regularization, relying on additional data or explicit optimization constraints. We instead focus on a different question: where should adaptation be applied? We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation. To characterize this difference, we use layer-wise empirical Fisher information to measure target-task sensitivity. However, computing Fisher scores requires backward computation and becomes increasingly expensive for large models. We therefore introduce input–output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity. Across models and tasks, layers with lower input–output similarity consistently exhibit higher empirical Fisher scores. Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input–output similarity. Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large Language Models, Parameter-Efficient Fine-Tuning, Layer-Selective Adaptation, Low-Rank Adaptation, Capability Retention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_G5kUOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that where LoRA adapters are placed affects both target-task gains and retention of general capabilities. It measures layer-wise target-task sensitivity with empirical Fisher scores and proposes input–output cosine similarity as a forward-only proxy, reporting Spearman correlations between −0.76-0.76 and −0.91-0.91 across four models and two tasks, together with a theorem that lower-bounds the Fisher score by Al(1−sl)A_l(1 - s_l). LS-LoRA trains adapters only in layers with similarity below γ=0.85\gamma = 0.85. On math (MATH-10K) and code (Magicoder-Evol) fine-tuning of four models, it matches or slightly improves target-task averages over all-layer LoRA (+0.4+0.4 to +1.1+1.1 points on math) while retaining more commonsense accuracy (+6.0+6.0 to +14.3+14.3 points after math fine-tuning, +1.2+1.2 to +6.2+6.2 after code fine-tuning), and it also reduces memory and training time.

Critical strengths

S1: The method is simple, cheap and consistently helpful on the reported metrics. Layer selection needs one forward pass over 100100 unlabeled inputs (2.82.8 seconds on Llama-3.2-3B), and LS-LoRA has the best target-task and commonsense averages among fine-tuned methods on all four models in both tasks. It also reduces peak memory by 4.24.2–14.414.4% and training time by 10.010.0–29.629.6%, and on Llama-3.2-3B the same selection also improves DoRA and PiSSA.

S2: The proxy is checked carefully. Its correlation with empirical Fisher is strong on all eight model–task pairs, layer rankings are stable with very few calibration samples (ρ=0.938\rho = 0.938 with a single sample) and with last-token versus token-averaged scoring (ρ=0.92\rho = 0.92–0.930.93), and on math, similarity-based selection beats random selection of the same number of layers on all four models.

Critical weaknesses

W1: The retention evaluation, which carries the main claim, is narrow and its reliability is unclear. Retention is measured only on three commonsense benchmarks, and the evaluation protocol (prompt format, answer extraction, number of shots) is not described. Several results suggest the scores may reflect output format rather than retained capability: Llama-3-8B Base scores 32.032.0 on average, below Llama-3.2-3B Base (59.159.1) and close to Llama-3.2-1B Base (33.433.4); code fine-tuning raises Llama-3-8B to 61.761.7–67.467.4; and Llama-3-8B Base reaches 49.649.6 on AQuA while all fine-tuned variants score lower (40.940.9–45.345.3). No variance over seeds is reported, and the target-task gains over LoRA are small.

W2: The link between the selection criterion and retention is not established, and key comparisons are missing. Empirical Fisher is computed on the target task only, so it explains target-task sensitivity but not why adapting these layers should reduce forgetting. Theorem 1 supports ranking only for fixed AlA_l, while AlA_l depends on layer-specific quantities that are not measured. The selection criterion is the same input–output similarity used by ShortGPT, yet the paper does not compare with selecting layers directly by empirical Fisher, nor with the layer-selection methods it cites (e.g., Ogawa et al. (2026), Pan et al. (2024), Liu et al. (2025a)). The threshold γ=0.85\gamma = 0.85 is fixed without sensitivity analysis, and the number of selected layers is not reported.

Questions for authors

Q1: How are the commonsense benchmarks evaluated (prompt format, answer extraction, zero- or few-shot)? Why is Llama-3-8B Base below Llama-3.2-3B Base, and why does code fine-tuning nearly double its commonsense accuracy?

Q2: How many and which layers are selected for each model and task at γ=0.85\gamma = 0.85? How was γ\gamma chosen, and how sensitive are the results to it?

Q3: How does LS-LoRA compare with selecting layers by empirical Fisher directly, and with the similarity-based layer-wise LoRA of Ogawa et al. (2026)?

Q4: Are the results from a single run? Can you report variance over seeds, and over several random draws for Rand-LoRA?

Q5: Figure 1 shows per-layer target gains and retention for Llama-3.2-1B. Does input–output similarity predict which single layers cause the least degradation there?

10 comments
AuthorsOpenReview ↗

Official Comment by Authors

Response to W1, Q1, and Q4: Evaluation reliability, protocol, and variability

1. Llama-3-8B evaluation anomalies.

We traced the unexpectedly low Llama-3-8B Base scores to an answer-format issue. The base model did not consistently produce answers in the format expected by the extraction procedure, causing extraction failures and underestimating its accuracy.

We evaluate the benchmarks in a zero-shot setting, following the regular-expression-based answer extraction procedure of Liu et al. (2025b), LIFT. The instruction–response prompt is:

<s> Below is an instruction that describes a task. Write a response
that appropriately completes the request.
### Instruction:
{instruction}
### Response:

Here, {instruction} contains the evaluation question and its answer choices. No few-shot demonstrations are included. For Llama-3-8B evaluation, we added the following answer-format instructions to the commonsense questions:

Solve the problem and select one of the answer choices.
End your response with: The answer is (X).
Replace X with answer1, answer2, answer3, answer4, or answer5.

These templates and constraints are documented in Appendix F.

With the added constraint, Llama-3-8B Base achieves 77.2% on ARC-C, 69.8% on OB-QA, and 68.6% on SI-QA, averaging 71.9%, compared with the originally reported 32.0%. This corrects the anomalously low Base reference underlying the apparent increase after code fine-tuning, and the corrected Base average exceeds Llama-3.2-3B Base’s 59.1%.

We also tested the revised prompt on the code-fine-tuned models and observed no clear accuracy increase.

The AQuA anomaly had the same cause. After adding the answer-format constraint, Llama-3-8B achieves 52.3% accuracy before fine-tuning, compared with 63.1% for LoRA, 59.7% for L2-SP-LoRA, 60.5% for EWC-LoRA, and 66.1% for LS-LoRA. All four fine-tuned variants now outperform Base, resolving the anomalous pattern in the original evaluation. These updated results are included in Table 2.

2. Variability across training runs and random layer selections.

We have expanded the mathematical reasoning layer-selection experiments to report means and standard deviations across all four models. Rand-LoRA uses five runs, with a newly sampled layer subset in each run; LoRA and LS-LoRA each use three random-seed runs. For each model, Rand-LoRA and LS-LoRA adapt the same number of layers.

The complete per-benchmark results for these three methods are shown below. Values are accuracy percentages, reported as means ± standard deviations. Math Avg. and Commonsense Avg. denote averages across their respective three benchmarks.

ModelMethodGSM8KAQuASVAMPMath Avg.ARC-COB-QASI-QACommonsense Avg.
Llama-3.2-1BLoRA49.2 ± 0.837.9 ± 0.866.3 ± 1.251.1 ± 0.823.1 ± 0.423.4 ± 1.929.6 ± 1.525.4 ± 1.0
Llama-3.2-1BRand-LoRA50.4 ± 0.736.3 ± 1.967.1 ± 1.051.2 ± 0.723.6 ± 1.726.2 ± 2.328.7 ± 4.426.2 ± 2.6
Llama-3.2-1BLS-LoRA50.3 ± 0.138.9 ± 0.967.2 ± 1.352.1 ± 0.230.3 ± 1.131.2 ± 1.033.2 ± 0.331.6 ± 0.6
Llama-3.2-3BLoRA69.8 ± 0.548.3 ± 1.980.3 ± 0.466.1 ± 0.945.1 ± 1.852.0 ± 0.538.0 ± 3.045.0 ± 1.4
Llama-3.2-3BRand-LoRA69.9 ± 0.546.6 ± 1.781.2 ± 1.165.9 ± 0.657.9 ± 1.559.9 ± 0.952.7 ± 2.956.8 ± 1.0
Llama-3.2-3BLS-LoRA70.1 ± 0.548.4 ± 1.182.5 ± 2.167.0 ± 0.459.7 ± 0.861.9 ± 1.553.8 ± 1.458.4 ± 0.3
Llama-3-8BLoRA69.4 ± 0.562.5 ± 1.679.0 ± 2.270.3 ± 1.250.5 ± 1.350.3 ± 1.146.4 ± 1.249.1 ± 1.1
Llama-3-8BRand-LoRA70.7 ± 0.759.9 ± 0.479.3 ± 1.069.9 ± 0.750.8 ± 1.554.2 ± 0.949.8 ± 1.051.6 ± 0.3
Llama-3-8BLS-LoRA70.6 ± 0.265.8 ± 0.878.4 ± 0.271.6 ± 0.359.9 ± 0.956.1 ± 0.852.9 ± 1.056.3 ± 0.2
Mistral-7BLoRA58.4 ± 1.332.7 ± 0.473.1 ± 1.254.8 ± 0.734.4 ± 1.439.3 ± 1.136.6 ± 0.936.8 ± 0.9
Mistral-7BRand-LoRA61.4 ± 1.032.5 ± 1.073.6 ± 1.655.8 ± 1.040.4 ± 2.941.0 ± 2.635.9 ± 2.539.1 ± 2.5
Mistral-7BLS-LoRA61.2 ± 0.533.4 ± 1.374.2 ± 0.556.3 ± 0.449.1 ± 1.142.5 ± 0.347.4 ± 0.546.3 ± 0.5

LS-LoRA achieves higher mean mathematical and commonsense accuracy than both all-layer LoRA and Rand-LoRA across all four models. The target-task gains are relatively small; the principal benefit is competitive target-task performance together with higher retained commonsense accuracy. The comparisons with Rand-LoRA also support the benefit of the selection criterion when the number of adapted layers is held fixed.

AuthorsOpenReview ↗

Official Comment by Authors

3. Broader coverage of the retention evaluation.

We have extended the retention evaluation beyond ARC-C, OB-QA, and SI-QA by adding BoolQ, HellaSwag, and PIQA on Llama-3.2-1B. These benchmarks broaden the evaluation to include yes/no reading comprehension, commonsense inference, and physical commonsense reasoning. The results are shown below as accuracy percentages:

MethodBoolQHellaSwagPIQAAvg.
Base57.629.451.146.0
LoRA41.422.523.629.2
CKA-LoRA45.425.343.638.1
LS-LoRA50.127.145.540.9

Bold indicates the best result among the fine-tuned methods.

LS-LoRA achieves higher accuracy than both LoRA and CKA-LoRA on all three additional benchmarks. Its average accuracy is 40.9%, compared with 29.2% for LoRA and 38.1% for CKA-LoRA. Although it remains below the Base model, it preserves more performance across each of these additional evaluations.

AuthorsOpenReview ↗

Official Comment by Authors

Response to W2, Q2, and Q3: Selection rationale, comparative evidence, and threshold sensitivity

We have added direct CKA- and Fisher-based comparisons, complete selected-layer lists, and a threshold sensitivity analysis. Below, we clarify the role of the Fisher analysis and provide the supporting results directly.

1. What connects the selection criterion to capability retention?

The Fisher analysis characterizes target-task sensitivity; the retention benefit is supported by controlled fine-tuning comparisons. Empirical Fisher scores are computed on the target task and do not directly measure a layer’s importance for non-target capabilities.

To examine whether the choice of layers contributes to retention beyond reducing the adaptation budget, we compare LS-LoRA with Rand-LoRA using the same number of adapted layers. The mathematical reasoning experiments report five runs for Rand-LoRA, with a newly sampled layer subset in each run, and three random-seed runs for LS-LoRA:

ModelRand-LoRA: MathLS-LoRA: MathRand-LoRA: CommonsenseLS-LoRA: Commonsense
Llama-3.2-1B51.2 ± 0.752.1 ± 0.226.2 ± 2.631.6 ± 0.6
Llama-3.2-3B65.9 ± 0.667.0 ± 0.456.8 ± 1.058.4 ± 0.3
Llama-3-8B69.9 ± 0.771.6 ± 0.351.6 ± 0.356.3 ± 0.2
Mistral-7B55.8 ± 1.056.3 ± 0.439.1 ± 2.546.3 ± 0.5

Values are mean accuracy (%) ± standard deviation, averaged within each benchmark category. LS-LoRA achieves higher mean mathematical and commonsense accuracy across all four models. These results support the value of selecting particular layers at a fixed adaptation budget.

The selected-layer patterns provide a possible interpretation. As listed below, LS-LoRA predominantly selects early Transformer layers and, in several settings, a small number of layers near the output. Concentrating adaptation in task-sensitive layers may support downstream learning while leaving the remaining layers without trainable adapters, potentially reducing interference with pretrained capabilities. This remains a hypothesis about the mechanism; the experiments establish the observed retention benefit without establishing a functional separation between “adaptation layers” and “knowledge-storage layers.”

Theorem 1 provides theoretical motivation, not a guarantee of cross-layer Fisher rankings. It establishes the conditional lower bound Il≥Al(1−sl)I_l \geq A_l(1-s_l), where AlA_l contains layer-dependent quantities that we have not measured across layers. Moreover, ordering lower bounds would not necessarily order the actual Fisher scores even with a common factor. The ranking interpretation instead rests on the observed Spearman correlations of −0.76 to −0.91 and the direct Fisher-based selection results reported below.

Reviewer_bWp4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the placement of LoRA adapters affects target-task adaptation and the retention of non-target capabilities. The authors seek indicators for selecting optimal layers for LoRA adaptation, and find a label- and gradient-free solution using callibration target-task inputs to get layer input-output cosine similarity as an empirical proxy of Fisher. They propose LS-LoRA, which places adapters only in layers whose similarity falls below a fixed threshold. 4 LLMs are adapted for mathematical reasoning and code generation, along with 3 commonsense benchmarks for retention evaluation. They compare against standard all-layer LoRA, regularization-based methods, and random layer selection, and a consistent improvements over these baselines are observed.

Critical strengths

The proposed method is intuitive, simple and practically light-weight. The layer selection process only requires forwarding a small number of inputs, after which fine-tuning can proceed directly without any additional regularization or design tweaks beyond standard LoRA. The reported improvements against other baselines are consistent and substantial, and the random control provides evidence that such improvements are not from naively reducing the number of adapted layers. The experiments on DoRA and PiSSA further suggest that the gains may be extended to improve other PEFT methods.

Critical weaknesses

About novelty and missing baselines

I find the method proposed here is not sufficiently distinguished from existing representation-based layer-selection approaches and is more of an incremental perspective contribution. I do not understand why some important baselines already identified in the paper have been neglected for experiments, which makes it difficult to assess its specific contribution.

As the authors have mentioned in Section 2.2, ShortGPT has already uses input–output similarity to identify influential and redundant layers, although for pruning rather than adaptation; and more directly Ogawa et al. (2026) already uses dissimilarity between layer's input and output representations the for LoRA fine-tuning, though using CKA rather than cosine similarity. Their selection is also performed before fine-tuning through inference. Therefore, the authors acknowledged that the central idea of using forward-pass representation changes to decide where to apply LoRA is already present in prior work. The distinction here needs to be established more precisely than introducing another similarity measure.

The authors identify their additional contributions in Section 2.2 as "to connect a forward-only layer-selection criterion to target-task empirical Fisher sensitivity" and "evaluate its consequences for both adaptation and capability retention". The former would just be a perspective contribution given that similar solution has been already proposed and examined in numerous previous studies. For the latter, since the proposed cos-similarity based method is a proxy for Fisher, a directly Fisher-based approach should be examined to help better establish the contribution. I think this paper did not satisfactorily justify what its particular design contributes beyond the closest existing approaches, like at least an variant using CKA should also be added as a strong baseline.

Currently, the reported comparisons only involve all-layer LoRA, regularization, and random selection, and improvements over these baselines do not establish whether the propsed solution provides any meaningful advantage over existing representation-based placement strategies.

About experimental details

The random-layer experiments does not fully resolve the attribution of the gains. In Table 4 and 8, random selection already maintains or even surpasses LoRA's target performance, and consistently outperforms standard LoRA on retention. This shows that a substantial part of the improvement of the proposed method over all-layer LoRA can arise with only layer-wise sparsity, without any specific selection criterion. Maybe the results can benefit from an additional statistical significance analysis, particularly considering that several target-task improvements are relatively small.

The choice of γ=0.85\gamma=0.85 is not empirically examined. Reporting sensitivity to this threshold might help further assess if there's an optimal sparsity-performance sweet point and the effectiveness of the proposed method.

I also didn't find any details about training configurations like learning rates, batch sizes, token lengths, target matrices to apply LoRA, and hardware settings etc. that are necessary for followers to reproduce this work.

Questions for authors

Please address the major concerns listed above in Critical Weaknesses. I also have found some minor points:

  1. Fig 1 can benefit from mentioning which benchmark(s) is reported for General Capability Retention in the caption.

  2. line 63 "Adapting some layers contribute more to ..." should be "contributes"

6 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your thoughtful feedback! We have carefully revised the manuscript to address each of the points raised.

About novelty and missing baselines

1. Direct comparison with Ogawa et al. (2026).

We have added the CKA-based layer-wise LoRA method of Ogawa et al. (2026), denoted as CKA-LoRA, to the mathematical reasoning and code generation experiments across all four models. The complete results are below. All scores are percentages; bold indicates the better category average within each model.

Mathematical reasoning fine-tuning

ModelMethodGSM8KAQuASVAMPMath Avg.ARC-COB-QASI-QACommonsense Avg.
Llama-3.2-1BCKA-LoRA51.236.267.751.728.230.429.329.3
Llama-3.2-1BLS-LoRA50.337.868.452.230.032.233.131.8
Llama-3.2-3BCKA-LoRA68.648.481.266.155.558.650.955.0
Llama-3.2-3BLS-LoRA69.548.882.366.959.362.452.658.1
Llama-3-8BCKA-LoRA70.163.778.270.754.353.851.153.1
Llama-3-8BLS-LoRA70.466.178.671.760.755.651.956.1
Mistral-7BCKA-LoRA59.333.174.355.646.742.243.144.0
Mistral-7BLS-LoRA61.033.174.656.248.342.447.946.2

LS-LoRA achieves higher average accuracy than CKA-LoRA in both mathematical and commonsense reasoning across all four models.

Code generation fine-tuning

ModelMethodPass@1Pass@5Pass@10Code Avg.ARC-COB-QASI-QACommonsense Avg.
Llama-3.2-1BCKA-LoRA*35.845.649.243.526.930.639.932.5
Llama-3.2-1BLS-LoRA35.845.649.243.526.930.639.932.5
Llama-3.2-3BCKA-LoRA51.161.062.658.267.262.463.964.5
Llama-3.2-3BLS-LoRA52.662.164.359.668.965.265.766.6
Llama-3-8BCKA-LoRA53.562.967.261.266.460.461.562.8
Llama-3-8BLS-LoRA53.265.268.862.472.663.865.767.4
Mistral-7BCKA-LoRA39.050.553.847.848.538.842.543.3
Mistral-7BLS-LoRA40.351.154.548.648.939.244.144.1

On Llama-3.2-1B, CKA-LoRA selects exactly the same layers as LS-LoRA and therefore shares its results.

On the other three models, LS-LoRA achieves higher mean pass@k and average commonsense accuracy than CKA-LoRA, supporting a better adaptation–retention trade-off.

These comparisons are included in Tables 2 and 3 of the revised manuscript. We additionally report mathematical reasoning comparisons with the number of adapted layers held fixed and means and standard deviations over three random-seed runs in Table 4 and Appendix G, Table 12, supporting the benefit of the criterion beyond layer sparsity alone.

2. Direct comparison with Fisher-based selection.

Fisher-LoRA directly tests whether cosine-based selection recovers the benefits of selecting layers using the sensitivity measure it approximates. On Llama-3.2-1B, Fisher-LoRA and LS-LoRA select exactly the same layers and share the same results. On the other three models, LS-LoRA achieves comparable mathematical and commonsense accuracy to Fisher-LoRA. The FLOPs comparison quantifies the computational advantage of cosine-based selection. Across the four models, LS-LoRA reduces layer-scoring FLOPs by 68.8–73.2% relative to Fisher-LoRA.

ModelFisher-LoRA: AccuracyLS-LoRA: AccuracyFisher-LoRA: Layer-scoring FLOPsLS-LoRA: Layer-scoring FLOPs
Llama-3.2-1B52.1 / 31.652.1 / 31.616.84.5
Llama-3.2-3B66.9 / 58.767.0 / 58.443.813.0
Llama-3-8B71.5 / 57.071.6 / 56.398.830.8
Mistral-7B56.6 / 46.956.3 / 46.387.225.4

These results support cosine similarity as an inexpensive approximation for layer selection. Direct empirical Fisher scoring requires target responses and backward computation, whereas cosine scoring requires only forward computation on calibration inputs. Thus, LS-LoRA approaches direct Fisher-based selection in both evaluated metrics with lower scoring cost.

AuthorsOpenReview ↗

Official Comment by Authors

3. Distinction from prior work and clarification of our contribution.

We acknowledge that ShortGPT already uses input–output similarity for pruning and that Ogawa et al. establish representation-based LoRA placement. Our contribution is supported by the comparisons above: cosine-based selection improves the observed adaptation–retention trade-off over CKA-based selection at the same layer budget and approaches direct Fisher-based selection using only forward computation.

Cosine similarity captures directional changes that can remain functionally relevant after RMSNorm but may be invisible to linear CKA.

The two criteria capture different aspects of representation change. Many modern LLMs, including LLaMA, apply RMSNorm before each Transformer sublayer, normalizing activation scale while directional changes can still affect subsequent projections with fixed weights. Cosine similarity measures the directional change between each input representation and its corresponding output, whereas linear CKA compares similarity structures across examples. In particular, an orthogonal rotation of the output features leaves linear CKA unchanged, even though individual representation directions, and consequently downstream predictions with fixed weights, can change. This distinction provides a possible explanation for cosine similarity’s closer alignment with task sensitivity in our experiments.

Cosine similarity aligns more closely with empirical Fisher sensitivity than CKA across all eight evaluated model–task pairs.

The absolute Spearman's rank correlations are 0.82–0.89 for cosine–Fisher versus 0.41–0.68 for CKA–Fisher on math, and 0.76–0.91 versus 0.44–0.74, respectively, on code. These results, reported in Appendix D, Table 8 and Figure 6, complement the direct fine-tuning comparisons above.

We have revised Section 2.2 to clarify our relationship to prior representation-based selection methods and to direct readers to the CKA- and Fisher-based comparisons in the main text and appendices.

AuthorsOpenReview ↗

Official Comment by Authors

About experimental details

We have added comparisons across multiple random seeds at matched layer counts, a threshold sensitivity analysis, and detailed training configurations. The results and settings are provided below.

1. Contribution of the selection criterion and variability across runs.

Random layer selection already improves retention over all-layer LoRA, indicating that restricting adaptation to fewer layers contributes to the improvement. To assess the additional benefit of our criterion, we compare LS-LoRA with Rand-LoRA using the same number of adapted layers for each model.

The expanded experiments cover mathematical reasoning fine-tuning on all four models. We report five runs for Rand-LoRA, with a newly sampled layer subset in each run, and three random-seed runs for LoRA and LS-LoRA. Results are accuracy percentages, reported as means ± standard deviations. Math Avg. and Commonsense Avg. denote averages across their respective three benchmarks.

ModelMethodGSM8KAQuASVAMPMath Avg.ARC-COB-QASI-QACommonsense Avg.
Llama-3.2-1BLoRA49.2 ± 0.837.9 ± 0.866.3 ± 1.251.1 ± 0.823.1 ± 0.423.4 ± 1.929.6 ± 1.525.4 ± 1.0
Llama-3.2-1BRand-LoRA50.4 ± 0.736.3 ± 1.967.1 ± 1.051.2 ± 0.723.6 ± 1.726.2 ± 2.328.7 ± 4.426.2 ± 2.6
Llama-3.2-1BLS-LoRA50.3 ± 0.138.9 ± 0.967.2 ± 1.352.1 ± 0.230.3 ± 1.131.2 ± 1.033.2 ± 0.331.6 ± 0.6
Llama-3.2-3BLoRA69.8 ± 0.548.3 ± 1.980.3 ± 0.466.1 ± 0.945.1 ± 1.852.0 ± 0.538.0 ± 3.045.0 ± 1.4
Llama-3.2-3BRand-LoRA69.9 ± 0.546.6 ± 1.781.2 ± 1.165.9 ± 0.657.9 ± 1.559.9 ± 0.952.7 ± 2.956.8 ± 1.0
Llama-3.2-3BLS-LoRA70.1 ± 0.548.4 ± 1.182.5 ± 2.167.0 ± 0.459.7 ± 0.861.9 ± 1.553.8 ± 1.458.4 ± 0.3
Llama-3-8BLoRA69.4 ± 0.562.5 ± 1.679.0 ± 2.270.3 ± 1.250.5 ± 1.350.3 ± 1.146.4 ± 1.249.1 ± 1.1
Llama-3-8BRand-LoRA70.7 ± 0.759.9 ± 0.479.3 ± 1.069.9 ± 0.750.8 ± 1.554.2 ± 0.949.8 ± 1.051.6 ± 0.3
Llama-3-8BLS-LoRA70.6 ± 0.265.8 ± 0.878.4 ± 0.271.6 ± 0.359.9 ± 0.956.1 ± 0.852.9 ± 1.056.3 ± 0.2
Mistral-7BLoRA58.4 ± 1.332.7 ± 0.473.1 ± 1.254.8 ± 0.734.4 ± 1.439.3 ± 1.136.6 ± 0.936.8 ± 0.9
Mistral-7BRand-LoRA61.4 ± 1.032.5 ± 1.073.6 ± 1.655.8 ± 1.040.4 ± 2.941.0 ± 2.635.9 ± 2.539.1 ± 2.5
Mistral-7BLS-LoRA61.2 ± 0.533.4 ± 1.374.2 ± 0.556.3 ± 0.449.1 ± 1.142.5 ± 0.347.4 ± 0.546.3 ± 0.5

LS-LoRA achieves higher mean mathematical and commonsense accuracy than Rand-LoRA across all four models at the same layer budget. These results support an additional benefit from selecting which layers to adapt, beyond the improvement associated with sparsity alone. The target-task gains are modest, while the commonsense retention gains are larger.

These results have been added to Table 4 and Appendix G, Table 12, alongside comparisons with shallow-layer, deep-layer, CKA-based, and Fisher-based selection.

2. Sensitivity to the threshold γ\gamma.

We have added a threshold sensitivity experiment on Llama-3.2-3B for mathematical reasoning, evaluating γ∈{0.75,0.80,0.85,0.90,1.00}\gamma \in \{0.75, 0.80, 0.85, 0.90, 1.00\} over three runs. The table below reports the mean target-task performance (TTP), measured by average accuracy across GSM8K, AQuA, and SVAMP, and general capability retention (GCR), measured by average accuracy across ARC-C, OB-QA, and SI-QA. Layer indices are one-based, and ranges are inclusive.

Threshold γ\gammaNumber of adapted layersTTP (%)GCR (%)
0.75364.059.2
0.80965.558.5
0.851667.058.4
0.902066.454.0
1.002866.145.0

The default threshold γ=0.85\gamma = 0.85 provides a useful balance between target-task performance and capability retention in this setting. Increasing γ\gamma from 0.75 to 0.85 improves mathematical accuracy while commonsense accuracy decreases only slightly. Increasing it further reduces both metrics, with a particularly pronounced decline in retention. At γ=1.00\gamma = 1.00, all layers receive adapters, recovering the all-layer LoRA configuration.

This analysis examines the trade-off between adaptation breadth and performance and supports the default threshold in the evaluated setting. The results and selected layer sets have been added to Section 4.4, Figure 5, and Appendix E, Table 10.

Reviewer_nz5iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Layer-Selective LoRA (LS-LoRA) to balance task adaptation and capability retention. The method computes layer-wise input–output cosine similarity on a calibration set and applies LoRA only to layers with scores below a threshold. The authors motivate this criterion through its negative correlation with empirical Fisher scores and a theoretical lower bound. Experiments on mathematical reasoning and code generation across four models show competitive target-task performance and improved commonsense retention compared with all-layer LoRA and regularization-based baselines, alongside reduced time and memory usage.

Critical strengths
  1. The method is simple and efficient, selecting adaptation layers without labels or backpropagation.
  2. Experiments show competitive target-task performance and improved commonsense capability retention across multiple models.
Critical weaknesses
  1. The substantial increase in commonsense accuracy after code fine-tuning, particularly for Llama-3-8B, requires further explanation.
  2. Comparisons with fixed shallow-layer and deep-layer selection are needed to establish the benefit of the proposed selection criterion.
Questions for authors

Please see the weaknesses above.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1: The substantial increase in commonsense accuracy after code fine-tuning, particularly for Llama-3-8B, requires further explanation.

Response: Thank you for highlighting this unexpected result. We traced the apparent increase for Llama-3-8B to an answer-extraction issue in the original evaluation. The base model did not consistently follow the required answer format, preventing the evaluation script from extracting its answers and underestimating its commonsense accuracy.

We have added explicit answer-format instructions to the evaluation prompt. These instructions require the response to end with “The answer is (X),” where X denotes an answer-choice label. After re-evaluating the base model with this constraint, we updated its results:

Llama-3-8B configurationARC-COB-QASI-QAAverage
Base, before fine-tuning77.269.868.671.9
After code fine-tuning with LoRA68.953.262.961.7
After code fine-tuning with LS-LoRA72.663.865.767.4

The corrected base accuracy is higher than both fine-tuned results. Relative to this baseline, commonsense accuracy decreases by 10.2 percentage points with LoRA and 4.5 points with LS-LoRA. Thus, the updated results show that LS-LoRA reduces the degradation associated with code fine-tuning on these benchmarks.

We have updated the results in the main text and documented the answer-format instructions in Appendix F.

W2: Comparisons with fixed shallow-layer and deep-layer selection are needed to establish the benefit of the proposed selection criterion.

Thank you for this suggestion. We have added Shallow-LoRA and Deep-LoRA, which adapt fixed subsets of shallow and deep layers, respectively. For each model, both baselines adapt the same number of layers as LS-LoRA, allowing us to compare layer placement while controlling the number of adapted layers.

The experiments cover mathematical reasoning fine-tuning on all four models. Results are accuracy percentages averaged over three random-seed runs. Math Avg. and Commonsense Avg. denote averages across the corresponding three benchmarks. Bold indicates the highest category average among the three methods for each model.

ModelMethodGSM8KAQuASVAMPMath Avg.ARC-COB-QASI-QACommonsense Avg.
Llama-3.2-1BShallow-LoRA48.837.267.351.124.928.330.728.0
Llama-3.2-1BDeep-LoRA50.036.665.750.825.529.231.028.6
Llama-3.2-1BLS-LoRA50.338.967.252.130.331.233.231.6
Llama-3.2-3BShallow-LoRA67.948.480.165.450.656.444.550.5
Llama-3.2-3BDeep-LoRA68.947.181.665.948.544.137.143.2
Llama-3.2-3BLS-LoRA70.148.482.567.059.761.953.858.4
Llama-3-8BShallow-LoRA69.258.677.568.452.650.549.750.9
Llama-3-8BDeep-LoRA69.357.978.768.654.548.948.950.8
Llama-3-8BLS-LoRA70.665.878.471.659.956.152.956.3
Mistral-7BShallow-LoRA*61.233.474.256.349.142.547.446.3
Mistral-7BDeep-LoRA54.529.369.351.038.738.135.437.4
Mistral-7BLS-LoRA61.233.474.256.349.142.547.446.3

On Mistral-7B, Shallow-LoRA selects exactly the same layers as LS-LoRA and therefore shares its results.

LS-LoRA achieves higher mean mathematical and commonsense accuracy than both fixed-placement baselines on all three Llama models. On Mistral-7B, it matches Shallow-LoRA and outperforms Deep-LoRA. These comparisons support the benefit of the proposed selection criterion over fixed shallow or deep placement when the selected layer sets differ, with the number of adapted layers held constant.

We have added these comparisons to Table 4 and Appendix G (Table 12) of the revised manuscript.

Thank you for the detailed rebuttal. The explanation of the answer-extraction issue adequately addresses my first concern, although I encourage the authors to check whether similar issues affect the results of other models. The additional comparisons with Shallow-LoRA and Deep-LoRA also provide convincing evidence for the effectiveness of the proposed layer-selection criterion.

Based on the current version, I've raised my soundness score to 3.

AuthorsOpenReview ↗

Response to Reviewer nz5i

Thank you for this suggestion. Following your recommendation, we further examined whether the answer-format issue also affected other models. We re-evaluated Llama-3.2-1B, Llama-3.2-3B, and Mistral-7B using the explicit answer-format instructions added to the evaluation prompt. The results are summarized below.

ModelPromptARC-COB-QASI-QAAvg.
Llama-3.2-1BOriginal prompt28.830.640.833.4
Prompt with answer-format instructions30.732.846.836.8
Llama-3.2-3BOriginal prompt62.256.458.859.1
Prompt with answer-format instructions70.065.866.567.4
Mistral-7BOriginal prompt66.661.465.164.4
Prompt with answer-format instructions69.067.266.867.7

The results show that the average commonsense accuracy increases by 3.4, 8.3, and 3.3 percentage points for Llama-3.2-1B, Llama-3.2-3B, and Mistral-7B, respectively, after adding the explicit answer-format instructions. These findings indicate that the evaluation prompt also affects the measured results of other models. We have updated the corresponding commonsense evaluation results in the revised manuscript.

To ensure consistent evaluation, we also re-evaluated the corresponding fine-tuned models, including LoRA, LS-LoRA, and all other compared baselines, using the same answer-format instructions. We have updated the corresponding results in the revised manuscript. The updated results continue to show that LS-LoRA achieves higher average commonsense accuracy than standard LoRA across these models after both mathematical reasoning and code generation fine-tuning.

We sincerely appreciate your positive reassessment of the paper's soundness. We hope that the additional checks and revisions have addressed your remaining concern, and we would be grateful if you could reconsider your overall rating in light of these updates.

AuthorsOpenReview ↗

Follow-up – Reviewer nz5i

Thank you again for your earlier feedback. We wanted to briefly follow up on our response to your suggestion regarding the answer-format issue. We have completed the additional cross-model checks and re-evaluated the corresponding fine-tuned configurations using the same answer-format instruction. The updated results are included in our response and the revised manuscript.

As the discussion period draws to a close, we would appreciate it if you could take a look at these additional results and let us know whether they address your concern. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Revisions

Dear Area Chair and Reviewers,

Thank you for the constructive discussion. We would like to summarize our contribution and the key revisions made in response to the reviewers’ feedback.

Our paper investigates where to adapt an LLM to improve target-task performance while retaining general capabilities. LS-LoRA uses layer-wise input–output cosine similarity computed from a small set of target-task calibration inputs to select layers for adaptation, without requiring response labels or backpropagation during selection. Experiments across four models on mathematical reasoning and code generation show competitive target-task performance and improved commonsense capability retention compared with all-layer LoRA. Reviewers bWp4, G5kU, and nz5i recognized the method’s simplicity, efficiency, and empirical benefits. bWp4 and G5kU also highlighted results suggesting its applicability to other PEFT methods, including DoRA and PiSSA.

In response to the main concerns, we strengthened the manuscript in three respects:

1. Comparative evidence and methodological contribution — bWp4, G5kU, and nz5i. Following bWp4 and G5kU, we added direct comparisons with CKA-based and Fisher-based layer selection and expanded matched-layer-count comparisons with random selection. Following nz5i, we also added fixed shallow-layer and deep-layer selection baselines. These comparisons, with means and standard deviations reported across multiple runs, support the value of choosing which layers to adapt beyond simply reducing the number of adapted layers.

2. Evaluation reliability and coverage — nz5i and G5kU. We identified and corrected answer-format mismatches that affected commonsense evaluation, re-evaluated the relevant models and configurations under a consistent protocol. Following G5kU, we also added BoolQ, HellaSwag, and PIQA evaluations on Llama-3.2-1B to assess general capability retention across a broader range of benchmarks. The updated results continue to support improved capability retention with LS-LoRA.

3. Interpretation and reproducibility — bWp4 and G5kU. We added a threshold sensitivity analysis, complete selected-layer indices and counts, and detailed training configurations and evaluation prompts. We also clarified the scope of the theoretical analysis and distinguished target-task sensitivity from the experimentally demonstrated retention benefits.

We appreciate the reviewers’ subsequent reassessments: bWp4 raised the overall rating from 2 to 3 after noting that most concerns had been addressed; G5kU confirmed that their concerns were addressed and raised the rating from 2 to 4; and nz5i raised the soundness score to 3, finding the evaluation explanation adequate and the additional layer-placement comparisons convincing. Following nz5i’s suggestion, we subsequently completed additional cross-model checks using consistent answer-format instructions.

Overall, the feedback has helped us strengthen comparative evidence, improve evaluation reliability, and clarify the contribution and its scope. We sincerely thank the reviewers and the Area Chair for their time and consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.