- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the mechanisms underlying layer adaptability in Transformer-based language models, focusing on why certain layers contribute more effectively to fine-tuning than others. The authors propose a theoretical framework that characterizes layer adaptability through two factors: functional signal, which measures the loss-reducing output corrections a layer can express, and spectral learning strength, which determines how efficiently those corrections can be learned.
The paper further examines how joint training redistributes functional contributions across layers and demonstrates that later layers can partially compensate for the loss of adaptability when earlier layers are frozen. Based on these theoretical and empirical findings, the authors propose a configurable broad-to-focused fine-tuning strategy, which initially trains across all layers before restricting updates to a task-dependent subset of output-nearest layers.
The proposed implementation, Suffix95, selects the smallest output-nearest layer subset that accounts for 95% of the measured functional change. Experiments across 39 model–dataset combinations demonstrate competitive performance compared with full-depth fine-tuning baselines while reducing training computation and memory consumption.
Critical strengths
S1. A theoretically grounded perspective on layer adaptability beyond conventional gradient-based importance measures.
The paper provides an interesting perspective on understanding layer adaptability by distinguishing between the functional corrections a layer can express and the spectral properties that determine how efficiently those corrections can be learned.
Rather than relying solely on conventional indicators such as gradient magnitude or parameter movement, the proposed analysis considers the actual contribution of each layer to loss-reducing output changes. This distinction is meaningful because layers with similar gradient magnitudes may exhibit substantially different functional contributions.
The theoretical characterization offers a useful foundation for understanding why certain layers are more effective for fine-tuning and provides insights that may extend beyond the specific optimization strategy proposed in this paper.
S2. An interesting analysis of functional redistribution during joint training.
The paper investigates how the functional contributions of individual layers evolve when multiple layers are trained jointly. In particular, the authors demonstrate that freezing earlier layers does not necessarily eliminate their entire contribution, as the remaining trainable layers can partially compensate for the missing corrections.
This observation provides a meaningful explanation for why selective fine-tuning can preserve model performance despite restricting trainable layers. The controlled experiments examining functional overlap, specialization, and compensation further strengthen the connection between the theoretical analysis and observed training behavior.
S3. A practical fine-tuning strategy motivated by theoretical and empirical observations.
The proposed broad-to-focused strategy connects the theoretical characterization of layer adaptability with practical computational constraints.
By combining early broad adaptation with task-dependent selection of output-nearest layers, Suffix95 achieves competitive performance while reducing training computation and memory consumption. The evaluation across multiple model configurations and datasets provides empirical support for the practical applicability of the proposed approach.
Critical weaknesses
W1. Limited methodological novelty of the proposed fine-tuning strategy.
Although the theoretical analysis provides interesting insights into layer adaptability, the resulting optimization strategy is relatively straightforward: train broadly during the early stages and subsequently freeze earlier layers while continuing to optimize the remaining layers.
The general idea of selectively training layers, dynamically freezing parameters, or concentrating adaptation on particular parts of a model has been explored in previous parameter-efficient fine-tuning approaches.
While the paper provides a theoretical motivation for this strategy, it remains unclear whether Suffix95 introduces a fundamentally new optimization mechanism or primarily offers a theoretically motivated layer-selection heuristic.
In particular, the distinction between the proposed approach and existing selective fine-tuning methods should be established more explicitly through direct methodological comparisons.
W2. Insufficient validation of the relationship between functional coverage and downstream performance.
A central assumption underlying Suffix95 is that retaining layers that account for a sufficiently large proportion of measured functional change can preserve downstream task performance.
However, the proposed coverage metric measures how well a subset of layers reproduces the accumulated output changes observed during training. This does not necessarily establish that those changes are equally important for downstream generalization.
In particular, reproducing 95% of the measured functional change does not guarantee preservation of 95% of the task-relevant information or predictive performance. Moreover, the functional contributions of layers may change after earlier layers are frozen, potentially weakening the relationship between the measured coverage and the actual effectiveness of the retained layers.
Although the experiments demonstrate competitive median performance, the paper does not sufficiently establish whether functional coverage is a reliable predictor of downstream performance across different tasks and model architectures.
W3. Limited evidence supporting the generality of the proposed layer-adaptability principles.
The paper presents its theoretical framework as a general characterization of layer adaptability in language models. However, several controlled experiments supporting the mechanistic claims are conducted primarily on relatively small GPT-2 models.
While the broader empirical evaluation includes Qwen3-8B and LLaMA2-7B, the detailed analyses of functional specialization and compensation after freezing are not demonstrated to the same extent on these larger models.
Furthermore, the practical strategy consistently prioritizes output-nearest layers, partly because of their computational advantages. However, the theoretical framework itself suggests that layer adaptability depends on task-specific functional signals and learning strength rather than depth alone.
This raises the question of whether output-nearest layers are consistently the most appropriate choice for preserving functional adaptability, or whether their selection is primarily motivated by computational efficiency.
The paper would benefit from a clearer distinction between functionally optimal layer selection and computationally efficient layer placement.
Questions for authors
Q1. What fundamentally distinguishes Suffix95 from existing selective fine-tuning and progressive layer-freezing approaches?
The paper provides a theoretically motivated explanation for why broad-to-focused fine-tuning can be effective. However, the resulting training procedure appears closely related to existing approaches that selectively update layers or progressively freeze model parameters.
Could the authors clarify the methodological novelty of Suffix95 beyond its functional coverage-based layer-selection criterion?
In particular, comparisons with simpler strategies, such as fixed output-nearest layer selection, progressive freezing, or gradient-based layer selection under matched computational budgets, would help establish whether the proposed functional analysis leads to meaningful practical improvements.
Q2. How strongly does functional coverage correlate with downstream task performance?
Suffix95 selects the smallest output-nearest layer subset that reproduces 95% of the accumulated functional change measured during training.
However, it remains unclear whether this coverage metric reliably predicts downstream performance, particularly when the retained layers are subsequently optimized under a different training configuration.
Could the authors provide a quantitative analysis of the relationship between functional coverage and downstream performance across different layer subsets, datasets, and model architectures?
Additionally, what motivates the selection of 95% coverage, and under what conditions might high functional coverage fail to preserve task performance?
Q3. Are output-nearest layers inherently more adaptable, or are they primarily more computationally efficient?
The theoretical analysis suggests that layer adaptability depends on functional signal, spectral learning strength, and interactions among layers. These factors are task-dependent and do not necessarily imply that later layers are always more adaptable.
However, the proposed fine-tuning strategy exclusively considers output-nearest layer subsets, partly because they reduce backward computation and activation storage.
Could the authors compare output-nearest layer selection with non-contiguous or functionally selected layer subsets under matched computational budgets?
Such an analysis would help clarify whether the effectiveness of Suffix95 arises from the intrinsic adaptability of later layers or primarily from the computational advantages of restricting training to output-nearest layers.
4 comments
Thank you for encouraging broader validation of the mechanisms (2/4)
W3: “Limited evidence supporting the generality of the proposed layer-adaptability principles. [...] the detailed analyses of functional specialization and compensation after freezing are not demonstrated to the same extent on these larger models. [...] This raises the question of whether output-nearest layers are consistently the most appropriate choice for preserving functional adaptability, or whether their selection is primarily motivated by computational efficiency.”
Response: Compensation is now tested on Qwen3-8B, the tension across depth on three models, and the revision separates the reasons for output-side layers.
Tension across depth. On Qwen3-8B, potential falls with depth and activation energy rises on MNLI and IMDB (Fig. 2; rank correlations −0.95 to −0.97 and 0.98 to 0.99), and at the switch the gradient ranks the cost of freezing the wrong way (−0.61 over 36 runs). The same opposite trends hold on Qwen3-1.7B and LLaMA2-7B (new Fig. A.1, App. A.5).
Compensation on Qwen3-8B. Over four tasks and three seeds (Fig. 3a, App. A.6):
- Retraining lowers the extra loss for every kept set.
- Suffix's kept layers win back 52% of their extra loss, against 36% for Fixed-18, and after retraining Suffix's extra loss is 0.34% against 1.52%.
- Adapting the kept layers adds 0.98 test-accuracy points over the same frozen layers without adaptation (9 of 12 pairs, p = 0.011).
Compensation at scale. In the broad validation, Suffix freezes the input-side layers, which hold the most potential, up to 24 of 36 on Qwen3-8B, yet performance stays near full depth across GPT-2, Qwen3-8B, and LLaMA2-7B (Sec. 7, Table 4).
Output-side layers. They are chosen for both reasons, and the revision says which is which (Sec. 4.1): cost decides the shape, an output-side set, because only such a set shortens the backward pass, and coverage decides its length for each task (see Q3).
Thank you for requesting direct comparisons at matched budgets (3/4)
Q1: “What fundamentally distinguishes Suffix95 from existing selective fine-tuning and progressive layer-freezing approaches? [...] comparisons with simpler strategies, such as fixed output-nearest layer selection, progressive freezing, or gradient-based layer selection under matched computational budgets, would help establish whether the proposed functional analysis leads to meaningful practical improvements.”
Response: The distinction is the derived, per-task depth decision (W1). The revision compares it head to head with each strategy you list:
Head-to-head comparison on Qwen3-8B (accuracy relative to Suffix in points, six tasks, three seeds)
| Method | Strategy | Accuracy vs. Suffix | FLOPs saved | Pairs Suffix wins |
|---|---|---|---|---|
| Last-9 | fixed output-nearest | −6.25 | 46.7% | 18 of 18 |
| Last-18 | fixed output-nearest | −3.30 | 31.1% | 16 of 18 |
| Fixed-18 | fixed, after the same switch | −2.44 | 30.1% | 17 of 18 |
| AutoFreeze | progressive freezing | +0.01 | 3.4% | 6 of 18 |
| AFLoRA | gradient-based selection | −2.90 | 0% | 15 of 18 |
| Suffix | coverage, per task | 0 | 17.3% |
Matched budgets. A fixed depth matched to Suffix's cost on each task needs hindsight: Suffix's own choices, or a sweep of depths per task that costs several full runs. Suffix gives what such a sweep would find from one run, so we compare it with the fixed-depth accuracy–compute curve instead (Sec. 6, App. B.2):
Fixed depths against Suffix, relative to Full (accuracy change in points; FLOPs saved)
| Method | Accuracy change | FLOPs saved |
|---|---|---|
| Last-9 (last quarter, from the start) | −6.27 | 46.7% |
| Last-18 (last half, from the start) | −3.32 | 31.1% |
| Fixed-18, switch at step 3 | −4.21 | 31.1% |
| Fixed-18, switch at step 12 | −3.19 | 30.1% |
| Fixed-18, switch at step 48 | −1.84 | 26.2% |
| Fixed-18, switch at step 120 | −0.85 | 18.2% |
| Suffix, automatic | −0.17 | 16.9% |
The Fixed-18 rows and Suffix's row come from the switch-timing runs (four tasks, two seeds), and Last-9 and Last-18 from Table 3 (six tasks, three seeds). Suffix lies above the fixed-depth curve: at nearly the same saving as Fixed-18 switched at step 120, it loses 0.17 points where Fixed-18 loses 0.85. Spending more compute on a fixed depth closes only part of the gap, because one depth is too shallow for the demanding tasks and more than the easy ones need.
A rule of thumb, found automatically. The depths Suffix chooses also give the rule a fixed-depth user would want. On twelve Qwen3-8B tasks, six need at most half of the 36 layers (sentiment, topic, HellaSwag, SVAMP: 12 to 18) and six need more (GSM8K, BoolQA, WiC, MathQA, RTE, MNLI: 20 to 36), and every LLaMA2-7B task needs more than half (19 to 32 of 32). So the last quarter is too shallow everywhere, and half suffices for sentiment and topic tasks but not for entailment and question answering, as the controlled runs confirm: Fixed-18 ties Suffix on AG News and IMDB and trails it by 2.65 to 4.54 points on RTE, WiC, BoolQA, and CSQA (new App. B.6 paragraph “A rule of thumb”).
Thank you for your questions on the coverage target and output-side layers (4/4)
Q2: “How strongly does functional coverage correlate with downstream task performance? [...] Additionally, what motivates the selection of 95% coverage, and under what conditions might high functional coverage fail to preserve task performance?”
Response: Correlation. Coverage ranks the accuracy lost against Full at 0.77, against 0.40 for layer count and 0.12 for gradient size, over 72 runs (W2).
Why 95%. On the 27 Qwen3 and LLaMA2 combinations, 95% beats 90% in 19 of the 23 where the two differ (p = 0.003), and lower targets trade accuracy for saving (Table 5):
Coverage targets against all-layer LoRA (20 Qwen3 and LLaMA2 combinations; accuracy change relative to LoRA)
| Target | FLOPs saved | Median accuracy change | Runs losing more than 3 points |
|---|---|---|---|
| 95% | 22.3% | +0.54% | 5% |
| 90% | 27.4% | −0.31% | 20% |
| 80% | 36.8% | −1.27% | 25% |
When coverage of the training loss and accuracy diverge. In the controlled runs they agree, with a rank correlation of 0.80 between the extra training loss and the accuracy lost. They can diverge when a fixed answer template dominates the training loss, and there coverage can be measured with accuracy instead, which recovers about 7 points on MRPC and CoLA (W2, App. B.3). If coverage calls for every layer, Suffix falls back to full-depth LoRA.
Q3: “Are output-nearest layers inherently more adaptable, or are they primarily more computationally efficient? [...] Could the authors compare output-nearest layer selection with non-contiguous or functionally selected layer subsets under matched computational budgets?”
Response: No layer is inherently more adaptable: the first mechanism shows that what a layer contributes is set by the training dynamics, and coverage measures that contribution directly, including compensation. Output-side layers are then chosen because, at any given cost, they give the most coverage.
The first mechanism: adaptability is dynamic. A layer's gradient is its potential, what it could fix, times its activation energy, how fast it learns, and the two run against each other with depth (Sec. 3.1). Input-side layers have the most potential, output-side layers learn fastest, and which layers do the work changes during training (Table 2). What a layer actually delivers in a run is its potential realized at its own speed. On 29 Qwen3-8B tasks, the change each layer actually makes ranks layers almost opposite to potential (median rank correlation −0.62; see our answer to 6tPJ Q4). So neither potential nor speed alone says which layers are most adaptable, and asking whether later layers are inherently better has no fixed answer.
Coverage measures the contribution directly. Coverage is built from the change each layer actually makes in the run, and it asks what the kept layers can reproduce together, so it includes both realization and compensation (Prop. 3.3). It predicts the accuracy lost at 0.77 over 72 runs, against 0.40 for layer count and 0.12 for gradient size (Fig. 3b). One by one, output-side layers are not the strongest: on RTE, six scattered layers reproduce 80% of Full's later change against 44% for the last six (App. A.7 data, 256 examples).
Why output-side sets, at matched cost. A backward pass must reach the earliest trained layer, so a set's cost is set by its earliest layer: keeping four layers at the output instead of the input cuts training time by 30 to 34% and peak memory by 56% (Sec. 4.1, Fig. 4a). Among all sets with the same earliest layer, the output-side suffix contains every other set and so has the most coverage (Cor. 3.4). On the same RTE data, at the cost of the last 18 layers, the suffix covers 99% of Full's change against a median of 84% for scattered sets of nine layers reaching the same depth. So cost decides the shape, an output-side set, and coverage decides its length for each task.
Measured comparisons. The trained baselines confirm this:
- AFLoRA selects scattered LoRA matrices by a gradient score. It saves no FLOPs, because the backward pass still reaches the first layer, and it is 2.90 points less accurate, because the score favors fast output-side matrices over slower ones that still had work to do (Sec. 6).
- LISA samples non-contiguous layers at random. Against it, Suffix's median accuracy is 2.32% higher on Qwen3-8B (Table 4), and Suffix saves 20.6% of FLOPs over seven runs at the 95% target (Table 5).
- AutoFreeze freezes a prefix only after gradients settle, so it keeps accuracy but saves 3.4% against Suffix's 17.3%.
Thank you for your thoughtful review and questions on novelty and coverage (1/4)
Dear Reviewer ArGX,
Thank you for the thoughtful review. We are glad you found the theoretical perspective useful and the compensation analysis “a meaningful explanation for why selective fine-tuning can preserve model performance”. We answer W1 to W3 and Q1 to Q3 in your numbering.
W1: “Limited methodological novelty of the proposed fine-tuning strategy. [...] it remains unclear whether Suffix95 introduces a fundamentally new optimization mechanism or primarily offers a theoretically motivated layer-selection heuristic. In particular, the distinction between the proposed approach and existing selective fine-tuning methods should be established more explicitly through direct methodological comparisons.”
Response: Suffix leaves the optimizer unchanged; what is new is how it decides which layers to train, and that decision is derived rather than heuristic.
Derived rule. Cor. 3.4 shows that the shortest output-side set reaching a coverage target is the cheapest set whose bound on the extra loss stays within that target, and Prop. 3.3(c) shows that training the kept set reaches the floor that coverage sets. Suffix keeps exactly this set, chosen once per task, early, from one run.
How it differs. Table 1 of the revision contrasts the methods by what they use to decide:
Direct comparison. On Qwen3-8B with six tasks and three seeds, Suffix is more accurate than every fixed rule and AFLoRA and as accurate as AutoFreeze at five times its saving (Q1 has the full table).
W2: “Insufficient validation of the relationship between functional coverage and downstream performance. [...] reproducing 95% of the measured functional change does not guarantee preservation of 95% of the task-relevant information or predictive performance. Moreover, the functional contributions of layers may change after earlier layers are frozen [...]”
Response: The revision tests this directly, and coverage predicts downstream accuracy best among the measures we compare.
Coverage and accuracy. Over 72 Qwen3-8B runs on six tasks, three seeds, and four kept sets:
How well each measure predicts the cost of freezing (rank correlation, 72 runs)
Contributions after freezing. Your point that contributions change after freezing is exactly compensation, now made precise by Prop. 3.3. We measured it directly by computing the coverage of Full's own updates after the switch: Suffix's kept set covers 99 to 100% of them, so the estimate taken before freezing is conservative, and its depth follows each task's true need (App. A.7).
Choosing what to preserve. Your concern also leads to an extension. Coverage can be measured with any per-example loss, and Prop. 3.3 holds for any such loss, so the user can preserve the metric they care about while training stays unchanged (App. B.3). This matters when a fixed answer template dominates the training loss:
Suffix with the training loss and with accuracy as the target (accuracy in %, kept layers in parentheses, Qwen3-8B, seed 110)
Knowing when to stop. At the switch, Suffix also reads how much of the chosen target is left, and it stops when almost none is (App. B.3):
The stop check at the switch (accuracy in %, Qwen3-8B, seed 110)
In the worst case coverage calls for every layer, and Suffix reduces to full-depth LoRA.
References