acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Layer Adaptable? A Mechanistic Study of Language Model Fine-Tuning

Abstract

What makes one Transformer layer more useful to fine-tune than another? We judge a layer by the loss-reducing output change, or correction, it can produce. Our first characterization separates the correction a layer can express from the spectral strength that determines how readily it is learned. These factors can trend in opposite directions with depth and produce a nearly flat gradient profile, so gradient size can hide clear differences in what layers can correct. Our second characterization shows how joint training redistributes the remaining error and available output directions across layers; as a result, when earlier layers are frozen, the retained layers can absorb part of the correction the frozen layers would otherwise have made. Controlled GPT-2 experiments test these mechanisms, and a systems study establishes the cost advantage of placing trainable layers nearest the output. Across 15 datasets on each of Qwen3-8B and LLaMA2-7B, most new loss-relevant output change occurs early in training, and the share of accumulated change covered by each output-nearest layer set remains stable over training within a dataset but differs strongly across datasets. These results yield a roadmap parameterized by when to stop broad training and how many output-nearest layers to retain, spanning all-layer, focus-only, and broad-to-focused tuning under different resource budgets. One implementation, Suffix95, trains across all layers until a chosen step and then retains the smallest output-nearest layer set covering % of the measured all-layer change. Across 39 model and dataset combinations, it achieves competitive median performance relative to full-depth baselines while reducing total training compute by –% and post-transition memory by –%.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, explainable AI, layer adaptability, Transformer fine-tuning, training dynamics, layer specialization, selective fine-tuning, functional coverage

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ArGXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mechanisms underlying layer adaptability in Transformer-based language models, focusing on why certain layers contribute more effectively to fine-tuning than others. The authors propose a theoretical framework that characterizes layer adaptability through two factors: functional signal, which measures the loss-reducing output corrections a layer can express, and spectral learning strength, which determines how efficiently those corrections can be learned.

The paper further examines how joint training redistributes functional contributions across layers and demonstrates that later layers can partially compensate for the loss of adaptability when earlier layers are frozen. Based on these theoretical and empirical findings, the authors propose a configurable broad-to-focused fine-tuning strategy, which initially trains across all layers before restricting updates to a task-dependent subset of output-nearest layers.

The proposed implementation, Suffix95, selects the smallest output-nearest layer subset that accounts for 95% of the measured functional change. Experiments across 39 model–dataset combinations demonstrate competitive performance compared with full-depth fine-tuning baselines while reducing training computation and memory consumption.

Critical strengths

S1. A theoretically grounded perspective on layer adaptability beyond conventional gradient-based importance measures.

The paper provides an interesting perspective on understanding layer adaptability by distinguishing between the functional corrections a layer can express and the spectral properties that determine how efficiently those corrections can be learned.

Rather than relying solely on conventional indicators such as gradient magnitude or parameter movement, the proposed analysis considers the actual contribution of each layer to loss-reducing output changes. This distinction is meaningful because layers with similar gradient magnitudes may exhibit substantially different functional contributions.

The theoretical characterization offers a useful foundation for understanding why certain layers are more effective for fine-tuning and provides insights that may extend beyond the specific optimization strategy proposed in this paper.

S2. An interesting analysis of functional redistribution during joint training.

The paper investigates how the functional contributions of individual layers evolve when multiple layers are trained jointly. In particular, the authors demonstrate that freezing earlier layers does not necessarily eliminate their entire contribution, as the remaining trainable layers can partially compensate for the missing corrections.

This observation provides a meaningful explanation for why selective fine-tuning can preserve model performance despite restricting trainable layers. The controlled experiments examining functional overlap, specialization, and compensation further strengthen the connection between the theoretical analysis and observed training behavior.

S3. A practical fine-tuning strategy motivated by theoretical and empirical observations.

The proposed broad-to-focused strategy connects the theoretical characterization of layer adaptability with practical computational constraints.

By combining early broad adaptation with task-dependent selection of output-nearest layers, Suffix95 achieves competitive performance while reducing training computation and memory consumption. The evaluation across multiple model configurations and datasets provides empirical support for the practical applicability of the proposed approach.

Critical weaknesses

W1. Limited methodological novelty of the proposed fine-tuning strategy.

Although the theoretical analysis provides interesting insights into layer adaptability, the resulting optimization strategy is relatively straightforward: train broadly during the early stages and subsequently freeze earlier layers while continuing to optimize the remaining layers.

The general idea of selectively training layers, dynamically freezing parameters, or concentrating adaptation on particular parts of a model has been explored in previous parameter-efficient fine-tuning approaches.

While the paper provides a theoretical motivation for this strategy, it remains unclear whether Suffix95 introduces a fundamentally new optimization mechanism or primarily offers a theoretically motivated layer-selection heuristic.

In particular, the distinction between the proposed approach and existing selective fine-tuning methods should be established more explicitly through direct methodological comparisons.

W2. Insufficient validation of the relationship between functional coverage and downstream performance.

A central assumption underlying Suffix95 is that retaining layers that account for a sufficiently large proportion of measured functional change can preserve downstream task performance.

However, the proposed coverage metric measures how well a subset of layers reproduces the accumulated output changes observed during training. This does not necessarily establish that those changes are equally important for downstream generalization.

In particular, reproducing 95% of the measured functional change does not guarantee preservation of 95% of the task-relevant information or predictive performance. Moreover, the functional contributions of layers may change after earlier layers are frozen, potentially weakening the relationship between the measured coverage and the actual effectiveness of the retained layers.

Although the experiments demonstrate competitive median performance, the paper does not sufficiently establish whether functional coverage is a reliable predictor of downstream performance across different tasks and model architectures.

W3. Limited evidence supporting the generality of the proposed layer-adaptability principles.

The paper presents its theoretical framework as a general characterization of layer adaptability in language models. However, several controlled experiments supporting the mechanistic claims are conducted primarily on relatively small GPT-2 models.

While the broader empirical evaluation includes Qwen3-8B and LLaMA2-7B, the detailed analyses of functional specialization and compensation after freezing are not demonstrated to the same extent on these larger models.

Furthermore, the practical strategy consistently prioritizes output-nearest layers, partly because of their computational advantages. However, the theoretical framework itself suggests that layer adaptability depends on task-specific functional signals and learning strength rather than depth alone.

This raises the question of whether output-nearest layers are consistently the most appropriate choice for preserving functional adaptability, or whether their selection is primarily motivated by computational efficiency.

The paper would benefit from a clearer distinction between functionally optimal layer selection and computationally efficient layer placement.

Questions for authors

Q1. What fundamentally distinguishes Suffix95 from existing selective fine-tuning and progressive layer-freezing approaches?

The paper provides a theoretically motivated explanation for why broad-to-focused fine-tuning can be effective. However, the resulting training procedure appears closely related to existing approaches that selectively update layers or progressively freeze model parameters.

Could the authors clarify the methodological novelty of Suffix95 beyond its functional coverage-based layer-selection criterion?

In particular, comparisons with simpler strategies, such as fixed output-nearest layer selection, progressive freezing, or gradient-based layer selection under matched computational budgets, would help establish whether the proposed functional analysis leads to meaningful practical improvements.

Q2. How strongly does functional coverage correlate with downstream task performance?

Suffix95 selects the smallest output-nearest layer subset that reproduces 95% of the accumulated functional change measured during training.

However, it remains unclear whether this coverage metric reliably predicts downstream performance, particularly when the retained layers are subsequently optimized under a different training configuration.

Could the authors provide a quantitative analysis of the relationship between functional coverage and downstream performance across different layer subsets, datasets, and model architectures?

Additionally, what motivates the selection of 95% coverage, and under what conditions might high functional coverage fail to preserve task performance?

Q3. Are output-nearest layers inherently more adaptable, or are they primarily more computationally efficient?

The theoretical analysis suggests that layer adaptability depends on functional signal, spectral learning strength, and interactions among layers. These factors are task-dependent and do not necessarily imply that later layers are always more adaptable.

However, the proposed fine-tuning strategy exclusively considers output-nearest layer subsets, partly because they reduce backward computation and activation storage.

Could the authors compare output-nearest layer selection with non-contiguous or functionally selected layer subsets under matched computational budgets?

Such an analysis would help clarify whether the effectiveness of Suffix95 arises from the intrinsic adaptability of later layers or primarily from the computational advantages of restricting training to output-nearest layers.

4 comments
AuthorsOpenReview ↗

Thank you for your thoughtful review and questions on novelty and coverage (1/4)

Dear Reviewer ArGX,

Thank you for the thoughtful review. We are glad you found the theoretical perspective useful and the compensation analysis “a meaningful explanation for why selective fine-tuning can preserve model performance”. We answer W1 to W3 and Q1 to Q3 in your numbering.

W1: “Limited methodological novelty of the proposed fine-tuning strategy. [...] it remains unclear whether Suffix95 introduces a fundamentally new optimization mechanism or primarily offers a theoretically motivated layer-selection heuristic. In particular, the distinction between the proposed approach and existing selective fine-tuning methods should be established more explicitly through direct methodological comparisons.”

Response: Suffix leaves the optimizer unchanged; what is new is how it decides which layers to train, and that decision is derived rather than heuristic.

Derived rule. Cor. 3.4 shows that the shortest output-side set reaching a coverage target is the cheapest set whose bound on the extra loss stays within that target, and Prop. 3.3(c) shows that training the kept set reaches the floor that coverage sets. Suffix keeps exactly this set, chosen once per task, early, from one run.

How it differs. Table 1 of the revision contrasts the methods by what they use to decide:

  1. Last-k freezing (Lee et al., 2019) fixes the depth before training.
  2. FreezeOut, AutoFreeze, and Egeria freeze layers gradually, by a schedule, a gradient-norm change, or a reference model.
  3. AFLoRA freezes LoRA matrices gradually by a gradient-based score, and LISA samples layers at random every few steps.
  4. Suffix measures coverage and picks one depth per task, once and early, and can measure it with any target the user cares about.

Direct comparison. On Qwen3-8B with six tasks and three seeds, Suffix is more accurate than every fixed rule and AFLoRA and as accurate as AutoFreeze at five times its saving (Q1 has the full table).

W2: “Insufficient validation of the relationship between functional coverage and downstream performance. [...] reproducing 95% of the measured functional change does not guarantee preservation of 95% of the task-relevant information or predictive performance. Moreover, the functional contributions of layers may change after earlier layers are frozen [...]”

Response: The revision tests this directly, and coverage predicts downstream accuracy best among the measures we compare.

Coverage and accuracy. Over 72 Qwen3-8B runs on six tasks, three seeds, and four kept sets:

How well each measure predicts the cost of freezing (rank correlation, 72 runs)

MeasureExtra loss from freezingAccuracy lost against Full
Coverage0.680.77
Number of frozen layers0.530.40
Gradient at the switch0.260.12

Contributions after freezing. Your point that contributions change after freezing is exactly compensation, now made precise by Prop. 3.3. We measured it directly by computing the coverage of Full's own updates after the switch: Suffix's kept set covers 99 to 100% of them, so the estimate taken before freezing is conservative, and its depth follows each task's true need (App. A.7).

Choosing what to preserve. Your concern also leads to an extension. Coverage can be measured with any per-example loss, and Prop. 3.3 holds for any such loss, so the user can preserve the metric they care about while training stays unchanged (App. B.3). This matters when a fixed answer template dominates the training loss:

Suffix with the training loss and with accuracy as the target (accuracy in %, kept layers in parentheses, Qwen3-8B, seed 110)

TaskFullFixed-18Suffix, training lossSuffix, accuracy
MRPC87.786.579.4 (9)85.8 (21)
CoLA86.484.877.6 (12)85.0 (21)

Knowing when to stop. At the switch, Suffix also reads how much of the chosen target is left, and it stops when almost none is (App. B.3):

The stop check at the switch (accuracy in %, Qwen3-8B, seed 110)

TaskTarget leftDecisionAccuracy if stoppedFullCompute saved
IMDB0.033Stop95.896.486%
MRPC0.158Continue68.487.7
CoLA0.234Continue68.686.4

In the worst case coverage calls for every layer, and Suffix reduces to full-depth LoRA.

References

AuthorsOpenReview ↗

Thank you for encouraging broader validation of the mechanisms (2/4)

W3: “Limited evidence supporting the generality of the proposed layer-adaptability principles. [...] the detailed analyses of functional specialization and compensation after freezing are not demonstrated to the same extent on these larger models. [...] This raises the question of whether output-nearest layers are consistently the most appropriate choice for preserving functional adaptability, or whether their selection is primarily motivated by computational efficiency.”

Response: Compensation is now tested on Qwen3-8B, the tension across depth on three models, and the revision separates the reasons for output-side layers.

Tension across depth. On Qwen3-8B, potential falls with depth and activation energy rises on MNLI and IMDB (Fig. 2; rank correlations −0.95 to −0.97 and 0.98 to 0.99), and at the switch the gradient ranks the cost of freezing the wrong way (−0.61 over 36 runs). The same opposite trends hold on Qwen3-1.7B and LLaMA2-7B (new Fig. A.1, App. A.5).

Compensation on Qwen3-8B. Over four tasks and three seeds (Fig. 3a, App. A.6):

  1. Retraining lowers the extra loss for every kept set.
  2. Suffix's kept layers win back 52% of their extra loss, against 36% for Fixed-18, and after retraining Suffix's extra loss is 0.34% against 1.52%.
  3. Adapting the kept layers adds 0.98 test-accuracy points over the same frozen layers without adaptation (9 of 12 pairs, p = 0.011).

Compensation at scale. In the broad validation, Suffix freezes the input-side layers, which hold the most potential, up to 24 of 36 on Qwen3-8B, yet performance stays near full depth across GPT-2, Qwen3-8B, and LLaMA2-7B (Sec. 7, Table 4).

Output-side layers. They are chosen for both reasons, and the revision says which is which (Sec. 4.1): cost decides the shape, an output-side set, because only such a set shortens the backward pass, and coverage decides its length for each task (see Q3).

AuthorsOpenReview ↗

Thank you for requesting direct comparisons at matched budgets (3/4)

Q1: “What fundamentally distinguishes Suffix95 from existing selective fine-tuning and progressive layer-freezing approaches? [...] comparisons with simpler strategies, such as fixed output-nearest layer selection, progressive freezing, or gradient-based layer selection under matched computational budgets, would help establish whether the proposed functional analysis leads to meaningful practical improvements.”

Response: The distinction is the derived, per-task depth decision (W1). The revision compares it head to head with each strategy you list:

Head-to-head comparison on Qwen3-8B (accuracy relative to Suffix in points, six tasks, three seeds)

MethodStrategyAccuracy vs. SuffixFLOPs savedPairs Suffix wins
Last-9fixed output-nearest−6.2546.7%18 of 18
Last-18fixed output-nearest−3.3031.1%16 of 18
Fixed-18fixed, after the same switch−2.4430.1%17 of 18
AutoFreezeprogressive freezing+0.013.4%6 of 18
AFLoRAgradient-based selection−2.900%15 of 18
Suffixcoverage, per task017.3%

Matched budgets. A fixed depth matched to Suffix's cost on each task needs hindsight: Suffix's own choices, or a sweep of depths per task that costs several full runs. Suffix gives what such a sweep would find from one run, so we compare it with the fixed-depth accuracy–compute curve instead (Sec. 6, App. B.2):

Fixed depths against Suffix, relative to Full (accuracy change in points; FLOPs saved)

MethodAccuracy changeFLOPs saved
Last-9 (last quarter, from the start)−6.2746.7%
Last-18 (last half, from the start)−3.3231.1%
Fixed-18, switch at step 3−4.2131.1%
Fixed-18, switch at step 12−3.1930.1%
Fixed-18, switch at step 48−1.8426.2%
Fixed-18, switch at step 120−0.8518.2%
Suffix, automatic−0.1716.9%

The Fixed-18 rows and Suffix's row come from the switch-timing runs (four tasks, two seeds), and Last-9 and Last-18 from Table 3 (six tasks, three seeds). Suffix lies above the fixed-depth curve: at nearly the same saving as Fixed-18 switched at step 120, it loses 0.17 points where Fixed-18 loses 0.85. Spending more compute on a fixed depth closes only part of the gap, because one depth is too shallow for the demanding tasks and more than the easy ones need.

A rule of thumb, found automatically. The depths Suffix chooses also give the rule a fixed-depth user would want. On twelve Qwen3-8B tasks, six need at most half of the 36 layers (sentiment, topic, HellaSwag, SVAMP: 12 to 18) and six need more (GSM8K, BoolQA, WiC, MathQA, RTE, MNLI: 20 to 36), and every LLaMA2-7B task needs more than half (19 to 32 of 32). So the last quarter is too shallow everywhere, and half suffices for sentiment and topic tasks but not for entailment and question answering, as the controlled runs confirm: Fixed-18 ties Suffix on AG News and IMDB and trails it by 2.65 to 4.54 points on RTE, WiC, BoolQA, and CSQA (new App. B.6 paragraph “A rule of thumb”).

AuthorsOpenReview ↗

Thank you for your questions on the coverage target and output-side layers (4/4)

Q2: “How strongly does functional coverage correlate with downstream task performance? [...] Additionally, what motivates the selection of 95% coverage, and under what conditions might high functional coverage fail to preserve task performance?”

Response: Correlation. Coverage ranks the accuracy lost against Full at 0.77, against 0.40 for layer count and 0.12 for gradient size, over 72 runs (W2).

Why 95%. On the 27 Qwen3 and LLaMA2 combinations, 95% beats 90% in 19 of the 23 where the two differ (p = 0.003), and lower targets trade accuracy for saving (Table 5):

Coverage targets against all-layer LoRA (20 Qwen3 and LLaMA2 combinations; accuracy change relative to LoRA)

TargetFLOPs savedMedian accuracy changeRuns losing more than 3 points
95%22.3%+0.54%5%
90%27.4%−0.31%20%
80%36.8%−1.27%25%

When coverage of the training loss and accuracy diverge. In the controlled runs they agree, with a rank correlation of 0.80 between the extra training loss and the accuracy lost. They can diverge when a fixed answer template dominates the training loss, and there coverage can be measured with accuracy instead, which recovers about 7 points on MRPC and CoLA (W2, App. B.3). If coverage calls for every layer, Suffix falls back to full-depth LoRA.

Q3: “Are output-nearest layers inherently more adaptable, or are they primarily more computationally efficient? [...] Could the authors compare output-nearest layer selection with non-contiguous or functionally selected layer subsets under matched computational budgets?”

Response: No layer is inherently more adaptable: the first mechanism shows that what a layer contributes is set by the training dynamics, and coverage measures that contribution directly, including compensation. Output-side layers are then chosen because, at any given cost, they give the most coverage.

The first mechanism: adaptability is dynamic. A layer's gradient is its potential, what it could fix, times its activation energy, how fast it learns, and the two run against each other with depth (Sec. 3.1). Input-side layers have the most potential, output-side layers learn fastest, and which layers do the work changes during training (Table 2). What a layer actually delivers in a run is its potential realized at its own speed. On 29 Qwen3-8B tasks, the change each layer actually makes ranks layers almost opposite to potential (median rank correlation −0.62; see our answer to 6tPJ Q4). So neither potential nor speed alone says which layers are most adaptable, and asking whether later layers are inherently better has no fixed answer.

Coverage measures the contribution directly. Coverage is built from the change each layer actually makes in the run, and it asks what the kept layers can reproduce together, so it includes both realization and compensation (Prop. 3.3). It predicts the accuracy lost at 0.77 over 72 runs, against 0.40 for layer count and 0.12 for gradient size (Fig. 3b). One by one, output-side layers are not the strongest: on RTE, six scattered layers reproduce 80% of Full's later change against 44% for the last six (App. A.7 data, 256 examples).

Why output-side sets, at matched cost. A backward pass must reach the earliest trained layer, so a set's cost is set by its earliest layer: keeping four layers at the output instead of the input cuts training time by 30 to 34% and peak memory by 56% (Sec. 4.1, Fig. 4a). Among all sets with the same earliest layer, the output-side suffix contains every other set and so has the most coverage (Cor. 3.4). On the same RTE data, at the cost of the last 18 layers, the suffix covers 99% of Full's change against a median of 84% for scattered sets of nine layers reaching the same depth. So cost decides the shape, an output-side set, and coverage decides its length for each task.

Measured comparisons. The trained baselines confirm this:

  1. AFLoRA selects scattered LoRA matrices by a gradient score. It saves no FLOPs, because the backward pass still reaches the first layer, and it is 2.90 points less accurate, because the score favors fast output-side matrices over slower ones that still had work to do (Sec. 6).
  2. LISA samples non-contiguous layers at random. Against it, Suffix's median accuracy is 2.32% higher on Qwen3-8B (Table 4), and Suffix saves 20.6% of FLOPs over seven runs at the 95% target (Table 5).
  3. AutoFreeze freezes a prefix only after gradients settle, so it keeps accuracy but saves 3.4% against Suffix's 17.3%.
Reviewer_6TeYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies why some Transformer layers are more useful to fine-tune than others. The authors analyse how much each layer can improve the model's predictions, and how its contribution changes when other layers are trained or frozen. Based on this analysis, they propose a method which first trains layers throughout the model, then freezes earlier layers and continues training only the final layers selected according to their measured contributions. Experiments show that the method achieves comparable performance to standard fine-tuning while reducing training compute and memory usage.

Critical strengths
  • Section 3.1 provides a theoretical analysis (or characterisation) of why gradient magnitude alone may not reflect how much a layer can improve model predictions. The distinction between the corrections a layer can make and how easily they can be learned is interesting and relevant to understanding fine-tuning behaviours.
Critical weaknesses
  • I believe the presentation of this paper needs significant improvement. Section 3.1 introduces several new concepts, primarily through mathematical definitions without sufficiently clear explanations. Section 3.2 then introduces more undefined jargons, making the overall argument difficult to follow. The paper also contains substantial mathematical analysis but the main ideas and their practical implications are not presented clearly.
  • Section 3.1 studies the corrections that a layer could theoretically produce, whereas Section 4.4 selects layers based on the measured effects of updates that have already occurred. It is unclear why the latter is a reliable measure of layer adaptability. Furthermore, Section 4.5 does not compare Suffix95 against simple baselines such as retaining a fixed number of final layers after the same training steps. Therefore it is very difficult to evaluate the reported improvements.
Questions for authors
  • In Sections 3.1 and 4.4, what is the relationship between the theoretically defined functional signal and the empirical functional coverage used by Suffix95?
  • Section 4.4 treats the switching step as a predefined hyperparameter. How sensitive are the results to this choice?
6 comments
AuthorsOpenReview ↗

Thank you for helping us improve the presentation (1/4)

Dear Reviewer 6TeY,

Thank you for the review. We are glad you found the distinction between the corrections a layer can make and how easily they are learned “interesting and relevant to understanding fine-tuning behaviours”. Your weaknesses and questions were not numbered, so we number them W1 to W3 and Q1 to Q2 in order.

W1: “I believe the presentation of this paper needs significant improvement. Section 3.1 introduces several new concepts, primarily through mathematical definitions without sufficiently clear explanations. Section 3.2 then introduces more undefined jargons, making the overall argument difficult to follow. The paper also contains substantial mathematical analysis but the main ideas and their practical implications are not presented clearly.”

Response: Thank you for this frank assessment; it led to the largest rewrite.

Section 3. It now states two mechanisms, each with its formula, a plain intuition, and a direct test. Sec. 3.1: a layer's gradient is what it could fix times how fast it learns, and the two run in opposite directions with depth, so the gradient measures speed, not reach. The test: on Qwen3-8B, potential falls with depth (rank correlation −0.95 to −0.97) while the gradient barely follows it (Fig. 2). Sec. 3.2: kept layers take over frozen work, so the cost of freezing is set by coverage. The test: retraining lowers the extra loss for every kept set, and coverage predicts that loss better than every alternative (Fig. 3).

Terms and structure. Every term is defined at first use, results the method does not use are removed, and a notation table is added (Table A.1). The same three-step message, tension, compensation, method, runs through the abstract, introduction, section openers, and conclusion. The four terms the argument rests on now read as follows:

TermPlain meaning in the revision
Potential (was functional signal)How much of the needed correction one layer could make on its own
Activation energy (was learning strength)How fast a layer's weight change moves the model's outputs
CompensationKept layers taking over work that frozen layers would have done
CoverageThe share of the change made by all layers that the kept layers can reproduce together

Main ideas and their practical implications. The revision states each idea with what it means for a practitioner:

  1. The gradient measures speed, not reach (Sec. 3.1). A layer's squared gradient norm is its potential times its activation energy, and the two run in opposite directions with depth. Implication: do not pick layers by gradient size. At the switch on Qwen3-8B, gradient size ranks the cost of freezing the wrong way (−0.61 over 36 runs).
  2. Freezing costs only what the kept layers cannot cover (Sec. 3.2). The extra loss is bounded by β(1−C)Q\beta(1-C)Q (Prop. 3.3), and the cheapest set meeting a coverage target is the shortest output-side set reaching it (Cor. 3.4). Implication: keep the shortest output-side set that covers 95% of the change, which Suffix computes.
  3. The depth can be read once, early, per task (Sec. 4). Most change happens in the first sweeps, and a task's coverage curve settles early while different tasks settle at different depths. Implication: one short all-layer phase decides the depth, and an automatic rule decides when, with the same constants for every task (Alg. 1).

Together these give the result a practitioner cares about: Suffix stays at Full LoRA's accuracy (85.11 against 85.13, p = 0.93) while saving 17.3% of training FLOPs and 19.1% of memory, with no per-task tuning (Table 3). The depths it finds also give a rule of thumb: the last quarter of the layers is too shallow for every task we tested, and half suffices for sentiment and topic tasks but not for entailment and question answering (App. B.6).

A worked example. The revision illustrates the argument with two tasks (Sec. 3.2). With seed 110, Suffix freezes 22 of 36 layers on AG News, and freezing costs only 0.24% of Full's loss reduction. Fixed-18 freezes fewer layers, 18, on RTE, yet freezing costs 0.96%. Layer count does not explain this, but coverage does: Suffix's kept layers leave 2.4% of the change uncovered on AG News, while the last 18 layers leave 37.5% uncovered on RTE. So a set that freezes more can cost less, and the right depth must be read per task.

AuthorsOpenReview ↗

Thank you for questioning the reliability of measured coverage (2/4)

W2: “Section 3.1 studies the corrections that a layer could theoretically produce, whereas Section 4.4 selects layers based on the measured effects of updates that have already occurred. It is unclear why the latter is a reliable measure of layer adaptability.”

Response: The method rests on Sec. 3.2. After freezing, the cost is set by what the kept set can cover, not by each layer's potential (Prop. 3.3), and the cheapest set meeting the bound is the shortest output-side set with enough coverage (Cor. 3.4). Suffix measures exactly this coverage from realized change, and three tests show the measurement is reliable.

Why realized change, not what a layer could do. What a layer could correct in theory is not what it corrects in a finite run, because that also depends on how fast it learns. On GPT-2 Large the first four layers have the most potential and make the largest weight change, 41.4% of the total by step 500, yet remove the least loss, 11.0%, while the last four remove 38.1% by step 45 (Table 2). The relation between gradient and potential also varies by model: the gradient hides potential on Qwen3-8B and Qwen3-1.7B but tracks it on LLaMA2-7B, where activation energy varies little (Fig. 2, new Fig. A.1). Realized change, the effect of the updates already made, includes both potential and speed, so it is the quantity that tells which layers can take over the work in the run at hand.

It predicts the cost. Over 72 Qwen3-8B runs on six tasks (Fig. 3b):

How well each measure predicts the cost of freezing (rank correlation, 72 runs)

MeasureExtra loss from freezingAccuracy lost against Full
Coverage0.680.77
Number of frozen layers0.530.40
Gradient share at the switch0.260.12

It is conservative. Against the true coverage computed from Full's own later updates, Suffix's kept set covers 99 to 100% of them (App. A.7, Table A.4):

TaskLayers truly neededLayers Suffix keepsTrue coverage of Suffix's set
AG News714100%
BoolQA172499.2%
RTE1830100%
WiC213099.7%

The margin is by design: early change is spread over many training examples, while after the switch it sits on 1.7 to 3 times fewer, which fewer layers can reproduce.

It follows each task's need. A task's coverage curve settles within the first sweeps and differs about eight times more across tasks than within a task (Sec. 4.3), and Suffix's depth rises across tasks in the same order as the true need (App. A.7).

AuthorsOpenReview ↗

Thank you for suggesting the fixed-layer comparison (3/4)

W3: “Furthermore, Section 4.5 does not compare Suffix95 against simple baselines such as retaining a fixed number of final layers after the same training steps. Therefore it is very difficult to evaluate the reported improvements.”

Response: The revision adds exactly this baseline, Fixed-18, which keeps the last 18 layers after the same all-layer start and switch as Suffix. Suffix beats it by 2.44 points, on 17 of 18 task–seed pairs (sign test p = 0.0001; Table 3). Because only the depth differs, this gain is what measuring the depth adds. The revision also adds Last-9, Last-18, AutoFreeze, and AFLoRA (general response).

Suffix against the fixed-layer baseline (accuracy in %, Qwen3-8B, three seeds, six tasks)

MethodAccuracyFLOPs saved
Full LoRA85.130%
Fixed-1882.6730.1%
Suffix85.1117.3%

Where measuring the depth helps. The gain appears exactly where the last 18 layers leave much of the change uncovered: 16 to 33% on RTE, WiC, BoolQA, and CommonsenseQA, against 9% on AG News and 4% on IMDB (Fig. 5b). On those four tasks Suffix keeps more than 18 layers and wins by 2.65 to 4.54 points; on AG News and IMDB it keeps fewer and ties while saving more:

Suffix's gain over Fixed-18 by task (accuracy in points, mean over three seeds)

RTEWiCBoolQACommonsenseQAAG NewsIMDB
Gain of Suffix+2.65+4.54+4.09+3.06+0.24+0.06

All baselines. Against every method in the controlled comparison, Suffix is the only one that stays at Full's accuracy while saving substantial compute:

MethodAccuracyFLOPs savedPairs Suffix wins (of 18)
Full LoRA85.130%6
AutoFreeze85.123.4%6
AFLoRA82.210%15
Last-978.8646.7%18
Last-1881.8131.1%16
Fixed-1882.6730.1%17
Suffix85.1117.3%

Same cost. Suffix keeps more layers than Fixed-18 on the demanding tasks, so one may ask how much of its gain comes from spending more compute. A fixed depth matched to Suffix's cost on each task needs hindsight, Suffix's own choices or a sweep of depths per task, so we compare with the fixed-depth accuracy–compute curve instead (Sec. 6, App. B.2):

Fixed-18 with the switch at stepAccuracy change vs FullFLOPs saved
3−4.2131.1%
12−3.1930.1%
48−1.8426.2%
120−0.8518.2%
Suffix, automatic−0.1716.9%

At nearly the same saving as Fixed-18 switched at step 120, Suffix loses 0.17 points where Fixed-18 loses 0.85 (four tasks, two seeds). Spending more compute on one fixed depth closes only part of the gap, because one depth is too shallow for the demanding tasks and more than the easy ones need.

Reviewer_6tPJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies which Transformer layers are most useful for fine-tuning, evaluating each layer by the loss-reducing change it can make to its output. It separates the functional signal a layer can express from its spectral learning strength: the signal decreases with depth while the strength increases, leaving gradient norms nearly flat and potentially misleading for layer ranking (Prop. 3.1, Fig. 2). It also shows that layers specialize during joint training and that, after early layers are frozen, the output-side layers recover part of the lost correction (10.6–21.6% on GPT-2 Large/E2E, Prop. 3.2). Empirically, training layers near the output reduces training time by 30–34% and memory use by about 56%. Most loss-relevant changes occur early in training, and the amount captured by each output-side layer set is stable within a dataset but varies across datasets. Based on these results, the paper proposes Suffix95: it trains all layers until tswitcht_{switch}, then retains the shortest output-side suffix that captures 95% of the measured change. Across 39 model–dataset pairs (GPT-2 M/L, Qwen3-8B, and LLaMA2-7B), its median performance is within −0.77% to +2.32% of full-depth LISA/LoRA, with 8.8–25.8% lower training compute and 11.5–19.1% lower memory use after the switch.

Critical strengths

S1. A clear explanation of why gradient norms can rank layers poorly. The paper decomposes the squared gradient norm into functional signal and learning strength (D=2σˉ2SD = 2\bar\sigma^2 S). This analysis is simple and well supported. Fig. 2 shows that the two terms change in opposite directions with depth, resulting in a nearly flat gradient profile (Kendall τ≤0.02\tau \le 0.02 on MNLI). Table 1A shows that early layers have the largest parameter changes but produce the smallest loss reduction. This finding is a useful caution for gradient-based layer selection.

S2. A practical, data-dependent recipe with transparent cost accounting. Suffix95 selects the retained depth for each dataset based on a single run, and all reported costs include profiling. The selected depth varies widely, from 12 layers on IMDB to all 36 layers on MNLI. When using fewer layers does not help, the method falls back to full depth, which is a sensible default.

Critical weaknesses

W1, missing simple baselines: there is no comparison with training a fixed number of last layers, with gradient-norm layer ranking, or with a “keep the last half” rule. So it’s unclear what the profiler adds.

W2, noisy results: every result is a single run. On HellaSwag, freezing layers beats full depth (92.5 vs. 89.5), and the full-depth Yelp score is low at 57.9. That suggests the full-depth baselines did not converge, which inflates the mean gains. W3, theory and method are loosely tied: the mechanism tests use only GPT-2 Large on one dataset. The profiler uses an empirical proxy, not the signal defined in the theory. Several findings are already known, such as “late layers are cheaper to train” and “most change happens early.”

Questions for authors
  1. Can you report results across multiple seeds and explain the large HellaSwag and Yelp gaps, where freezing layers outperforms full-depth training?
  2. How sensitive are the results to tswitcht_{switch}? How can users choose it without running a sweep, given that the reported values range from 6 to 264 steps?
  3. Does the opposite depth trend for signal and strength (Fig. 2) also appear with other models and tasks, and when using Adam instead of gradient flow?
  4. The profiler uses u^ℓ,k\hat u_{\ell,k} rather than SDS_{\mathcal{D}}. How closely do the rankings from these two measures agree in practice?

Minor:

  • Clarify the distinction between the mechanistic study and the method, or strengthen the connection between them.
  • The notation is dense, and Fig. 1 is difficult to read at its current size.
4 comments
AuthorsOpenReview ↗

Thank you for your careful review and baseline suggestions (1/4)

Dear Reviewer 6tPJ,

Thank you for the careful review. We are glad you found the gradient decomposition “simple and well supported” and the recipe “a practical, data-dependent recipe with transparent cost accounting”. Your questions and minor points were not numbered, so we number them Q1 to Q4 and M1 to M2 in order.

W1: “missing simple baselines: there is no comparison with training a fixed number of last layers, with gradient-norm layer ranking, or with a ‘keep the last half’ rule. So it's unclear what the profiler adds.”

Response: The revision adds the fixed-depth baselines you named and two gradient-based methods, in a controlled comparison on Qwen3-8B with six tasks, three seeds, and one shared recipe (Sec. 6, Table 3), and it tests gradient-norm ranking directly.

  1. Fixed number of last layers: Last-9 trains the last quarter from the start, the common rule of Lee et al. (2019).
  2. Keep the last half: Last-18 trains the last half from the start, and Fixed-18 trains the last half after the same all-layer start and switch as Suffix.
  3. Gradient-based selection: AFLoRA freezes LoRA matrices by a gradient-based score, and AutoFreeze freezes once the gradient-norm change settles.
  4. Gradient-norm ranking: at the switch, the share of the gradient norm on the frozen layers ranks the accuracy lost at only 0.12 over 72 runs, against 0.77 for coverage, and among runs that freeze the same 18 layers it points the wrong way (−0.61; Sec. 3.1, App. A.7).

Suffix against each baseline on Qwen3-8B (accuracy in points, 18 task–seed pairs)

BaselineAccuracy gain of SuffixPairs Suffix winsFLOPs saved by the baseline
Last-9+6.2518 of 1846.7%
Last-18+3.3016 of 1831.1%
Fixed-18+2.4417 of 1830.1%
AFLoRA+2.9015 of 180%
AutoFreeze−0.016 of 183.4%

What the profiler adds. Fixed-18 differs from Suffix only in how the depth is chosen, so the gap between them is exactly what the profiler adds: 2.44 points (sign test p = 0.0001). The all-layer start alone adds 0.86 points (Last-18 to Fixed-18). The gain appears exactly where the last 18 layers leave much of the needed change uncovered: 16 to 33% on RTE, WiC, BoolQA, and CommonsenseQA, against 9% on AG News and 4% on IMDB (Fig. 5b).

Suffix's gain over Fixed-18 by task (accuracy in points, mean over three seeds)

RTEWiCBoolQACommonsenseQAAG NewsIMDB
Gain of Suffix+2.65+4.54+4.09+3.06+0.24+0.06

AutoFreeze matches Suffix's accuracy but saves one fifth of the compute (3.4% against 17.3%).

W2: “noisy results: every result is a single run. On HellaSwag, freezing layers beats full depth (92.5 vs. 89.5), and the full-depth Yelp score is low at 57.9. That suggests the full-depth baselines did not converge, which inflates the mean gains.”

Response: Thank you for pointing to this pattern. It is what compensation predicts, and the revision tests it with three seeds, paired tests, and one shared recipe for all methods (Table 3).

Why partial tuning lands close to Full. By Prop. 3.3, once the kept layers cover the needed change, little is left for freezing to lose, so partial and full tuning should land close together, and training every layer adds run-to-run variation. Suffix's kept set covers 99 to 100% of Full's own later updates (App. A.7), and the seeded results show both effects:

Full and Suffix on Qwen3-8B (accuracy in %, three seeds, six tasks)

MeasureFullSuffix
Mean accuracy85.1385.11
Seed-to-seed spread (s.e.)0.460.12
Task–seed pairs where it scores higher126

The gap is not significant (p = 0.93), and training all layers varies almost four times as much across seeds. A single-seed gap such as HellaSwag (92.5 against 89.5) is therefore the expected result of near-complete coverage plus this variation.

Full is well trained. A lower learning rate does not help it, and its training loss matches Suffix's: Suffix ends lower in only 17 of the 37 Qwen3 and LLaMA2 broad runs (App. B.2).

Full LoRA with two learning-rate recipes (accuracy in %, Qwen3-8B, seed 110)

RecipeBoolQARTEWiC
Constant learning rate 2×10−42\times10^{-4}87.192.173.8
Shared recipe (10−310^{-3} with $\times$0.2 step-down)87.691.774.5

Yelp. Yelp is a hard 5-class task on which every method scores between 57.0 and 58.3, so its score reflects the task rather than the baseline.

References

AuthorsOpenReview ↗

Thank you for encouraging a tighter connection between theory and method (2/4)

W3: “theory and method are loosely tied: the mechanism tests use only GPT-2 Large on one dataset. The profiler uses an empirical proxy, not the signal defined in the theory. Several findings are already known, such as ‘late layers are cheaper to train’ and ‘most change happens early.’”

Response: The revision ties each step of Suffix to one result and one test (Sec. 5, App. A.8, Table A.5).

Mechanisms to rule. Prop. 3.3 bounds the extra loss from freezing by β(1−C)Q\beta(1-C)Q, where CC is the coverage of the kept set. Cor. 3.4 shows that the cheapest set meeting a coverage target is the shortest output-side set reaching it, which is exactly the set Suffix keeps. Prop. 3.3(c) adds that gradient descent on the kept set reaches the floor that coverage sets at a linear rate, so training finds the compensation by itself.

Rule to measurement. Suffix's overlap matrix is the matrix GG of Prop. 3.3, built from the changes made so far (Sec. 5), so the profiler measures the quantity that sets the cost of freezing (see Q4).

Measurement to outcome. Over 72 Qwen3-8B runs on six tasks, coverage at the switch predicts the cost of freezing better than every alternative, including the gradient measured after training has finished:

How well each measure predicts the cost of freezing (rank correlation, 72 runs)

MeasureExtra loss from freezingAccuracy lost against Full
Coverage, at the switch0.680.77
Number of frozen layers0.530.40
Gradient share, at the switch0.260.12
Gradient share, after training0.630.58

Beyond GPT-2. Compensation is now measured on Qwen3-8B (Fig. 3), and the tension across depth on Qwen3-8B, Qwen3-1.7B, and LLaMA2-7B (Fig. 2, new Fig. A.1; see Q3).

Supporting findings. The two findings you name serve as supporting measurements (Sec. 4): keeping four output-side layers instead of four input-side ones cuts training time by 30 to 34% and peak memory by 56%, and new change falls by about three quarters between sweeps 2 to 5 and sweeps 13 to 16. The new results they support are the tension across depth and compensation, which explain why coverage, and not gradient size, sets the cost of freezing.

Q1: “Can you report results across multiple seeds and explain the large HellaSwag and Yelp gaps, where freezing layers outperforms full-depth training?”

Response: Yes. Every controlled result now has three seeds (Table 3). Suffix matches Full within one standard error on four of the six tasks, is above it on CommonsenseQA, and below it on BoolQA:

Full and Suffix by task (accuracy in %, mean ± s.e. over three seeds)

TaskFullSuffix
RTE92.18 ± 0.4892.18 ± 0.43
WiC73.51 ± 0.5973.51 ± 0.31
BoolQA87.47 ± 0.1086.21 ± 0.30
AG News88.89 ± 2.4789.49 ± 1.74
CommonsenseQA85.53 ± 0.5986.13 ± 0.27
IMDB83.20 ± 0.2083.13 ± 0.07

The HellaSwag and Yelp gaps come from single-seed broad runs, and W2 explains them: Suffix's kept layers cover 99 to 100% of the needed change, so partial and full tuning land close, while training every layer adds almost four times the seed-to-seed variation.

AuthorsOpenReview ↗

Thank you for raising switching sensitivity and generality (3/4)

Q2: “How sensitive are the results to tswitcht_{\mathrm{switch}}? How can users choose it without running a sweep, given that the reported values range from 6 to 264 steps?”

Response: Users do not choose it. An automatic rule sets the switch, and the revision gives it in full (Sec. 5, Alg. 1): Suffix switches once new change falls below 20% of its peak and the chosen depth has held for three sweeps. The constants are the same for every task and model, and the switch fired between steps 9 and 21 in all 18 controlled runs. The 6 to 264 range in the broad study comes from runs that train a few layers at a time, where one sweep spans several groups of steps.

Sensitivity. Forcing the switch to step 6, 24, or 36 on seed 110 changes accuracy by at most 1.5 points and depth by at most three layers (App. B.2):

Suffix with the switch forced to step 6, 24, or 36 (accuracy in %, seed 110)

TaskAccuracy rangeLayers kept
RTE90.6 to 92.128 to 30
WiC73.0 to 73.830
BoolQA86.3 to 86.624 to 25
AG News90.9 to 91.414 to 17

A fixed depth catches up only by switching late. Fixed-18 needs a switch at step 120 to come close to Full, and by then it saves little:

Fixed-18 with a later switch, against Full (accuracy change in points, four tasks, two seeds)

Switch at step31248120Suffix, automatic
Accuracy change−4.21−3.19−1.84−0.85−0.17
FLOPs saved31.1%30.1%26.2%18.2%16.9%

Q3: “Does the opposite depth trend for signal and strength (Fig. 2) also appear with other models and tasks, and when using Adam instead of gradient flow?”

Response: Yes, on three models, two tasks, and in training with AdamW.

Other models and tasks. On Qwen3-8B with MNLI and IMDB, three batches each, potential falls with depth in every batch (rank correlation −0.95 to −0.97) while activation energy rises (0.98 to 0.99), so the gradient barely follows depth (−0.17 to 0.24; Fig. 2). We repeated the measurement on Qwen3-1.7B and LLaMA2-7B (new Fig. A.1, App. A.5). On both, potential again falls with depth in all six batches (−0.99 and −0.96 to −0.97) and activation energy rises (0.98 to 0.99, and 0.74 to 0.80), with the potential ranking stable across batches (Kendall τ≥0.96\tau \ge 0.96 and ≥0.94\ge 0.94). How much the gradient hides potential depends on how widely activation energy varies: by four orders of magnitude on Qwen3-1.7B, where the gradient orders 86 to 116 of 378 layer pairs opposite to potential, and about thirtyfold on LLaMA2-7B, where it orders 38 to 54 of 496 pairs opposite.

Adam instead of gradient flow. Potential and activation energy are measured at a model state, here the start of training, so they do not depend on the optimizer. The results hold for any smooth loss, without gradient flow or a linear model, and the Polyak–Łojasiewicz inequality enters only the rates (Sec. 3). Our training runs use AdamW, and the behavior the tension predicts appears in them. On GPT-2 Large the last four layers remove 38.1% of the loss at step 45 but 31.4% by step 500, as earlier layers catch up. At the switch on Qwen3-8B, among the 36 runs that freeze the same 18 layers, freezing layers that hold more of the gradient costs less, not more (rank correlation −0.61), while coverage keeps the right sign (0.29; Sec. 3.1, App. A.7).

AuthorsOpenReview ↗

Thank you for your questions on the profiler and presentation (4/4)

Q4: “The profiler uses u^ℓ,k\hat{u}_{\ell,k} rather than SDS_{\mathcal D}. How closely do the rankings from these two measures agree in practice?”

Response: Per layer, the two rank layers almost oppositely, which is what the analysis predicts: neither potential nor activation energy alone decides what a layer contributes; the training dynamics do. We computed both from the profiler logs of 29 Qwen3-8B tasks, in the all-layer sweeps before the switch:

Rank correlation across layers (29 Qwen3-8B tasks, seed 110)

PairMedianRange
SDS_{\mathcal D} and u^ℓ,k\hat{u}_{\ell,k}−0.62−0.81 to −0.04
SDS_{\mathcal D} and depth−0.97−0.99 to −0.92
u^ℓ,k\hat{u}_{\ell,k} and depth+0.65+0.04 to +0.83

SDS_{\mathcal D}, potential in the revision, is what a layer could fix on its own, and it falls with depth on every task. Activation energy, how fast a layer learns, rises with depth (Sec. 3.1). u^ℓ,k\hat{u}_{\ell,k} is the change a layer actually made in the run, its potential realized at its own speed, so it follows neither factor alone and leans toward the fast output-side layers early in training. This is the tension of Sec. 3.1 measured on 29 tasks, and it is why the profiler measures realized change instead of either factor.

Why this is the right quantity. The cost of freezing depends on what the kept layers can reproduce together in the run at hand, including compensation, not on any per-layer ranking. Suffix builds coverage from u^ℓ,k\hat{u}_{\ell,k} through the overlap matrix, which is exactly the GG of Prop. 3.3. Coverage predicts the cost of freezing at 0.68, against 0.26 for gradient size (Fig. 3b), and it tracks the true need, which we computed directly from Full's adapters at the switch and at the end, with forward passes only (App. A.7, Table A.4):

Layers needed and kept at the 95% target (Qwen3-8B, seed 110; true coverage in %)

TaskLayers truly neededLayers Suffix keepsTrue coverage of Suffix's set
AG News714100
BoolQA172499.2
RTE1830100
WiC213099.7

Across tasks, Suffix's depth rises in the same order as the true need. The estimate is also conservative by design. Early change is spread over many training examples, while after the switch it sits on 1.7 to 3 times fewer, which fewer layers can reproduce, so a set that covers the early change also covers the later one.

M1: “Clarify the distinction between the mechanistic study and the method, or strengthen the connection between them.”

Response: The revision does both. Sec. 3 states the two mechanisms, Sec. 4 the supporting measurements, and Sec. 5 the method, where each step cites the result it uses (Prop. 3.2, Prop. 3.3, Cor. 3.4). App. A.8 collects every step, its result, and its test in one table.

M2: “The notation is dense, and Fig. 1 is difficult to read at its current size.”

Response: The notation is simplified and collected in Table A.1, and Fig. 1 is redrawn as a simpler diagram with readable labels.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

General response

We thank all three reviewers for careful and constructive reviews. Reviewers found the split of the gradient into what a layer can fix and how fast it learns “simple and well supported” (6tPJ) and “interesting and relevant to understanding fine-tuning behaviours” (6TeY), and the compensation analysis “a meaningful explanation for why selective fine-tuning can preserve model performance” (ArGX). The main requests were simple baselines and multiple seeds, a tighter link between the analysis and the method, the switch step, evidence that coverage predicts performance, and clearer presentation. The revision answers each with new experiments and rewritten sections.

What the revised paper shows:

  1. A tension across depth. A layer’s squared gradient norm is the product of its potential, how much of the needed correction it can reach, and its activation energy, how fast it learns. Potential falls and activation energy rises with depth, so the gradient measures speed, not reach, and cannot choose layers (Sec. 3.1).
  2. Why freezing is possible at all. The kept layers compensate for the frozen ones, so the extra loss from freezing is bounded by the share of the needed change that the kept set cannot cover, its coverage. The cheapest set meeting this bound is the shortest output-side set that reaches a coverage target (Prop. 3.3, Cor. 3.4). The main assumption is a smooth loss.
  3. A method that follows from both. Suffix trains all layers briefly, measures coverage, and switches automatically to that set, with no per-task tuning (Sec. 5, Alg. 1). Its depth follows each task’s need and is conservative by design. It can also choose the depth for any target the user cares about, and stop once that target is used up (App. B.3).

New experiments and analyses:

  1. A controlled comparison on Qwen3-8B with 7 methods, 6 tasks (RTE, WiC, BoolQA, AG News, CommonsenseQA, IMDB), and 3 seeds, giving 18 task–seed pairs with paired sign-flip tests (Sec. 6, Table 3, Fig. 5).
  2. Coverage against performance over 72 runs on six tasks: coverage predicts the extra loss and the accuracy lost better than layer count or gradient size (Fig. 3b, App. A.7).
  3. True coverage computed from Full’s own updates after the switch: Suffix’s kept set covers 99 to 100% of them (App. A.7).
  4. Compensation measured on Qwen3-8B (Fig. 3, App. A.6), and the tension across depth on three models, Qwen3-8B, Qwen3-1.7B, and LLaMA2-7B (Fig. 2, new Fig. A.1).
  5. A step-by-step bridge from each result to each step of Suffix, with a test for each (Sec. 5, App. A.8).
  6. Switch timing and learning-rate checks (App. B.2).
  7. A user-chosen target and a built-in stop: with accuracy as the target, Suffix recovers about 7 points on template-dominated tasks, and it stops at the switch when the target is used up (App. B.3).

Controlled comparison on Qwen3-8B (accuracy in %, mean ± s.e. over three seeds and six tasks; savings against Full LoRA)

MethodAccuracyFLOPs savedMemory savedSuffix wins–ties–losses
Full LoRA85.13 ± 0.460%0%6–0–12
AutoFreeze85.12 ± 0.333.4%2.9% ± 1.16–2–10
AFLoRA82.21 ± 0.030%0.3% ± 0.215–0–3
Last-9 (last quarter)78.86 ± 0.5446.7%39.9% ± 2.718–0–0
Last-18 (last half)81.81 ± 0.4531.1%33.5% ± 1.916–1–1
Fixed-18 (last half, same switch)82.67 ± 0.4130.1%35.6% ± 0.117–0–1
Suffix85.11 ± 0.1217.3%19.1% ± 2.5

Suffix is the only method that stays at Full’s accuracy (0.02 points below, p = 0.93) while saving substantial compute. It saves five times as much as AutoFreeze at the same accuracy and is more accurate than every fixed-depth rule on at least 16 of the 18 task–seed pairs.

Summary for the AC:

  1. Reviewer 6tPJ asked for simple baselines, seeds, and a tighter link between analysis and method. The fixed-depth and gradient-based baselines are added in a three-seed comparison, and gradient-norm ranking is tested directly, where Suffix beats each fixed rule on at least 16 of 18 pairs; Full is shown to be well trained; and each step of Suffix is tied to a result and a test.
  2. Reviewer 6TeY asked for clearer presentation, a reason to trust measured effects, and a fixed-layer baseline. Section 3 is rewritten with an intuition and a test for each mechanism, measured coverage predicts the cost of freezing best and is conservative against the true coverage, and Fixed-18 loses on 17 of 18 pairs.
  3. Reviewer ArGX asked about novelty, coverage versus performance, and generality. The depth rule follows from the mechanisms and beats fixed, progressive, and gradient-based methods head to head; coverage predicts the accuracy lost at 0.77 over 72 runs; compensation is tested on Qwen3-8B and the tension on three models; and the target can be the metric the user cares about.
AuthorsOpenReview ↗

A kind reminder before the discussion closes

Dear Reviewers,

Thank you again for your time and careful reviews. The discussion closes on Nov 18 (AoE), so we would like to kindly follow up on our responses. We would be grateful to hear whether they address your concerns.

In brief, the revision adds:

  • Baselines and seeds. A controlled comparison on Qwen3-8B with six tasks and three seeds adds fixed-depth, progressive and gradient-based baselines. Suffix stays at Full LoRA's accuracy (85.11 against 85.13) while saving 17.3% of training FLOPs and 19.1% of memory.
  • Analysis to method. Each step of Suffix now follows from one result and passes one test. Coverage predicts the accuracy lost at 0.77 over 72 runs, against 0.40 for layer count and 0.12 for gradient size.
  • Generality. Compensation is now measured on Qwen3-8B and the tension across depth on three models. The switch is automatic and needs no per-task tuning.
  • Presentation. Section 3 is rewritten in plain words, with an intuition and a direct test for each mechanism, and a notation table is added.

Please let us know if anything remains unclear, and we will be glad to clarify.

Sincerely,

Authors of Submission 36925

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.