- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a three-stage functional segmentation of LLM layers into conceptualization, reasoning, and textualization, with reasoning further divided into active and idle layers. It introduces LIFT, which selectively fine-tunes task-specific critical layers. Experiments on two LLMs and four reasoning benchmarks report competitive accuracy and reduced training time compared with LoRA-based baselines.
Critical strengths
1- The paper connects layer-wise representation analysis with selective fine-tuning, providing an intuitive framework for identifying task-specific critical layers. 2- LIFT achieves competitive or improved accuracy with shorter training times across several reasoning benchmarks, suggesting practical benefits from targeted layer adaptation.
Critical weaknesses
1- Cross-lingual cosine similarity and adjacent-layer similarity provide indirect evidence, but do not establish that the identified layers actually perform conceptualization, reasoning, or textualization (Sec 4). 2- The experiments cover only two 7–8B models and four benchmarks. Moreover, the reported training-time savings exclude the potentially substantial cost of functional segmentation and quantization-based bottleneck identification (Table 2).
Questions for authors
1- What evidence distinguishes the proposed functional stages from general patterns of representation similarity or layer redundancy? 2- What is the total computational overhead of segmentation and quantization sensitivity analysis, and how many fine-tuning runs are needed to amortize this cost? 3- Does LIFT outperform random or uniformly spaced layer selection under identical trainable-parameter budgets? 4- Why does LIFT underperform AdaLoRA and the original model on Qwen2 GSM8K (Table 2), despite identifying the active reasoning layers as critical?
4 comments
Response to Reviewer yijH (Part 2/2): Layer selection and scope
3. Matched-budget layer selection (Q3)
We compare LIFT with Random- under identical trainable-parameter budgets and include Last- as a contiguous-layer control. All methods use the same adapted-layer count, LoRA configuration, optimizer, training data and epochs.
Table Y1. Matched-budget layer selection on the eight original settings.
| Model | Task | LIFT | Last- | Random- | |
|---|---|---|---|---|---|
| Llama-3 | FOLIO | 18 | 58.21 ± 2.35 | 54.90 ± 1.33 | 55.79 ± 0.87 |
| Llama-3 | Logical | 20 | 53.58 ± 2.47 | 47.50 ± 1.04 | 52.58 ± 0.73 |
| Llama-3 | ProofWriter | 18 | 57.67 ± 0.36 | 48.17 ± 0.62 | 51.37 ± 2.77 |
| Llama-3 | GSM8K | 2 | 73.96 ± 0.98 | 56.60 ± 0.35 | 66.12 ± 1.27 |
| Qwen2 | FOLIO | 23 | 44.73 ± 1.98 | 42.28 ± 1.29 | 43.90 ± 0.74 |
| Qwen2 | Logical | 21 | 57.08 ± 3.29 | 59.00 ± 1.87 | 54.96 ± 0.73 |
| Qwen2 | ProofWriter | 23 | 53.46 ± 2.36 | 46.62 ± 0.28 | 47.67 ± 2.77 |
| Qwen2 | GSM8K | 23 | 75.78 ± 0.24 | 75.53 ± 0.29 | 75.32 ± 0.30 |
| Macro average | — | — | 59.31 | 53.83 | 55.96 |
Test accuracy (%), mean ± standard deviation over four fine-tuning seeds. Random- samples subsets; Last- selects the final contiguous block. Uniformly spaced selection was not evaluated. Logical = LogicalDeduction.
LIFT has higher mean accuracy than Random- in all eight settings and Last- in seven. On Llama-3/GSM8K, the two diagnosed early layers reach 73.96%, versus 66.12% for Random-2 and 56.60% for Last-2. These comparisons show that layer location matters even at the same parameter budget.
4. Accuracy variation on Qwen2/GSM8K (Q4)
Submission Table 2 reports means and standard deviations over four seeds. On Qwen2/GSM8K, LIFT reaches 75.78%, below the original model (78.85%) and AdaLoRA (77.50%), but above all-layer LoRA (74.53%), while reducing training time from 3,500 s to 3,208 s. We regard this as an improvement over all-layer adaptation, not a claim of universal superiority.
5. Model scale and diagnostic scope (weakness 2)
We agree that evaluation on substantially larger models would better establish the scope of LIFT. Our available computational resources did not allow us to add these experiments during the rebuttal period.
As discussed in Part 1, the 11.4% average training-time reduction excludes one-time segmentation and bottleneck diagnosis. The capped diagnostic budget can be reused in the same model-task setting and amortized over repeated adaptation runs. We therefore present the measured savings as training-stage efficiency, rather than an end-to-end speedup claim.
Dear Reviewer,
As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.
If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.
Thank you again for your time and valuable feedback.
Thank the authors for their responses. Based on the discussions, I would like to keep my rating.
Response to Reviewer yijH (Part 1/2): Functional evidence and overhead
Thank you for your thoughtful feedback. We address the functional interpretation, total diagnostic cost, layer-selection controls and the Qwen2/GSM8K result below.
1. Functional stages and task-level evidence (Q1)
Figure 2 compares translations of the same question with different questions in the same language. Same-question representations converge in middle layers and separate near the output, while different questions show an approximately opposite pattern. This contrast appears across two models, five languages and four datasets, with 1,000 pairs per curve. PCA, t-SNE and UMAP show the same overall structure (Section 4.1).
This pattern suggests a progression from language-specific inputs to shared representations and then output language. Wendler et al. (ACL 2024) describe a related input-concept-output progression, while Lad et al. (NeurIPS 2025) find depth-dependent behavior through layer deletion and swapping.
Within reasoning, we identify an idle region as a consecutive block whose adjacent-layer cosine similarity exceeds β=0.94 (Section 5.1). This follows layer-redundancy research: ShortGPT (Men et al., Findings of ACL 2025) tests cosine-based rankings through layer removal, and Gromov et al. (ICLR 2025) use similarity to select blocks for pruning.
We also examine task behavior through the Drop/Recovery interventions in submission Table 1. Drop quantizes a selected block to simulated INT4 while keeping the rest in BF16; Recovery preserves it in BF16 and quantizes the rest. For Llama-3, quantizing the active block causes greater damage than quantizing the idle block on all four tasks, and preserving it gives higher Recovery accuracy on the three logical tasks. These interventions, together with matched-budget fine-tuning results in Part 2, test the hypothesis beyond representational similarities alone.
2. Diagnostic overhead and amortization (Q2; weakness 2)
Submission Table 2 reports shorter training time in all eight settings, averaging an 11.4% reduction relative to all-layer LoRA (FullFT); for example, Llama-3/GSM8K takes 2,667 s instead of 3,347 s.
Fine-tuning processes the full training set for multiple epochs, while diagnosis is a one-time procedure using capped subsets: 1,000 similarity pairs per curve, 128 AWQ calibration examples and at most 1,000 ranking examples. Boundaries are computed from the collected curves. With a fixed model-task setup, this budget does not grow with the adaptation data size and the diagnosis can be reused over repeated runs.
Thus, diagnostic cost can be amortized over larger workloads or repeated adaptations. The 11.4% figure concerns measured training time only; we have not reported a measured end-to-end break-even number of runs.
References