- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This study investigates how to compensate for the distributional changes in intermediate representations resulting from depth pruning of LLMs. The authors propose inserting simple affine correction modules in place of removed Transformer blocks, aiming to approximate their residual contributions while preserving the identity mapping. The module parameters are determined using a closed-form solution to a regularized regression problem based on a small calibration set, rather than through conventional gradient-based fine-tuning. The method also incorporates low-rank decomposition and exact fusion of adjacent affine transformations to improve computational efficiency.
Critical strengths
- The proposed method is simple to integrate into existing depth-pruning pipelines, using lightweight affine transformations to approximate missing residual updates. Its closed-form estimation avoids gradient-based optimization, enabling efficient post-pruning correction.
- The paper considers inference efficiency, showing that low-rank LRAs require fewer parameters and FLOPs than full Transformer blocks, while merging consecutive adapters further reduces runtime overhead.
- The authors also investigate combining closed-form-initialized LRAs with LoRA and partial-layer fine-tuning, demonstrating additional performance improvements in most evaluated settings.
Critical weaknesses
-
The proposed method shares similarities with existing post-pruning recovery approaches, particularly LLM-Streamline and ReplaceMe, which also approximate the effects of removed Transformer blocks using lightweight replacement modules or linear transformations.
- Although Table 15 shows that SHIFT-LLM outperforms ReplaceMe (L2), the practical significance of these improvements remains unclear. Unlike ReplaceMe, which can fuse the learned linear transformation into existing model weights without introducing additional parameters, SHIFT-LLM requires additional adapters. Given that some accuracy improvements are relatively modest, a direct comparison of inference latency and parameter overhead would help clarify whether the additional cost is justified.
- Similarly, Table 5 shows only a marginal improvement of Navigation LLM + LRA over Navigation LLM (59.54 vs. 59.08), with Navigation LLM alone performing better on several individual benchmarks.
- Furthermore, although Table 5 includes LLM-Streamline, its results are taken from prior work rather than obtained under directly comparable experimental settings. LLM-Streamline explores more expressive nonlinear replacement modules, including FFNs and Transformer layers, which may offer advantages in approximating the effects of removed blocks. It is unclear why SHIFT-LLM would be preferable to LLM-Streamline, and the authors should clarify its advantages over this existing approach.
-
The proposed LRA and generic affine baseline in Table 6 have equivalent representational capacity in the full-rank setting, as W=I+A. However, their performance may differ depending on the regularization applied during closed-form estimation. The paper does not clearly specify the regularization settings used for the generic affine baseline or the value and selection procedure of lambda. Furthermore, no sensitivity analysis of lambda is provided. Clarifying these details and conducting a sensitivity analysis of lambda would help determine whether the improvements in Table 6 arise from residual parameterization, regularization, or both.
-
The paper evaluates benchmarks primarily focused on short-answer tasks. While these experiments provide a useful initial assessment, they do not fully characterize the performance of depth-pruned LLMs on more demanding generation tasks, such as multi-step reasoning, instruction following, and structured generation. It would be valuable to extend the evaluation to these tasks and compare the results with existing recovery methods.
-
The evaluation is limited to conventional dense Transformer architectures. It remains unclear whether the proposed affine residual approximation would be effective for recent hybrid architectures such as Qwen3.5, which combines linear and full attention, and Nemotron, which incorporates Mamba-based layers. Their stateful computations may pose additional challenges for affine approximation. Similarly, the method has not been evaluated on MoE models, where input-dependent expert routing may complicate recovery. Further experiments would help establish whether the proposed method generalizes to these architectures in terms of both accuracy and inference efficiency.
Questions for authors
- How was the generic affine baseline in Table 6 fitted, particularly regarding regularization? How was the regularization coefficient lambda for LRA selected, and could the authors provide an ablation study on its effect?
- How does SHIFT-LLM compare with ReplaceMe in terms of inference latency and parameter overhead, given that ReplaceMe can fuse its correction into existing weights without introducing additional parameters?
- What are the main advantages of SHIFT-LLM over LLM-Streamline, which explores more expressive replacement modules? Could the authors provide a comparison under matched experimental settings?
- Could the authors extend the evaluation to more demanding generation tasks, such as reasoning and instruction following, and compare the results with existing recovery methods?
- Could the authors discuss or evaluate the applicability of SHIFT-LLM to hybrid architectures combining full and linear attention, as well as MoE models?
4 comments
Q3. Advantages over LLM-Streamline, under matched settings (new).
The LLM-Streamline numbers in Table 5 were taken from prior work. We now ran LLM-Streamline under matched settings with its official MSE recipe:
- the removed layers plus one neighbouring layer are replaced by one trained Transformer layer, initialized from the first layer of the window;
- the layer is trained with AdamW (lr , cosine schedule, best validation epoch kept);
- the parameter count equals plain depth pruning;
- the removed layers and the 256 C4 sequences are the same as for SHIFT-LLM.
Table C. Llama-3.1-8B-Instruct, 25% pruning, average accuracy (%) over the 7 benchmarks; identical layers and C4 calibration data.
| Criterion | Pruned | ReplaceMe | LinearPatch | LLM-Streamline | SHIFT-LLM |
|---|---|---|---|---|---|
| BI | 57.18 | 59.46 | 61.58 | 61.54 | 61.62 |
| Reverse* | 43.33 | 58.02 | 51.92 | 59.80 | 59.06 |
Recovery cost: LLM-Streamline retrains 218M parameters with backpropagation (10 epochs, ≈193 s); SHIFT-LLM adds 4.2M parameters, fitted in closed form (one forward pass and one linear solve per block).
With the same data, the more expressive trained layer is not clearly more accurate than the closed-form affine correction: it is slightly lower with BI (61.54 vs 61.62) and slightly higher with Reverse* (59.80 vs 59.06). SHIFT-LLM reaches about the same accuracy with several practical advantages:
- no backpropagation, optimizer or training-length tuning (Streamline's best epoch was its last, so its result depends on the training budget);
- every retained weight stays unchanged;
- consecutive corrections merge exactly into one map;
- it plugs into any criterion and any later fine-tuning (Tables 2, 4, 5).
The official LLM-Streamline trains on about sequences. At the small calibration budgets typical of training-free recovery, the extra expressiveness does not pay off.
Q4. More demanding generation tasks (new).
GSM8K. We ran 8-shot chain-of-thought with free-form generation, scored by exact match on all 1,319 problems. All corrections are fitted on the same 256 C4 sequences; no GSM8K data is used.
Table D. GSM8K accuracy (%), Llama-3.1-8B-Instruct, Reverse*, 8-shot chain-of-thought, exact match; all corrections fitted on the same 256 C4 sequences. The unpruned model scores 77.03.
| Method | 12.5% pruning | 25% pruning |
|---|---|---|
| Pruned (no correction) | 0.45 | 0.00 |
| LinearPatch | 19.56 | 3.26 |
| SHIFT-LLM | 58.91 | 8.49 |
Uncorrected depth pruning destroys multi-step generation. At 12.5%, the LRA recovers GSM8K from 0.45 to 58.91 (76% of the unpruned model), 3× better than LinearPatch, and it remains the best method at 25%.
Reasoning model. We pruned DeepSeek-R1-Distill-Llama-8B by 25% with both criteria (Table E).
Table E. DeepSeek-R1-Distill-Llama-8B, 25% depth pruning, zero-shot accuracy (%), same evaluation as the paper.
| Setting | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|
| BI, pruned | 71.31 | 63.22 | 49.56 | 62.83 | 47.01 | 34.73 | 32.80 | 51.64 |
| BI, + LRA | 74.28 | 66.87 | 58.53 | 65.67 | 55.64 | 34.64 | 33.60 | 55.60 |
| Reverse*, pruned | 62.20 | 61.75 | 36.06 | 59.59 | 42.30 | 31.66 | 31.80 | 46.48 |
| Reverse*, + LRA | 63.36 | 68.61 | 56.93 | 64.40 | 56.90 | 36.09 | 35.20 | 54.50 |
The LRA improves 6 of 7 benchmarks with BI and all 7 with Reverse*: +3.96 and +8.02 on average.
Instruction following. We are now running IFEval on Llama-3.1-8B-Instruct with the same methods and will post the results during the discussion period.
Q5. MoE and hybrid architectures (new).
MoE. We tested OLMoE-1B-7B (64 experts, 8 used per token) at 25% pruning; BI and Reverse* remove the same layers (11–14). Here the LRA does not help: the average drops from 49.06 to 47.37 (Table F; BoolQ +3.0, HellaSwag −6.6). We report this openly as a limitation.
Table F. OLMoE-1B-7B, 25% depth pruning (BI and Reverse* remove the same layers 11–14), zero-shot accuracy (%), same evaluation as the paper.
| Setting | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|
| Pruned | 65.50 | 63.49 | 48.68 | 61.09 | 42.47 | 30.38 | 31.80 | 49.06 |
| + LRA | 68.50 | 62.68 | 42.10 | 61.09 | 39.14 | 29.10 | 29.00 | 47.37 |
| Change | +3.00 | −0.81 | −6.58 | 0.00 | −3.33 | −1.28 | −2.80 | −1.69 |
We believe the reason is the routing:
- In a dense block, every token passes through the same weights, so the missing update is close to one function shared by all tokens, and one affine map approximates it well.
- In an MoE block, each token is sent to a different set of experts, so the missing update is a different function for each token. One shared affine map can only fit their average: it helps tasks that rely on general features (BoolQ) but blurs token-specific information (HellaSwag).
A natural fix keeps the closed form: fit one LRA per group of experts and select it with the same router. We will add this result and discussion to the limitations.
Hybrid models. The LRA only needs a block's input and output, so it can be applied to any block type, including linear-attention and Mamba blocks. We could not test Qwen3.5 or Nemotron within the rebuttal period, because they require a newer software stack than our environment; we leave this evaluation to future work.
Follow-up: instruction-following results (IFEval).
As promised, we report IFEval (Zhou et al., 2023) on Llama-3.1-8B-Instruct with Reverse* pruning at 12.5%: chat template, greedy decoding, up to 1,280 new tokens, scored with Google's official IFEval checker on all 541 prompts. All corrections are fitted on the same 256 C4 sequences; no IFEval data is used.
Table F. IFEval accuracy (%), Llama-3.1-8B-Instruct, Reverse*, 12.5% pruning; prompt- and instruction-level, strict and loose.
| Method | Prompt-strict | Inst-strict | Prompt-loose | Inst-loose |
|---|---|---|---|---|
| Original (unpruned) | 75.05 | 82.61 | 79.30 | 85.73 |
| Pruned (no correction) | 17.93 | 30.34 | 21.07 | 34.65 |
| LinearPatch | 20.70 | 32.97 | 24.21 | 36.93 |
| ReplaceMe | 31.42 | 45.20 | 34.20 | 48.80 |
| SHIFT-LLM | 38.26 | 51.56 | 41.04 | 54.44 |
SHIFT-LLM is the best training-free correction on all four metrics. It more than doubles the prompt-level strict accuracy of the pruned model (17.93 → 38.26) and is 6.84 points above ReplaceMe and 17.56 points above LinearPatch.
We thank the reviewer for the detailed and expert review. Every question below is answered with a new experiment under identical settings: same removed layers, same 256 C4 calibration sequences, same evaluator.
Q1. Regularization of the generic affine baseline, choice of λ, and sensitivity.
Table A. Qwen2-1.5B, Reverse*, 25% pruning: average accuracy (%) over 7 benchmarks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA); the pruned model without correction scores 43.71.
The LRA is nearly flat for c≤1 and stays above the pruned model even at c=10. The generic map degrades steadily as λ grows and is worse than the LRA at every λ. Our default c=0.1 is not even the best value for the LRA (45.86 vs 45.95 at c=0.01), which shows that our results do not rely on a tuned λ. At the default, the generic map scores 38.05, matching Table 6 (37.99). The reason is the prior that ridge regularization imposes. Ridge shrinks the fitted map toward zero: for the LRA, zero means the identity (plain pruning), but for the generic map, zero means discarding the hidden state. The gain in Table 6 therefore comes from the residual parameterization and from how it interacts with regularization, not from a tuned λ. We will add the λ rule and this sweep to the paper.
Q2. Latency and parameter overhead vs. ReplaceMe.
We measured all methods under identical conditions: Llama-3.1-8B-Instruct, 25% pruning, one V100, fp16, batch size 1, 128-token prompt, 128 generated tokens.
Table B. Llama-3.1-8B-Instruct, 25% pruning: runtime, parameter overhead and average accuracy over the 7 benchmarks. Merging the consecutive LRAs is exact, so the merged model has the same accuracy.
The rank-64 LRA adds 0.07% parameters and 0.16 ms (0.4%) over ReplaceMe. With exact merging, SHIFT-LLM runs at the same speed as ReplaceMe and plain pruning: the latency differences (≤0.12 ms) are within measurement noise, and generation is only 0.3% slower than ReplaceMe (41.65 vs 41.79 tokens/s), the cost of one extra matrix multiply. In exchange, it is 1.04 points more accurate. On all 8 model/criterion settings, it is more accurate than ReplaceMe (Table 15). The extra cost is therefore negligible relative to the gain.