acceptodds
Under review as a conference paper at ICLR 2027

SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs

Abstract

Depth pruning removes entire Transformer blocks to reduce the inference cost of large language models, but disrupts the hidden-state distributions expected by downstream layers, leading to significant accuracy loss. We introduce SHIFT- LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter (LRA) at each pruning site. Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction. This correction is calibrated via closed-form least-squares regression on a small held-out set, without gradient computation, to approximate the missing residual update produced by the pruned block. Together with the preserved identity pathway, the resulting LRA output approximates the hidden state produced by the original block, thereby mitigating the distributional mismatch introduced by layer removal while avoiding the expensive attention and feed-forward computations of the removed blocks. The resulting LRAs support low-rank factorization and exact merging across consecutive pruned layers for additional compression, and combine naturally with parameter-efficient fine-tuning for further recovery beyond fine-tuning the pruned model alone. Experiments on five model families, six layer-selection criteria, and seven zero-shot benchmarks show that SHIFT-LLM consistently recovers accuracy lost to depth pruning across most configurations, achieving gains up to +15.7 points on Llama-3.1-8B-Instruct while requiring only a few hundred calibration samples and no gradient computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models (LLMs), Depth Pruning, Post-Pruning Correction, Linear Residual Adapter (LRA)

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tJECOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study investigates how to compensate for the distributional changes in intermediate representations resulting from depth pruning of LLMs. The authors propose inserting simple affine correction modules in place of removed Transformer blocks, aiming to approximate their residual contributions while preserving the identity mapping. The module parameters are determined using a closed-form solution to a regularized regression problem based on a small calibration set, rather than through conventional gradient-based fine-tuning. The method also incorporates low-rank decomposition and exact fusion of adjacent affine transformations to improve computational efficiency.

Critical strengths
  • The proposed method is simple to integrate into existing depth-pruning pipelines, using lightweight affine transformations to approximate missing residual updates. Its closed-form estimation avoids gradient-based optimization, enabling efficient post-pruning correction.
  • The paper considers inference efficiency, showing that low-rank LRAs require fewer parameters and FLOPs than full Transformer blocks, while merging consecutive adapters further reduces runtime overhead.
  • The authors also investigate combining closed-form-initialized LRAs with LoRA and partial-layer fine-tuning, demonstrating additional performance improvements in most evaluated settings.
Critical weaknesses
  • The proposed method shares similarities with existing post-pruning recovery approaches, particularly LLM-Streamline and ReplaceMe, which also approximate the effects of removed Transformer blocks using lightweight replacement modules or linear transformations.

    • Although Table 15 shows that SHIFT-LLM outperforms ReplaceMe (L2), the practical significance of these improvements remains unclear. Unlike ReplaceMe, which can fuse the learned linear transformation into existing model weights without introducing additional parameters, SHIFT-LLM requires additional adapters. Given that some accuracy improvements are relatively modest, a direct comparison of inference latency and parameter overhead would help clarify whether the additional cost is justified.
    • Similarly, Table 5 shows only a marginal improvement of Navigation LLM + LRA over Navigation LLM (59.54 vs. 59.08), with Navigation LLM alone performing better on several individual benchmarks.
    • Furthermore, although Table 5 includes LLM-Streamline, its results are taken from prior work rather than obtained under directly comparable experimental settings. LLM-Streamline explores more expressive nonlinear replacement modules, including FFNs and Transformer layers, which may offer advantages in approximating the effects of removed blocks. It is unclear why SHIFT-LLM would be preferable to LLM-Streamline, and the authors should clarify its advantages over this existing approach.
  • The proposed LRA and generic affine baseline in Table 6 have equivalent representational capacity in the full-rank setting, as W=I+A. However, their performance may differ depending on the regularization applied during closed-form estimation. The paper does not clearly specify the regularization settings used for the generic affine baseline or the value and selection procedure of lambda. Furthermore, no sensitivity analysis of lambda is provided. Clarifying these details and conducting a sensitivity analysis of lambda would help determine whether the improvements in Table 6 arise from residual parameterization, regularization, or both.

  • The paper evaluates benchmarks primarily focused on short-answer tasks. While these experiments provide a useful initial assessment, they do not fully characterize the performance of depth-pruned LLMs on more demanding generation tasks, such as multi-step reasoning, instruction following, and structured generation. It would be valuable to extend the evaluation to these tasks and compare the results with existing recovery methods.

  • The evaluation is limited to conventional dense Transformer architectures. It remains unclear whether the proposed affine residual approximation would be effective for recent hybrid architectures such as Qwen3.5, which combines linear and full attention, and Nemotron, which incorporates Mamba-based layers. Their stateful computations may pose additional challenges for affine approximation. Similarly, the method has not been evaluated on MoE models, where input-dependent expert routing may complicate recovery. Further experiments would help establish whether the proposed method generalizes to these architectures in terms of both accuracy and inference efficiency.

Questions for authors
  • How was the generic affine baseline in Table 6 fitted, particularly regarding regularization? How was the regularization coefficient lambda for LRA selected, and could the authors provide an ablation study on its effect?
  • How does SHIFT-LLM compare with ReplaceMe in terms of inference latency and parameter overhead, given that ReplaceMe can fuse its correction into existing weights without introducing additional parameters?
  • What are the main advantages of SHIFT-LLM over LLM-Streamline, which explores more expressive replacement modules? Could the authors provide a comparison under matched experimental settings?
  • Could the authors extend the evaluation to more demanding generation tasks, such as reasoning and instruction following, and compare the results with existing recovery methods?
  • Could the authors discuss or evaluate the applicability of SHIFT-LLM to hybrid architectures combining full and linear attention, as well as MoE models?
4 comments

We thank the reviewer for the detailed and expert review. Every question below is answered with a new experiment under identical settings: same removed layers, same 256 C4 calibration sequences, same evaluator.


Q1. Regularization of the generic affine baseline, choice of λ\lambda, and sensitivity.

  • How λ\lambda is set. λ\lambda is relative to the data scale, λ=c⋅mean(diag(Haug⊤Haug))\lambda = c\cdot\mathrm{mean}\big(\mathrm{diag}(H_{\mathrm{aug}}^\top H_{\mathrm{aug}})\big), with c=0.1c=0.1 fixed for all models, layers and criteria.
  • How the generic baseline is fitted. The generic affine baseline of Table 6 uses exactly the same solver, data and λ\lambda rule. Only the regression target differs: h(ℓ)h^{(\ell)} instead of Δ=h(ℓ)−h(ℓ−1)\Delta = h^{(\ell)} - h^{(\ell-1)}.
  • New sweep. Table A sweeps cc over three orders of magnitude (Qwen2-1.5B, Reverse*, full rank).

 

Table A. Qwen2-1.5B, Reverse*, 25% pruning: average accuracy (%) over 7 benchmarks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA); the pruned model without correction scores 43.71.

cc0.010.1 (default)110
LRA (ours)45.9545.8645.5044.32
Generic affine44.3238.0536.1634.91

 

The LRA is nearly flat for c≤1c \le 1 and stays above the pruned model even at c=10c=10. The generic map degrades steadily as λ\lambda grows and is worse than the LRA at every λ\lambda. Our default c=0.1c=0.1 is not even the best value for the LRA (45.86 vs 45.95 at c=0.01c=0.01), which shows that our results do not rely on a tuned λ\lambda. At the default, the generic map scores 38.05, matching Table 6 (37.99). The reason is the prior that ridge regularization imposes. Ridge shrinks the fitted map toward zero: for the LRA, zero means the identity (plain pruning), but for the generic map, zero means discarding the hidden state. The gain in Table 6 therefore comes from the residual parameterization and from how it interacts with regularization, not from a tuned λ\lambda. We will add the λ\lambda rule and this sweep to the paper.


Q2. Latency and parameter overhead vs. ReplaceMe.

We measured all methods under identical conditions: Llama-3.1-8B-Instruct, 25% pruning, one V100, fp16, batch size 1, 128-token prompt, 128 generated tokens.

 

Table B. Llama-3.1-8B-Instruct, 25% pruning: runtime, parameter overhead and average accuracy over the 7 benchmarks. Merging the consecutive LRAs is exact, so the merged model has the same accuracy.

MethodParamsExtra paramsLatency (ms)Tokens/sAvg. acc (Reverse*)
Original8.030B–57.26 ± 0.1131.97 ± 0.6171.03
Pruned6.285B043.88 ± 0.0941.68 ± 0.0343.33
ReplaceMe (fused)6.285B043.97 ± 0.1141.79 ± 0.0358.02
LinearPatch6.302B16.8M44.29 ± 0.0841.58 ± 0.0251.92
SHIFT-LLM (rank 64)6.290B4.2M44.13 ± 0.1141.52 ± 0.0259.06
SHIFT-LLM (merged)6.302B16.8M43.85 ± 0.0941.65 ± 0.0259.06

 

The rank-64 LRA adds 0.07% parameters and 0.16 ms (0.4%) over ReplaceMe. With exact merging, SHIFT-LLM runs at the same speed as ReplaceMe and plain pruning: the latency differences (≤0.12 ms) are within measurement noise, and generation is only 0.3% slower than ReplaceMe (41.65 vs 41.79 tokens/s), the cost of one extra matrix multiply. In exchange, it is 1.04 points more accurate. On all 8 model/criterion settings, it is more accurate than ReplaceMe (Table 15). The extra cost is therefore negligible relative to the gain.

Q3. Advantages over LLM-Streamline, under matched settings (new).

The LLM-Streamline numbers in Table 5 were taken from prior work. We now ran LLM-Streamline under matched settings with its official MSE recipe:

  • the removed layers plus one neighbouring layer are replaced by one trained Transformer layer, initialized from the first layer of the window;
  • the layer is trained with AdamW (lr 2⋅10−42\cdot10^{-4}, cosine schedule, best validation epoch kept);
  • the parameter count equals plain depth pruning;
  • the removed layers and the 256 C4 sequences are the same as for SHIFT-LLM.

 

Table C. Llama-3.1-8B-Instruct, 25% pruning, average accuracy (%) over the 7 benchmarks; identical layers and C4 calibration data.

CriterionPrunedReplaceMeLinearPatchLLM-StreamlineSHIFT-LLM
BI57.1859.4661.5861.5461.62
Reverse*43.3358.0251.9259.8059.06

 

Recovery cost: LLM-Streamline retrains 218M parameters with backpropagation (10 epochs, ≈193 s); SHIFT-LLM adds 4.2M parameters, fitted in closed form (one forward pass and one linear solve per block).

With the same data, the more expressive trained layer is not clearly more accurate than the closed-form affine correction: it is slightly lower with BI (61.54 vs 61.62) and slightly higher with Reverse* (59.80 vs 59.06). SHIFT-LLM reaches about the same accuracy with several practical advantages:

  • no backpropagation, optimizer or training-length tuning (Streamline's best epoch was its last, so its result depends on the training budget);
  • every retained weight stays unchanged;
  • consecutive corrections merge exactly into one map;
  • it plugs into any criterion and any later fine-tuning (Tables 2, 4, 5).

The official LLM-Streamline trains on about 10510^5 sequences. At the small calibration budgets typical of training-free recovery, the extra expressiveness does not pay off.


Q4. More demanding generation tasks (new).

GSM8K. We ran 8-shot chain-of-thought with free-form generation, scored by exact match on all 1,319 problems. All corrections are fitted on the same 256 C4 sequences; no GSM8K data is used.

 

Table D. GSM8K accuracy (%), Llama-3.1-8B-Instruct, Reverse*, 8-shot chain-of-thought, exact match; all corrections fitted on the same 256 C4 sequences. The unpruned model scores 77.03.

Method12.5% pruning25% pruning
Pruned (no correction)0.450.00
LinearPatch19.563.26
SHIFT-LLM58.918.49

 

Uncorrected depth pruning destroys multi-step generation. At 12.5%, the LRA recovers GSM8K from 0.45 to 58.91 (76% of the unpruned model), 3× better than LinearPatch, and it remains the best method at 25%.

Reasoning model. We pruned DeepSeek-R1-Distill-Llama-8B by 25% with both criteria (Table E).

 

Table E. DeepSeek-R1-Distill-Llama-8B, 25% depth pruning, zero-shot accuracy (%), same evaluation as the paper.

SettingBoolQPIQAHellaSwagWinoGrandeARC-eARC-cOBQAAvg.
BI, pruned71.3163.2249.5662.8347.0134.7332.8051.64
BI, + LRA74.2866.8758.5365.6755.6434.6433.6055.60
Reverse*, pruned62.2061.7536.0659.5942.3031.6631.8046.48
Reverse*, + LRA63.3668.6156.9364.4056.9036.0935.2054.50

 

The LRA improves 6 of 7 benchmarks with BI and all 7 with Reverse*: +3.96 and +8.02 on average.

Instruction following. We are now running IFEval on Llama-3.1-8B-Instruct with the same methods and will post the results during the discussion period.

Q5. MoE and hybrid architectures (new).

MoE. We tested OLMoE-1B-7B (64 experts, 8 used per token) at 25% pruning; BI and Reverse* remove the same layers (11–14). Here the LRA does not help: the average drops from 49.06 to 47.37 (Table F; BoolQ +3.0, HellaSwag −6.6). We report this openly as a limitation.

 

Table F. OLMoE-1B-7B, 25% depth pruning (BI and Reverse* remove the same layers 11–14), zero-shot accuracy (%), same evaluation as the paper.

SettingBoolQPIQAHellaSwagWinoGrandeARC-eARC-cOBQAAvg.
Pruned65.5063.4948.6861.0942.4730.3831.8049.06
+ LRA68.5062.6842.1061.0939.1429.1029.0047.37
Change+3.00−0.81−6.580.00−3.33−1.28−2.80−1.69

 

We believe the reason is the routing:

  • In a dense block, every token passes through the same weights, so the missing update is close to one function shared by all tokens, and one affine map approximates it well.
  • In an MoE block, each token is sent to a different set of experts, so the missing update is a different function for each token. One shared affine map can only fit their average: it helps tasks that rely on general features (BoolQ) but blurs token-specific information (HellaSwag).

A natural fix keeps the closed form: fit one LRA per group of experts and select it with the same router. We will add this result and discussion to the limitations.

Hybrid models. The LRA only needs a block's input and output, so it can be applied to any block type, including linear-attention and Mamba blocks. We could not test Qwen3.5 or Nemotron within the rebuttal period, because they require a newer software stack than our environment; we leave this evaluation to future work.

Follow-up: instruction-following results (IFEval).

As promised, we report IFEval (Zhou et al., 2023) on Llama-3.1-8B-Instruct with Reverse* pruning at 12.5%: chat template, greedy decoding, up to 1,280 new tokens, scored with Google's official IFEval checker on all 541 prompts. All corrections are fitted on the same 256 C4 sequences; no IFEval data is used.

 

Table F. IFEval accuracy (%), Llama-3.1-8B-Instruct, Reverse*, 12.5% pruning; prompt- and instruction-level, strict and loose.

MethodPrompt-strictInst-strictPrompt-looseInst-loose
Original (unpruned)75.0582.6179.3085.73
Pruned (no correction)17.9330.3421.0734.65
LinearPatch20.7032.9724.2136.93
ReplaceMe31.4245.2034.2048.80
SHIFT-LLM38.2651.5641.0454.44

 

SHIFT-LLM is the best training-free correction on all four metrics. It more than doubles the prompt-level strict accuracy of the pruned model (17.93 → 38.26) and is 6.84 points above ReplaceMe and 17.56 points above LinearPatch.

Reviewer_xMZvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SHIFT-LLM, a training-free framework for mitigating hidden-state distribution shifts caused by depth pruning in large language models. The key idea is to replace each pruned Transformer block with a lightweight Linear Residual Adapter (LRA), which preserves the identity mapping and approximates the missing residual update through an affine transformation. The adapter parameters are estimated using closed-form ridge regression on a small calibration set, without gradient-based optimization. The proposed adapters can be further compressed through low-rank factorization. Experiments demonstrate that SHIFT-LLM can recover part of the performance lost due to depth pruning while largely preserving inference efficiency.

Critical strengths
  1. The proposed method is straightforward to implement and does not require gradient-based optimization. Its closed-form estimation and low-rank compression make it computationally attractive for practical LLM compression.
  2. The paper evaluates SHIFT-LLM across five language models, six pruning criteria, and seven zero-shot benchmarks. It also investigates different pruning ratios, calibration sizes, and fine-tuning strategies, providing relatively comprehensive empirical evidence.
Critical weaknesses
  1. Limited novelty. The central technique essentially applies ridge regression to learn the residual updates of removed Transformer blocks. However, the ideas like training-free linear compensation and low-rank decomposition have already been explored in prior work [1-3]. Although SHIFT-LLM targets depth pruning, these modifications appear relatively incremental.
  2. Insufficient comparison with strong width-pruning baselines. The baselines focus primarily on depth pruning. However, the current pruning work of LLM mainly focuses on width pruning. The paper neither discusses the advantages of depth pruning over width pruning nor compares SHIFT-LLM with SOTA width-pruning methods, such as those in [1-4].
  3. The related work section reads more like a list of individual studies, lacking a more logical organization.
  4. The writing could be more concise. For example, lines 40–42 and 44–46 contain overlapping descriptions of structured pruning and its efficiency benefits, which could be consolidated. Reference: [1] Nova A, Dai H, Schuurmans D. Gradient-free structured pruning with unlabeled data, ICML, 2023. [2] An Y, Zhao X, Yu T, et al. Fluctuation-based adaptive structured pruning for large language models, AAAI, 2024. [3] Jiujun He, Huazhen Lin, Olica: Efficient Structured Pruning of Large Language Models without Retraining, ICML, 2025. [4] Tao Yu, et al. Improving Generalization in LLM Structured Pruning via Function-Aware Neuron Grouping, AAAI, 2026.
Questions for authors
  1. Beyond its application to depth pruning in LLMs, how does SHIFT-LLM differ methodologically from the methods in [1-3]? Please provide a detailed comparison and clearly identify the novel contributions of SHIFT-LLM.
  2. Please compare SHIFT-LLM with strong width-pruning methods under matched compression budgets and comparable calibration resources.
4 comments

We thank the reviewer for the careful review. We answer both questions with new experiments and give detailed tables for every claim.


Q1. How does SHIFT-LLM differ from [1–3], and what is new?

[1–3] are width-pruning methods: they make layers narrower and compensate the error of the layers that remain. SHIFT-LLM is a depth-pruning method: it removes whole blocks and replaces each one with a closed-form estimate of its missing output.

 

Table A. What each method prunes and compensates.

MethodPruned unitWhat is compensatedHowModel depth
KCM [1]FFN neurons (BERT)output of the pruned FFNrescaling of the kept neuronsunchanged
FLAP [2]heads, FFN channelsmean of the removed channelsconstant biasunchanged
Olica [3]head dimensions (PCA), FFN neuronserror of a pruned FFNlow-rank linear map, no biasunchanged
SHIFT-LLMwhole blocksfull output change of the removed block (attention + FFN)identity + affine map (AA, bb), closed formreduced

We share the ridge-regression estimator with Olica and do not claim it as new. What is new is what we estimate and how we parameterize it. Three results support this.

 


(1) Keeping the identity path is what makes it work. A generic affine map has the same capacity, solver and data as ours; the only difference is that it predicts the full output instead of the change. It is much worse (Table B), at every regularization strength (Table C).

 

Table B. Qwen2-1.5B, 25% pruning; average accuracy (%) over 7 benchmarks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA).

CriterionPruned+ Generic affine+ LRA (ours)
BI45.7836.9646.85
Reverse*43.7137.9945.71
Magnitude-ℓ1\ell_147.4335.7448.47

 

Table C (new). Qwen2-1.5B, Reverse*, 25% pruning, full rank; average accuracy (%) over the 7 benchmarks; regularization λ=c⋅mean(diag(H⊤H))\lambda = c\cdot\mathrm{mean}(\mathrm{diag}(H^\top H)). The pruned model scores 43.71.

cc0.010.1 (default)110
Generic affine44.3238.0536.1634.91
LRA (ours)45.9545.8645.5044.32

 


(2) It works with any pruning criterion and any model. The LRA improves 22 of 24 settings on the four main models (Table D), and all 4 settings on a larger model and a reasoning model (Table E).

 

Table D. Accuracy gain of + LRA over the pruned model (points), 25% pruning, average over the 7 benchmarks (paper, Table 1).

CriterionQwen2-1.5BQwen1.5-7BLlama-3.1-8BVicuna-7B
BI+1.08+0.76+4.44+0.48
Reverse*+2.00+4.54+15.74−0.51
Magnitude-ℓ1\ell_1+1.03+6.88+1.61+0.22
Magnitude-ℓ2\ell_2+2.02+1.91+3.58+1.92
Taylor+0.98+1.85+15.71+0.01
PPL+2.03+1.64+1.05−4.43
Improved6/66/66/64/6

 

Table E. Larger and reasoning models, 25% pruning; average accuracy (%) over the 7 benchmarks.

ModelCriterionPruned+ LRA (ours)Gain
Qwen2.5-14B (paper, Table 9)BI53.3456.32+2.98
Reverse*46.7252.08+5.36
DeepSeek-R1-Distill-Llama-8B (new)BI51.6455.60+3.96
Reverse*46.4854.50+8.02

 


(3) It beats the existing linear corrections for depth pruning (Table F), and the gap is largest on generation (Tables G and H).

 

Table F (new). Average accuracy (%) over the 7 benchmarks, 25% pruning; same removed layers and the same 256 C4 calibration sequences for all methods.

ModelCriterionPrunedReplaceMeLinearPatchSHIFT-LLM
Qwen2-1.5BBI45.7843.9045.0546.85
Reverse*43.7144.3046.3245.71
Qwen1.5-7BBI52.0650.3454.4252.82
Reverse*49.2553.6253.6953.79
Llama-3.1-8BBI57.1859.4661.5861.62
Reverse*43.3358.0251.9259.06
Vicuna-7BBI55.0554.3356.5855.52
Reverse*53.4746.4053.9652.96
Mean49.9851.3052.9453.54

 

Table G (new). GSM8K accuracy (%), Llama-3.1-8B-Instruct, Reverse*, 8-shot chain-of-thought, exact match; all corrections fitted on the same 256 C4 sequences. The unpruned model scores 77.03.

Method12.5% pruning25% pruning
Pruned (no correction)0.450.00
LinearPatch19.563.26
SHIFT-LLM58.918.49

Q2. Comparison with strong width-pruning methods under matched budgets (new).

We ran this comparison as requested, but accuracy at an equal parameter count is not a like-for-like comparison between depth and width pruning. They are separate research directions with different goals: width pruning keeps every layer and mainly preserves accuracy, while depth pruning removes whole blocks to obtain real inference speedups. Kim et al. [5] show that depth pruning gives clear speedups at small batch sizes where width pruning does not, and both [5] and Men et al. [6] report competitive accuracy. Moreover, SHIFT-LLM is not a competing pruning method but a correction for depth-pruned models; its natural baselines are other depth-pruning corrections, where it performs best on average (Table F) and on generation (Table G). Table H should therefore be read as an accuracy–speed trade-off rather than a ranking.

Setup.

  • Method: we ran FLAP [2] with its official code and default settings.
  • Model: Vicuna-7B, a model supported by its codebase.
  • Budget: the same for all methods, about 24% of the parameters removed (= 25% depth pruning).
  • Calibration: C4 for all methods. FLAP uses its default size; SHIFT-LLM uses 256 sequences.
  • Evaluation: the same evaluator as the paper. The unpruned model scores 66.75 here vs 66.80 in the paper.

 

Table H. Vicuna-7B. Accuracy: average over the 7 benchmarks. Runtime: one V100, fp16, batch size 1, 128-token prompt, 128 generated tokens.

MethodLayersParamsAvg. accLatency (ms)SpeedupTokens/s
Original326.74B66.7554.081.00×31.81
FLAP [2]325.12B59.5556.340.96×17.55†
SHIFT-LLM (BI)245.12B55.5241.021.32×41.80

†FLAP's official model class; its speedup is relative to the original model measured in the same run.

 

What this shows.

  • Accuracy: width pruning keeps more accuracy at the same budget.
  • Speed: width pruning gives no speedup at batch size 1, because it keeps all 32 layers. Removing 8 blocks makes the model 1.32× faster, and SHIFT-LLM keeps this speed while adding accuracy at no cost.

Why depth pruning. Small-batch autoregressive inference is memory-bound: the GPU is underused, so speed depends on memory accesses and the number of sequential operations, not on FLOPs. Width pruning shrinks matrices but keeps every block, so its FLOP savings barely reduce latency, and irregular matrix sizes can even lower GPU efficiency. Depth pruning removes whole blocks, cutting both, and gives direct speedups with standard dense kernels [5].

This agrees with prior work. Kim et al. [5] compare depth and width pruning at similar model sizes: depth pruning gives real speedups at small batch sizes, where width pruning gives little or none, with competitive zero-shot accuracy under moderate pruning. Men et al. [6] (ShortGPT, the origin of our BI criterion) show that simple layer removal is competitive with, and in most of their settings better than, LLM-Pruner and SliceGPT. Table H confirms the speed result for a recent width-pruning method.

We will add these comparisons to the revision and discuss [4] in the related work.

 

[5] B.-K. Kim et al., Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods, arXiv:2402.02834, 2024.

[6] X. Men et al., ShortGPT: Layers in Large Language Models Are More Redundant Than You Expect, arXiv:2403.03853, 2024.

Table H (new). IFEval accuracy (%), Llama-3.1-8B-Instruct, Reverse*, 12.5% pruning; chat template, greedy decoding, Google's official checker on all 541 prompts; all corrections fitted on the same 256 C4 sequences.

MethodPrompt-strictInst-strictPrompt-looseInst-loose
Original (unpruned)75.0582.6179.3085.73
Pruned (no correction)17.9330.3421.0734.65
LinearPatch20.7032.9724.2136.93
ReplaceMe31.4245.2034.2048.80
SHIFT-LLM38.2651.5641.0454.44

Why SHIFT-LLM wins most when pruning causes large damage. LinearPatch can only rescale channels in a fixed rotated basis. This is enough when pruning mainly changes activation magnitudes, as in the milder settings of Table F, where the two methods are within 2 points of each other. When the damage is large, the missing update also changes the direction of the hidden states, which only a full affine correction (channel mixing and bias) fitted to the actual update can restore. This is exactly where SHIFT-LLM wins clearly: Llama-3.1-8B with Reverse*, the largest drop in Table F (+7.1 over LinearPatch), and generation, where the uncorrected model collapses (GSM8K 0.45%, IFEval 17.93%) and SHIFT-LLM recovers 58.91% and 38.26%, far above LinearPatch and 6.84 points above ReplaceMe on IFEval.

I thank the authors for their response and additional experiments. However, compared with the related works discussed above, I still find the technical novelty of this work to be limited. Therefore, I tend to keep my score.

Reviewer_jRZHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

They introduce SHIFT- LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter (LRA) at each pruning site, which approximate the missing residual update produced by the pruned block.

Critical strengths
  1. This article is clear and easy to read.
  2. This work considered various model architectures, but the size was limited to small models.
Critical weaknesses

The phenomenon observed by this work has been confirmed by previous literatures. Fair comparison with these previous works should be complemented.

Observation of the activation distribution gap caused by layer pruning:

[1] Chen, Xinrui, et al. "A simple linear patch revives layer-pruned large language models." Advances in Neural Information Processing Systems 38 (2026): 154484-154509.

Using linear transformation to compensate for layer pruning:

[2] Shopkhoev, Dmitriy, et al. "Replaceme: Network simplification via depth pruning and transformer block linearization." Advances in Neural Information Processing Systems 38 (2026): 19487-19517.

Questions for authors
  1. It states that "the proposed affine residual correction should be estimable from a small calibration set without gradient-based optimization", however, the current layer pruning methods are typically combined with training. Can the proposed method be extended to finetuning?

[1] Chen, Xinrui, et al. "A simple linear patch revives layer-pruned large language models." Advances in Neural Information Processing Systems 38 (2026): 154484-154509.

[3] Chen, Xiaodong, et al. "Streamlining redundant layers to compress large language models." International Conference on Learning Representations. Vol. 2025. 2025.

  1. Formula 7-11 is very similar to the existing work[1-2]. Unless a fair comparison is provided, I will maintain my score.\

  2. Can the author provide the experimental results of the Reasoning model? And more challenging dataset results such as the GSM8K and AIME datasets?

6 comments

We thank the reviewer for the clear and constructive comments. We ran all requested comparisons under identical settings: same removed layers, same 256 C4 calibration sequences, same evaluator.


W1/Q2. Fair comparison with LinearPatch [1] and ReplaceMe [2] ("Eq. 7–11 are similar").

We thank the reviewer for pointing us to LinearPatch, which we will cite and discuss. We re-implemented its training-free version from the official code (P=H diag(d) H⊤P = H\,\mathrm{diag}(d)\,H^\top, one patch per pruned span). Table A compares all methods at 25% depth pruning.

 

Table A. Average zero-shot accuracy (%) over 7 benchmarks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA), 25% depth pruning, identical layers and C4 calibration data.

ModelCriterionPrunedReplaceMe [2]LinearPatch [1]SHIFT-LLM
Qwen2-1.5BBI45.7843.9045.0546.85
Reverse*43.7144.3046.3245.71
Qwen1.5-7BBI52.0650.3454.4252.82
Reverse*49.2553.6253.6953.79
Llama-3.1-8BBI57.1859.4661.5861.62
Reverse*43.3358.0251.9259.06
Vicuna-7BBI55.0554.3356.5855.52
Reverse*53.4746.4053.9652.96
Mean (8 settings)49.9851.3052.9453.54

 

  • SHIFT-LLM has the best mean accuracy and outperforms ReplaceMe in all 8 settings.
  • Against LinearPatch, each method wins 4 settings. The differences are small when the shift is mild. When the shift is large, SHIFT-LLM is much more robust: on Llama-3.1-8B with Reverse*, it scores 59.06 vs 51.92 (+7.1).
  • The gap is decisive on generation (GSM8K, Table D below): 58.91 vs 19.56 for LinearPatch.

How Eq. 7–11 differ from [1,2]. All three methods use a linear map at the pruning site, but they estimate different objects:

  • LinearPatch targets an activation-magnitude mismatch: it rescales each channel, in a Hadamard-rotated basis, by the ratio of activation magnitudes after and before the pruned span (P=H diag(d) H⊤P = H\,\mathrm{diag}(d)\,H^\top, d>0d>0). Its training-free patch matches magnitude statistics rather than fitting the block output, has no bias, and cannot mix channels beyond the fixed rotation. It is therefore a constrained special case of our correction, with A=H(diag(d)−I)H⊤A = H(\mathrm{diag}(d)-I)H^\top and b=0b=0. Its optional refinement trains PP by logit distillation; our LRA can likewise be fine-tuned (Q1).
  • ReplaceMe estimates a linear map TT applied only to the MLP output MiM_i of the preceding block (MiT+Yi≈Li+nM_iT + Y_i \approx L_{i+n}), by least squares on the full output hidden state and without a bias, and fuses TT into that MLP (Table 15, App. B.8).
  • SHIFT-LLM regresses only the missing residual update Δ=h(ℓ)−h(ℓ−1)\Delta = h^{(\ell)}-h^{(\ell-1)} with an affine map (identity path, plus AA, plus bias). The ridge prior then shrinks the correction toward "no correction" rather than toward the zero map.

This residual parameterization is the source of the gain. A generic affine map with the same capacity and the same ridge solver loses 7.7–12.7 points (Table 6). Our new λ\lambda sweep (response to Reviewer tJEC) shows that this holds for every regularization strength.

Q1. Can the method be extended to fine-tuning?

Yes. The LRA is a drop-in module that is fine-tuned jointly with the pruned model; the paper already evaluates this (Tables 2, 4, 5, 14). Table B summarizes the averages.

 

Table B. Average zero-shot accuracy (%) over the 7 benchmarks after fine-tuning on C4 (paper Tables 2 and 4). Unpruned: 59.87 / 65.54 / 71.03 / 66.80.

Fine-tuning (Reverse*, 25%)Qwen2-1.5BQwen1.5-7BLlama-3.1-8BVicuna-7B
LoRA47.5355.5159.9859.38
LoRA + LRA48.94 (+1.41)56.28 (+0.77)61.12 (+1.14)60.10 (+0.72)
Partial-layer FT46.9753.2156.6256.19
Partial-layer FT + LRA46.39 (−0.58)54.19 (+0.98)59.20 (+2.58)58.16 (+1.97)

 

Adding the LRA improves 7 of the 8 fine-tuned settings. Two further results support this:

  • Strong fine-tuned pipeline (Navigation LLM, Table 5): adding the LRA improves 59.08 to 59.54. The gain is small because Navigation LLM already fine-tunes the remaining layers, which absorbs most of the shift; the LRA still adds accuracy on top of this strong baseline.
  • Trainable vs. frozen LRA (Table 14): keeping the LRA trainable is better on all three models tested (+0.96 / +0.20 / +0.80). The closed-form solution is therefore a good initialization for gradient-based recovery.

 


Q3. Reasoning model, GSM8K and AIME.

Reasoning model (new). We pruned DeepSeek-R1-Distill-Llama-8B by 25% with both criteria (Table C).

 

Table C. DeepSeek-R1-Distill-Llama-8B, 25% depth pruning, zero-shot accuracy (%), same evaluation as the paper.

SettingBoolQPIQAHellaSwagWinoGrandeARC-eARC-cOBQAAvg.
BI, pruned71.3163.2249.5662.8347.0134.7332.8051.64
BI, + LRA74.2866.8758.5365.6755.6434.6433.6055.60
Reverse*, pruned62.2061.7536.0659.5942.3031.6631.8046.48
Reverse*, + LRA63.3668.6156.9364.4056.9036.0935.2054.50

 

The LRA improves 6 of 7 benchmarks with BI and all 7 with Reverse*: +3.96 and +8.02 on average. The largest gains are on HellaSwag (+9.0 / +20.9) and ARC-e (+8.6 / +14.6).

GSM8K (new). Llama-3.1-8B-Instruct, Reverse* pruning (the setting with the largest shift on this model), 8-shot chain-of-thought, greedy decoding, exact match, all 1,319 test problems. All corrections are fitted on the same 256 C4 sequences; no GSM8K data is used.

 

Table D. GSM8K accuracy (%), Llama-3.1-8B-Instruct, Reverse. The unpruned model scores 77.03.*

Method12.5% pruning25% pruning
Pruned (no correction)0.450.00
LinearPatch19.563.26
SHIFT-LLM58.918.49

 

At 12.5%, SHIFT-LLM recovers GSM8K from 0.45 to 58.91 (76% of the unpruned model), 3× better than LinearPatch. At 25%, it remains the best method.

Why the accuracy is low. Unlike the zero-shot tasks, which only rank given answers, GSM8K requires generating a full 100–200-token solution with an exact final answer, so small hidden-state errors compound over the generation steps. Removing 25% of the blocks goes beyond what the training-free corrections we tested can recover; fine-tuning on top of the LRA (Tables 2, 4, 14) is the natural next step.

 

IFEval (new).

Table H (new). IFEval accuracy (%), Llama-3.1-8B-Instruct, Reverse*, 12.5% pruning; chat template, greedy decoding, Google's official checker on all 541 prompts; all corrections fitted on the same 256 C4 sequences.

MethodPrompt-strictInst-strictPrompt-looseInst-loose
Original (unpruned)75.0582.6179.3085.73
Pruned (no correction)17.9330.3421.0734.65
LinearPatch20.7032.9724.2136.93
ReplaceMe31.4245.2034.2048.80
SHIFT-LLM38.2651.5641.0454.44

 

Why SHIFT-LLM wins most when pruning causes large damage. LinearPatch can only rescale channels in a fixed rotated basis. This is enough when pruning mainly changes activation magnitudes, as in the milder settings of Table F, where the two methods are within 2 points of each other. When the damage is large, the missing update also changes the direction of the hidden states, which only a full affine correction (channel mixing and bias) fitted to the actual update can restore. This is exactly where SHIFT-LLM wins clearly: Llama-3.1-8B with Reverse*, the largest drop in Table F (+7.1 over LinearPatch), and generation, where the uncorrected model collapses (GSM8K 0.45%, IFEval 17.93%) and SHIFT-LLM recovers 58.91% and 38.26%, far above LinearPatch and 6.84 points above ReplaceMe on IFEval.

AIME. We are now running AIME 2024 on DeepSeek-R1-Distill-Llama-8B at 12.5% pruning with Reverse* (Original, Pruned, LinearPatch and SHIFT-LLM; same 256 C4 calibration sequences) and will post the results during the discussion period.

Comments on the paper

2 comments

Dear Area Chair,

We sincerely thank you for handling our submission, and we thank the reviewers for their time and feedback. We respectfully summarize all rebuttal experiments, the novelty of SHIFT-LLM, and the main clarifications. All comparisons use identical settings: the same removed layers, the same 256 C4 calibration sequences and the same evaluator.

 

1. Experiments addressing the reviewers' concerns (new experiments unless marked "paper")

Concern (reviewer)ExperimentMain result
Fair comparison with LinearPatch and ReplaceMe (jRZH, tJEC)4 LLMs × 2 criteriaBest mean accuracy (53.54 vs 52.94 LinearPatch, 51.30 ReplaceMe); better than ReplaceMe in all 8 settings
LLM-Streamline under matched settings (tJEC)official MSE recipe, same layers and dataSimilar accuracy (61.62 vs 61.54 BI; 59.06 vs 59.80 Reverse*) without backpropagation: 4.2M added parameters vs 218M retrained
Latency and parameters vs ReplaceMe (tJEC)one V100, batch size 1Same speed within noise after exact merging; +1.04 accuracy
Choice of λ and generic affine baseline (tJEC, xMZv)λ over three orders of magnitudeLRA stable (45.95 → 44.32, always above pruning); generic map degrades (44.32 → 34.91)
Fine-tuning (jRZH)LoRA and partial-layer fine-tuning (paper, Tables 2, 4)+ LRA improves 7 of 8 fine-tuned settings
Generality (xMZv)6 criteria × 4 models and Qwen2.5-14B (paper), R1-Distill (new)Improves 26 of 28 settings
Multi-step reasoning, GSM8K (jRZH, tJEC)Llama-3.1-8B-Instruct, 12.5%0.45 → 58.91, i.e. 76% of the unpruned model (LinearPatch 19.56)
Instruction following, IFEval (tJEC)12.5%17.93 → 38.26 prompt-strict; best on all four metrics (ReplaceMe 31.42, LinearPatch 20.70)
Reasoning model (jRZH)DeepSeek-R1-Distill-Llama-8B, 25%+3.96 (BI), +8.02 (Reverse*)
AIME 2024 (jRZH)R1-Distill, 12.5%Only method that still solves problems: 2/30 vs 0/30 (pruned, LinearPatch)
Width pruning (xMZv)FLAP, official code, same budgetFLAP keeps more accuracy but is not faster (0.96×); SHIFT-LLM keeps the full 1.32× speedup
MoE (tJEC)OLMoE-1B-7B49.06 → 47.37; explanation in 3c

 

2. Novelty: concept and formulation

  • Concept. SHIFT-LLM estimates the entire missing residual update of a removed nonlinear block (attention + FFN), for any depth-pruning criterion. LinearPatch only corrects an activation-magnitude mismatch; ReplaceMe re-maps the MLP output of the preceding block; width-pruning methods (KCM, FLAP, Olica) compensate the error of layers that remain in the network.
  • Formulation. h^=h+hA+1b⊤\hat h = h + hA + \mathbf{1}b^\top, fitted in closed form on Δ=h(ℓ)−h(ℓ−1)\Delta = h^{(\ell)} - h^{(\ell-1)} with ridge regularization. LinearPatch (P=H diag(d) H⊤P = H\,\mathrm{diag}(d)\,H^\top, no bias) is a constrained special case (A=H(diag(d)−I)H⊤A = H(\mathrm{diag}(d)-I)H^\top, b=0b = 0); ReplaceMe fits the full output without a bias; Olica fits a bias-free low-rank map to a narrowed FFN; FLAP adds only a constant bias.
  • Why it matters. A generic affine map with the same capacity and solver loses 7.7–12.7 points (Table 6) and remains worse at every λ. Consecutive corrections merge exactly, so the full speedup of depth pruning is preserved.

We do not claim the ridge estimator itself as new; the contribution is what is estimated and how it is parameterized, and the experiments show that this choice is what makes training-free recovery work.

3. Clarifications on the remaining concerns

(a) Reasoning and generation (Reviewer jRZH). The drops on GSM8K and AIME are caused by depth pruning itself: without correction, removing only 4 of 32 layers collapses multi-step generation (GSM8K 77.03 → 0.45; AIME 8/30 → 0/30). SHIFT-LLM is what recovers it: GSM8K 58.91, the best results on IFEval, and the only method that still solves AIME problems. It extends, rather than limits, what depth-pruned models can do.

(b) Width vs. depth pruning (Reviewer xMZv). These are separate research directions with different goals: width pruning keeps every layer and mainly preserves accuracy, while depth pruning removes whole blocks to obtain real inference speedups, as shown by prior work (Kim et al., 2024; Men et al., 2024). A comparison at an equal parameter count is therefore not like-for-like. As requested, we nevertheless compared with FLAP under the same budget: FLAP keeps more accuracy but is not faster than the dense model, whereas SHIFT-LLM keeps the full 1.32× speedup. SHIFT-LLM is a correction for depth pruning, and among depth-pruning corrections it gives the best average accuracy.

(c) MoE (Reviewer tJEC). The OLMoE result has a natural explanation. In an MoE block, each token is routed to different experts, so the missing update differs from token to token, and one shared affine map can only fit its average. This points to a direct closed-form extension, one LRA per routing group selected by the same router, rather than to a problem with the residual formulation.

 

4. Planned revisions

We will (i) cite and discuss LinearPatch, KCM, FLAP, Olica and the other suggested works; (ii) add all new tables; (iii) state the λ rule and add the sensitivity study; (iv) add the MoE analysis with the routing-aware extension, and hybrid architectures as future work; and (v) reorganize the related work.

 

In summary, SHIFT-LLM is a closed-form, training-free correction that contains LinearPatch as a special case, matches a trained replacement layer (LLM-Streamline) without any training, gives the best average accuracy among training-free corrections and the best results on IFEval and AIME, and preserves the full speedup of depth pruning. We would be grateful if these results could be taken into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.