acceptodds
Under review as a conference paper at ICLR 2027

Output-aware Residual Stream Pruning for Large Language Models

Abstract

Residual stream pruning methods reduce inference cost by shrinking the model's hidden dimension, but existing approaches typically choose these dimensions by minimizing activation reconstruction error. This criterion implicitly treats all perturbation directions as equally important, ignoring the sensitivity of downstream layers. We introduce a sensitivity-aware approach to residual-stream pruning that directly accounts for this direction-dependent sensitivity. Using a second-order approximation to the output KL divergence, we characterize the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output. The resulting subspace selection objective couples these two quantities, but is difficult to optimize directly. We derive a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the efficiency and structural simplicity of rotation-based pruning methods. Across several instruction-tuned language model families, our method consistently reduces calibration KL divergence relative to activation-only pruning and improves perplexity and downstream task performance over a range of compression levels. Our results show that preserving activation energy alone is insufficient for residual-stream pruning, and that explicitly accounting for how perturbations propagate to the model output provides a more effective criterion for selecting dimensions to remove.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model compression, network pruning, structured pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fKdWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Residual stream pruning methods typically choose the dimensions by minimizing activation reconstruction error. They characterized the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output and derived a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the efficiency and structural simplicity of rotation-based pruning methods.

Critical strengths

This article is well-written, with a clear description of the motivation (as shown in Figure 1), and the proposed method is simple and effective.

Critical weaknesses
  1. The author chose SliceGPT as the main baseline because it is the most similar method. However, did the author consider comparing with newer pruning methods?

Training-free pruning:

[1] Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, and Le Sun. ShortV: Efficient multimodal large language models by freezing visual tokens in ineffective layers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 329–339, 2025.

[2] Xinrui Chen, Hongxing Zhang, Fanyi Zeng, Yongxian Wei, Yizhi Wang, Xitong Ling, Guanghao Li, and Chun Yuan. Prune&Comp: Free lunch for layer-pruned LLMs via iterative pruning with magnitude compensation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 20316–20324, 2026.

Training-base pruning:

[3] Chen, Xiaodong, et al. "Streamlining redundant layers to compress large language models." International Conference on Learning Representations. Vol. 2025. 2025.

[4] Chen, Xinrui, et al. "A simple linear patch revives layer-pruned large language models." Advances in Neural Information Processing Systems 38 (2026): 154484-154509.

  1. My main concern is that this work is an improvement for SliceGPT and it has relatively hard scalability. I hope the author can clarify whether the proposed method is applicable to quantitative methods such as YAQA, and whether it can improve the evaluation of other pruning methods?

  2. The intermediate activation distribution of the Reasoning model is highly correlated with the input type. Did the authors consider the selection of the calibration set, and did they take into account the relevance between FineWeb/REDPAJAMA and the downstream tasks?

Questions for authors

see weakness

3 comments

We thank the reviewer for their time and helpful comments. Please find the answers to the raised concerns and questions below.

Comparison with newer layer pruning methods and compatibility with different compression approaches

We thank the reviewer for bringing these layer pruning methods to our attention. We have updated our related works section to mention these. Below, we provide additional results both comparing residual-streaming pruning with one of the suggested layer pruning methods as well as an exploration into whether we may achieve greater performance at matched parameter counts through combining the two approaches.

For this experiment, we use the suggested Prune & Comp as the layer pruning method. Like residual-stream pruning, layer pruning incurs substantial performance degradation at larger pruning rates. However, as layer pruning leaves the size of the original layers intact, residual-stream pruning provides an alternative approach for reducing model size after layers have been removed. We conduct this experiment on Llama 3.1 8B Instruct with a fixed target parameter count of 6.28B parameters, equivalent to removing 8 transformer layers. For both layer pruning and residual-stream pruning, we use the same set of 1,024 FineWeb-edu samples as calibration. We vary both the number of layers removed as well as the residual-stream pruning method and present results below.

# Layers prunedResidual-stream Pruning MethodKLwikiKL_{\text{wiki}}PPLAvg. Acc.
8N/A1.58435.2958.05
6SliceGPT1.66439.3857.02
6Ours1.18424.0557.90
4SliceGPT1.72642.2456.75
4Ours1.12222.8958.21
2SliceGPT1.82946.5155.75
2Ours1.24125.7957.00
0SliceGPT2.07860.0653.46
0Ours1.46432.4255.17

First, it can be seen that without combining the two approaches, our method achieves better WikiText performance but performs worse on common-sense reasoning. Combining our method with layer pruning consistently improves performance on both WikiText and common-sense reasoning relative to stand-alone residual-stream pruning. The best performance across all metrics is achieved when Prune & Comp removes 4 layers and residual-stream pruning is applied using our method. Finally, it can be seen that SliceGPT is not able to improve over layer pruning in this configuration. Even the best combination of SliceGPT with layer pruning results in both worse WikiText PPL and worse common-sense reasoning performance.

The above results provide an initial exploration into combining residual-stream pruning with other compression methods. These results show that layer pruning and residual-stream pruning are complementary and, when used together, can achieve greater results than when each is used in isolation.

The intermediate activation distribution of the Reasoning model is highly correlated with the input type. Did the authors consider the selection of the calibration set, and did they take into account the relevance between FineWeb/REDPAJAMA and the downstream tasks?

We thank the reviewer for the opportunity to clarify the choice of calibration data. We agree with the reviewer that the choice of calibration data may significantly impact performance on downstream tasks. FineWeb-edu was chosen as the calibration dataset for our main experiments as it is a high quality dataset that has been heavily filtered and is composed entirely of text from web pages deemed to have high educational quality. We therefore anticipate that it will be relatively representative of both the language modeling and common-sense reasoning tasks we evaluate on.

Unlike FineWeb-edu, RedPajama is composed of general web text. We therefore choose RedPajama as an additional calibration set for evaluation due to its greater diversity. By additionally evaluating our method with RedPajama, we are able to evaluate whether our method continues to provide benefits as the calibration data distribution changes. Finally, by comparing the performance of models compressed with FineWeb-edu versus models compressed with RedPajama, we see that FineWeb-edu calibration results in greater retention of common-sense reasoning performance while achieving comparable language modeling performance.

The Table you submitted doesn't seem to include the comparison of Prune&Comp and the combination. Could you please check if you have provided the wrong table?

Thank you for your response, and I apologize for the confusion regarding the comparison and combination with Prune & Comp. The table I provided is correct, but the presentation could have been made clearer. The first row corresponds to Prune & Comp alone (8 layers removed) and the bottom two rows correspond to SliceGPT and our method without the combination. The intermediate rows correspond to Prune & Comp combined with either SliceGPT or our method, where fewer layers are removed and residual-stream pruning is applied to reach the same target parameter count of 6.28B. I hope this clarifies the table, and I am happy to answer any further questions or provide additional clarification.

Reviewer_KS25OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an output-aware approach to residual-stream pruning of large language models. Building on the SliceGPT framework, it incorporates the sensitivity of the final output distribution into the selection of directions to remove. A local second-order approximation of output KL divergence yields a surrogate objective combining activation second moments and Fisher information. The authors provide an exact characterization of the single-direction problem and a spectral upper-bound construction for removing multiple directions. Candidate subspaces are generated using different trade-off parameters and then selected according to the original surrogate objective.

Experiments across multiple model families and compression ratios show consistent improvements over SliceGPT in KL divergence and perplexity, with additional benefits on downstream tasks and moderate additional calibration cost.

Critical strengths
  1. A well-motivated and practical extension of residual-stream pruning. Activation reconstruction error alone does not distinguish perturbations according to their effects on the final output distribution. Incorporating output sensitivity directly addresses this limitation. The paper develops this motivation into a concrete subspace-selection objective and a tractable spectral procedure, distinguishing the exact single-direction result from the upper-bound construction for multiple directions.

  2. Consistent empirical improvements over the most directly relevant baseline. In Table 1, the method improves both KL divergence and perplexity over SliceGPT across all five models and four compression ratios. The downstream evaluations provide additional evidence of its usefulness. The approximately 17–19% increase in calibration time reported in Table 4 appears moderate relative to the observed quality improvements. I also appreciate the reconstruction-error-controlled analysis in Figure 4, which provides empirical support for using a shared sensitivity matrix.

Critical weaknesses
  1. Make the practical estimation of HH more explicit. Appendix C briefly describes the sampling procedure and the use of a single backward pass. Adding a compact formula showing how the resulting gradient outer products are accumulated and normalized would help readers connect the implementation to the Fisher quantity defined in the main text. This is a suggestion to make the explanation more self-contained, rather than a concern that the estimator is incorrect.

  2. Clarify the algorithmic progression in Section 4.2. The section would be easier to follow with a short explanation that first fixes τ\tau: construct MτM_\tau, obtain a candidate subspace from its smallest eigenvectors, and explain that this candidate minimizes the corresponding upper bound. The text could then describe varying τ\tau to generate several candidates and selecting among them using the original surrogate objective L(U)L(U). This would make the respective roles of the upper bound and the original objective more immediately apparent.

Questions for authors

see the weaknesses

1 comment

We thank the reviewer for their positive assessment as well as for their feedback, which helps improve the clarity of our submission. Regarding both the practical estimation of HH and the algorithmic progression of Section 4.2, we have updated the manuscript to make these sections more clear. For convenience, we include the revised sections below.

Practical estimation of HH

We estimate the sensitivity matrices HH using one token sampled from the model output distribution per token and accumulate these estimates in single precision. Computing an independent backward pass for every sampled class would be prohibitively expensive; instead, we average the corresponding sampled log-probabilities and obtain the corresponding gradient using a single backward pass. We describe this formally for a single sequence ss and the activation xix_i corresponding to token ii below:

\mathcal{J}(s) &= -\sum_{j=1}^{|s|} \mathbb{E}_{y\sim p_\theta(\cdot \mid s_{\leq j})} \left[\log p_\theta(y\mid s_{\leq j})\right] \\\\ H_i(s) &= \frac{1}{|s|} \nabla_{x_i}\mathcal{J}(s) \nabla_{x_i}\mathcal{J}(s)^\top \end{aligned}$$ where the inner expectation is computed by sampling only a single value of $y$. Note that $H_i(s)$ in this example represents the sensitivity estimate for a single activation and sequence. As described in the main text, the final sensitivity is averaged over all activations and sequences: $$\begin{aligned} H &= \mathbb{E}_{s\sim\mathcal{D}_{\mathrm{cal}}} \left[ \frac{1}{|s|} \sum_{i=1}^{|s|}H_i(s) \right]. \end{aligned}$$ > Section 4.2 algorithmic progression **Candidate selection.** Given $\tau$, the $k$-dimensional subspace which minimizes $\mathcal{B}_\tau(U)$ is: $$\begin{aligned} U_\tau &= \underset{U^\top U=I_k}{\arg\min} \mathcal{B}_\tau(U) = \operatorname{eig}_{\min,k}(M_{\tau}). \end{aligned}$$ As this is available in closed form, what remains is selecting a value of $\tau$ which minimizes $\mathcal{L}(U_\tau)$ most effectively. We solve this problem by generating a set of candidate subspaces and evaluating each. Given a grid $\mathcal{T}=\{\tau_j\}_{j=1}^{n}\subset(0,\infty)$, we compute the corresponding candidate subspaces $\{U_{\tau_j}\}_{j=1}^{n}$. We then select $U_{\tau_{j^\star}}$ where $$\begin{aligned} j^\star &\in \arg\min_{1\leq j\leq n} \mathcal{L}(U_{\tau_j}) \\\\ &= \arg\min_{1\leq j\leq n} \operatorname{Tr}\left( U_{\tau_j}^\top C U_{\tau_j}\, U_{\tau_j}^\top H U_{\tau_j} \right). \end{aligned}$$ This procedure requires $n$ independent eigendecompositions, which can be parallelized to reduce wall-clock time at the cost of additional memory. In our experiments, a grid of $n=5$ values of $\tau$ is sufficient to achieve strong performance.
Reviewer_jFSVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a sensitivity-aware variant of SliceGPT for residual-stream pruning. SliceGPT chooses which hidden dimensions to remove by activation energy (PCA). This paper instead derives a layerwise criterion from a second-order approximation of the output KL divergence, so directions are scored by both activation magnitude and how sensitive the model output is to them, with sensitivity estimated by a dataset-averaged Fisher matrix. The resulting objective has no closed-form solution. The authors show that the single-direction case is solved exactly by an eigendecomposition of a weighted combination of the covariance and sensitivity matrices, and that for multiple directions the same construction minimizes an upper bound. In practice they sweep five weighting values and keep the best candidate.

Experiments on five instruction-tuned models and three base models (3B–14B, Llama, Mistral, and Phi families) at 10–30% slicing show consistently lower KL and perplexity than SliceGPT, mostly higher zero-shot accuracy, and larger gains on GSM8K and IFEval. Calibration time increases by about 17–19%.

Critical strengths
  1. Consistent improvement at no extra inference cost. The pruned model has exactly the same structure as SliceGPT’s, so the comparison is fair. The KL and perplexity gains hold across all 8 models and sparsity levels, sometimes by large margins.
Critical weaknesses
  1. Incremental contribution. The method only replaces SliceGPT’s PCA criterion with a Fisher-weighted one, an idea already standard in compression (YAQA, LLM Surgeon, GFWSVD). Moreover, Table 3 shows that directly optimizing the derived objective hurts WikiText KL and perplexity, suggesting the gains may come partly from the approximation rather than the objective itself.

  2. Limited evaluation. The only main baseline is SliceGPT. There are no recent hidden-dimension pruning methods (e.g., DISP-LLM, MoDeGPT), no measured speedup or memory, and no recovery fine-tuning. All models are dense 3B–14B Llama-style architectures, with no larger or MoE models.

Questions for authors
  1. Table 3 shows that directly optimizing your objective worsens WikiText results. Is the gain coming from the derived objective or from the regularization implicit in the approximation? Results for a fixed weighting value would help.
  2. The sensitivity matrix is estimated once on the unpruned model, but pruning is sequential. How is it aligned with the rotated, partially pruned basis at each layer?
  3. Is the pure SliceGPT solution included in the candidate set, and how sensitive are results to the weighting grid?
  4. Several accuracy gains are under 1 point, and Phi-3 at 10% is worse. Can you report variance over multiple calibration subsets?
  5. Sensitivity estimation needs full-model backward passes, unlike SliceGPT. What is the memory and time cost at 70B scale?
3 comments

We thank the reviewer for the positive assessment and their helpful comments. Please find responses to the raised questions below.

Effect of additional optimization on WikiText and weighting grid sensitivity.

Regarding the behavior of the pruning basis with additional optimization, it should first be noted that, although the WikiText results worsen with additional optimization, the calibration KL improves. Our objective approximates the KL divergence on the calibration data via a second-order approximation. Hence the reviewer makes the keen observation that, while additional optimization does generally result in further minimization of the calibration KL, additional exploration is necessary to confirm whether the improvement in WikiText perplexity is truly a result of minimizing our proposed objective. Below we address this concern by investigating whether reductions in calibration KL via our layerwise optimization correlate with WikiText performance.

As you recall, the weighting grid defines the set of values of τ\tau which we search during optimization. The weighting grid, therefore, provides a mechanism for us to explore this question directly. By removing points from the grid, we expect to find worse solutions for the layerwise surrogate and thus achieve worse calibration KL. We fix the minimum and maximum values of our weighting grid to τmin=1\tau_{min}=1 and τmax=70\tau_{max}=70 and sample nn points evenly on a log scale between them. We conduct this experiment on Llama 3.1 8B Instruct at 10% and 20% pruning rates and measure calibration KL, WikiText KL, and WikiText PPL performance below:

10% pruning rate:

nnKLcalib\text{KL}_{\text{calib}}KLwiki\text{KL}_{\text{wiki}}PPL
20.1500.57713.18
30.1480.56413.01
40.1380.48512.08
50.1350.37210.71
60.1370.40611.06
70.1340.39510.99

20% pruning rate:

nnKLcalib\text{KL}_{\text{calib}}KLwiki\text{KL}_{\text{wiki}}PPL
20.3591.10822.69
30.3531.07121.94
40.3391.02220.86
50.3280.86517.75
60.3290.89518.27
70.3270.87017.83

First, it can be seen that increasing the resolution of the grid up to n=5n=5 consistently results in an improvement in both calibration KL and WikiText KL/PPL. These results suggest that, up to the solution quality achieved at n=5n=5, finding a better solution to our proposed objective corresponds to both better WikiText performance and calibration KL. In addition, our method achieves better performance than SliceGPT even at n=2n=2. This shows that, although performance does depend on the quality of the weighting grid, our method is beneficial even when used with a suboptimal grid.

We believe this experiment has substantial value in characterizing the behavior of our method under different weighting grids. We have therefore updated the manuscript to include this experiment within the appendix.

Results over multiple calibration subsets

Please find below additional results on Llama-3.1 8B instruct and Phi-3 Medium Instruct. We report mean and standard deviation across three sets of calibration data generated using different random seeds.

Llama-3.1 8B Instruct

SparsityMethodPPLKLAvg Acc.
10%SliceGPT15.73±0.1915.73 \pm 0.190.758±0.0130.758 \pm 0.01365.85±0.1365.85 \pm 0.13
10%Ours10.93±0.1210.93 \pm 0.120.393±0.0120.393 \pm 0.01266.45±0.3066.45 \pm 0.30
20%SliceGPT27.29±0.5827.29 \pm 0.581.295±0.0211.295 \pm 0.02161.29±0.2161.29 \pm 0.21
20%Ours18.04±0.0318.04 \pm 0.030.881±0.0010.881 \pm 0.00161.92±0.2061.92 \pm 0.20
25%SliceGPT38.38±1.6438.38 \pm 1.641.635±0.0411.635 \pm 0.04158.19±0.2258.19 \pm 0.22
25%Ours23.69±0.2823.69 \pm 0.281.153±0.0101.153 \pm 0.01058.92±0.1058.92 \pm 0.10
30%SliceGPT53.41±2.1653.41 \pm 2.161.965±0.0381.965 \pm 0.03854.00±0.0154.00 \pm 0.01
30%Ours30.96±0.6130.96 \pm 0.611.419±0.0201.419 \pm 0.02055.49±0.5155.49 \pm 0.51

Phi-3 Medium Instruct

SparsityMethodPPLKLAvg Acc.
10%SliceGPT6.44±0.076.44 \pm 0.070.531±0.0120.531 \pm 0.01272.76±0.2972.76 \pm 0.29
10%Ours6.00±0.066.00 \pm 0.060.428±0.0110.428 \pm 0.01172.69±0.1972.69 \pm 0.19
20%SliceGPT10.87±0.1610.87 \pm 0.161.058±0.0141.058 \pm 0.01468.59±0.0468.59 \pm 0.04
20%Ours8.52±0.098.52 \pm 0.090.812±0.0100.812 \pm 0.01070.09±0.4870.09 \pm 0.48
25%SliceGPT13.70±0.1213.70 \pm 0.121.289±0.0081.289 \pm 0.00865.60±0.3765.60 \pm 0.37
25%Ours10.43±0.1410.43 \pm 0.141.021±0.0111.021 \pm 0.01167.74±0.3067.74 \pm 0.30
30%SliceGPT16.95±0.4016.95 \pm 0.401.502±0.0231.502 \pm 0.02362.16±0.3462.16 \pm 0.34
30%Ours13.04±0.1713.04 \pm 0.171.247±0.0111.247 \pm 0.01164.79±0.2864.79 \pm 0.28

As can be seen, our method remains above SliceGPT in all cases except for the common-sense reasoning results for Phi-3 Medium at 10% pruning. If accepted, we will update all results in Table 1 and Table 2 to include standard deviations using calibration data generated from three random seeds.

Is the pure SliceGPT solution included in the candidate set?

We do not explicitly include the SliceGPT solution in the candidate set. However, as τ→∞\tau \rightarrow \infty the pruning basis should exactly recover the SliceGPT basis. Thus, if our weighting grid contains large enough values, then our solution set will effectively contain the SliceGPT solution. We check whether this is the case for the weighting grid used in our experiments by measuring the average projection similarity 1k∣∣Uτ⊤USliceGPT∣∣F2\frac{1}{k}||U_{\tau}^\top U_{\text{SliceGPT}}||_F^2 for both τ=1\tau=1 and τ=70\tau=70 at each pruning rate on Llama 3.1 8B instruct.

Pruning rateτ=1\tau=1τ=70\tau=70
10%0.110.97
20%0.220.98
25%0.280.98
30%0.340.99

As can be seen, when τ=70\tau=70 the resulting basis is extremely similar to the basis used by SliceGPT. When τ=1\tau=1, however, the basis has very low similarity with the SliceGPT basis. We include this result primarily to show that the bases generated with smaller values of τ\tau are sufficiently different from the SliceGPT bases.

Recovery fine-tuning

Recovery fine-tuning (RFT) is interesting as it allows us to explore whether our method maintains an advantage over SliceGPT when both are fine-tuned afterward. To explore this, we fine-tune models pruned with our method and SliceGPT with LoRA (rank=8) using the Alpaca-GPT4 dataset. We fine-tune on 8,000 sequences of length 1,024 for 1 epoch. Results for Llama 3.1 8B Instruct at 20% and 30% pruning rates are presented below. For convenience, we include both values before and after RFT.

Pruning rateMethodPPLKLAvg. Acc
20%SliceGPT27.87→18.5427.87\to18.541.317→0.9411.317\to0.94161.06→63.0061.06\to63.00
20%Ours18.04→13.0218.04\to13.020.882→0.5980.882\to0.59862.13→63.3162.13\to63.31
30%SliceGPT55.90→23.5855.90\to23.582.009→1.1722.009\to1.17251.30→59.5251.30\to59.52
30%Ours31.32→16.7631.32\to16.761.430→0.8381.430\to0.83855.72→59.7855.72\to59.78

As can be seen, our method maintains a consistent advantage over SliceGPT in WikiText performance after RFT. In addition, although the common-sense reasoning gap closes, our method maintains a slight advantage there as well. The manuscript has been updated to include these results in the appendix.

Compression timing for 70B model

Please find compression timing results for Llama 3.1 70B measured on an RTX PRO 6000 Blackwell GPU below.

MethodSensitivity EstimationDirection SelectionTotal
SliceGPTN/A10:5710:57
Ours03:4011:2515:05

Missing hidden-dimension pruning methods (DISP-LLM, MoDeGPT)

We agree that these are relevant and have expanded the related works section to discuss both of these and their relationship to our work. As this reviewer pointed out, DISP-LLM and MoDeGPT remove input and output dimensions from the self-attention and MLP blocks and are thus similar to our work. However, we believe it is worth distinguishing that, unlike our work and SliceGPT, they do so in a manner such that the dimensionality of the residual connection is preserved.

Effect of sequential pruning on sensitivity estimation

As this reviewer describes, the sensitivity matrix is estimated once prior to pruning and then remains fixed. The sensitivity matrix is built from a second-order Taylor series expansion around the original unpruned model. Therefore, the original sensitivity matrix may not be applicable to the model after layers have been pruned. We thus investigate the impact of only estimating the sensitivity once prior to pruning as follows. We begin by estimating the sensitivity for the first pruning location with respect to the original unpruned model. We then prune the first location and subsequently estimate the sensitivity matrix at the following pruning site with respect to the partially pruned model. We repeat this process until the entire model has been pruned. Sensitivity is estimated with respect to the partially pruned model as otherwise the first-order terms in the Taylor series expansion could no longer be ignored. We conduct this experiment on Llama-3.2 3B and present results below:

Pruning rateUpdated HHPPLKLwiki\text{KL}_{\text{wiki}}KLcalib\text{KL}_{\text{calib}}Avg. Acc.
10%N14.300.20170.099658.53
10%Y14.170.19660.096258.42
20%N19.670.51390.266054.92
20%Y19.110.49230.254855.03
25%N24.370.72900.387151.95
25%Y22.790.66530.365052.22
30%N30.140.94350.526249.43
30%Y27.730.85580.491149.64

As can be seen, interweaving sensitivity matrix estimation with pruning consistently provides modest benefits in terms of WikiText performance and calibration KL and often provides a modest increase to common-sense reasoning scores. Importantly, the amount of improvement increases with pruning rate. This is expected as the original sensitivity estimate may become less reliable as the model changes more significantly. Finally, while this experiment shows that performance can be further improved by accounting for the change in the sensitivity estimate due to pruning, the improvements are relatively modest. We thus conclude that, despite estimating the sensitivity matrices prior to pruning, the estimates remain beneficial even after layers have been pruned.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.