acceptodds
Withdrawn from ICLR 2027

Causal Importance Distillation for Transformer Module Pruning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Existing adaptive inference methods for Large Language Models rely on observational heuristics, such as hidden-state similarity or activation magnitudes, to drop redundant modules. However, these correlation-based metrics often fail to capture subtle, non-linear structural computations vital for semantic accuracy. We introduce CausalGate, an intervention-guided framework for compute-efficient transformer inference. During a calibration phase, CausalGate isolates individual Attention and MLP sub-layers, zeros out their respective outputs, and measures the resulting change in the predictive distribution via the Kullback-Leibler divergence of the final logit distribution. To eliminate runtime routing overhead, this intervention-derived importance hierarchy is distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average smoothing objective paired with a differentiable pairwise ranking loss. We comprehensively evaluate CausalGate on TinyLlama-1.1B across language modeling and commonsense reasoning benchmarks, with comparisons against prominent routing and module-skipping baselines. Scaling experiments on Qwen2.5-3B and Llama-3.1-8B further evaluate perplexity under increasing module removal, while hardware measurements on TinyLlama-1.1B and Qwen2.5-3B demonstrate practical inference speedups.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Causal Intervention, Efficient language models, Structured pruning, Compute-aware inference, Transformer interpretability, Model compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Pr7eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CausalGate, a method for pruning attention and MLP submodules of pretrained transformers. It quantifies importance by dropping individual modules and computing the KL divergence of the resulting predictive distribution. These scores are used to supervise static scalar gates via EMA smoothing and a pairwise ranking objective, along with language-modeling and sparsity losses. The learned ranking decides which modules are dropped at inference time. Experiments We mainly evaluate TinyLlama-1.1B and also provide additional perplexity results for Qwen2.5-3B and Llama-3.1-8B models and inference measurements for 1.1B and 3B models.

Critical strengths
  • The method assesses the importance of modules by analyzing changes in the final predictions, hence capturing downstream effects that local activation-based metrics may not capture using the KL metric.
  • Static module selection avoids input-dependent routing and online importance estimation.
  • The paper demonstrates measured inference speedups on TinyLlama-1.1B and Qwen2.5-3B and evaluates both language-modeling perplexity and downstream task accuracy.
Critical weaknesses
  1. The novelty is insufficiently distinguished from prior work. FinerCut [1] already evaluates separate attention/FFN removals using output discrepancies, including Jensen–Shannon divergence. BlockPruner [2] and SLEB [3] evaluate candidate removals using perplexity or negative log-likelihood and iteratively update their choices. KL-based approaches also exist: ACDC [4] evaluates edge ablations through output KL, Edge Pruning [5] optimizes differentiable masks using a KL-based objective, and task-specific circuit extraction [6] uses component ablations and KL changes to guide pruning. Although these methods differ in granularity and application, they undermine the broad characterization of previous work as relying on observational proxies. Separate attention/MLP pruning and heterogeneous redundancy across these modules were also studied in [7].

  2. The calibration cost and scalability are not well evaluated. Section 4.1 describes the training steps and hardware. Calibration wall-clock time, GPU-hours, and peak memory usage are not reported. Algorithms 1–2 appear to recompute all single-module interventions at each training iteration, requiring roughly (M+1)(M+1) forward evaluations per intervention sweep, plus gradient computation. I can see this process getting more and more expensive as model size and number of modules increase. The paper shows perplexity experiments for 3B and 8B, but no evaluation beyond 8B or calibration-cost metrics to show practical scalability. This cost of calibration is not justified by speedups in inference alone.

  3. It is not clear whether learning at the gate adds any benefit. Aggregate intervention scores directly over ranks. This produces a static mask with no routing at runtime. Appendix E shows a useful comparison. EMA-only ranking preserves perplexity better at 30-40% removal rate. Also, its EMA targets are from the gate-training run, so this does not test a stand-alone calibration procedure without gate optimization. Comparison against averaged-KL ranking and iterative removal under matched calibration budgets would better isolate the contribution.

  4. The empirical gains are not consistent. ShortGPT achieves better WikiText-2 and C4 perplexity, HellaSwag accuracy, and WinoGrande accuracy at 10% removal. At 20% CausalGate reduces perplexity compared to ShortGPT, but its perplexity on WikiText-2 increases from 13.58 for the unpruned model to 67.00. These results support a conditional trade-off rather than a general superiority. Matching module removal counts does not necessarily translate to computational savings because the attention and MLP modules have different costs. It would be more informative to compare at matched measured latencies.

[1] Yang Zhang et al. FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models. 2024. arXiv:2405.18218. [2] Longguang Zhong et al. BlockPruner: Fine-grained Pruning for Large Language Models. Findings of ACL, 2025. ACL Anthology. [3] Jiwon Song et al. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks.ICML, 2024. arXiv:2402.09025. [4] Arthur Conmy et al. Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS, 2023. arXiv:2304.14997. [5] Adithya Bhaskar et al. Finding Transformer Circuits with Edge Pruning. NeurIPS, 2024. arXiv:2406.16778. [6] Jorge García-Carrasco, Alejandro Maté, and Juan Trujillo. Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference. arXiv preprint, 2024. arXiv:2412.15750. [7] Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What Matters in Transformers? Not All Attention is Needed. 2024. arXiv:2406.15786.

Questions for authors
  • The table-1 there is no clear advantage of CausalGate compared to the baseline methods, especially on QA tasks where the metric is given as accuracy. Could the authors deliver more details on how their method yields any benefit over these baselines?
  • What is the precise advance over prior intervention-based approaches, and how does CausalGate compare with FinerCut, BlockPruner, and SLEB?
  • What are the calibration time, peak memory usage, and intervention frequency for each model? How do these costs scale beyond 8B?
  • Does gate learning consistently outperform standalone averaged-KL ranking and iterative removal under matched calibration budgets?
Reviewer_9MeWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CausalGate, a framework for identifying and skipping less important Attention and MLP modules to accelerate LLM inference. Instead of relying on activation magnitude or hidden-state similarity, the authors estimate module importance by individually removing each module and measuring the resulting KL divergence in the model's output distribution.

The intervention-derived importance scores are then distilled into static scalar gates using EMA smoothing and a pairwise ranking loss. At inference time, modules with lower gate values are skipped according to a target compute budget.

The authors evaluate CausalGate on language modeling and commonsense reasoning benchmarks, comparing it with existing pruning and adaptive inference methods. They also report module-ranking agreement and inference speedups on several model configurations.

Critical strengths
  1. The motivation is reasonable. Small activation changes do not necessarily mean a module can be removed safely. Measuring the effect on the final output distribution is a more direct way to assess the consequences of removal.

  2. The final selection is static, which makes deployment straightforward. The hardware measurements confirm that module skipping produces actual speedups.

Critical weaknesses
  1. I find the practical trade-off unconvincing in the main results. At 5% and 10% removal, ShortGPT has lower perplexity on both WikiText-2 and C4. CausalGate is more competitive at moderate removal, but the absolute degradation is large: at 20% removal, TinyLlama's WikiText-2 perplexity rises from 13.58 to 67.00 for a measured 1.20× speedup. This makes it hard to identify a setting where the quality cost is attractive. The larger-model results help, but mostly report perplexity, so the broader capability-preservation claim has limited support.

  2. The benefit of distilling the intervention scores into gates seems narrower than the main presentation suggests. Appendix E is useful here: the learned ranking improves all six reported metrics at 10% removal, but results are mixed at 5%, and direct EMA ranking preserves perplexity better at 30–40%. There is therefore evidence that gate training helps in some regimes, but not that it is consistently preferable to the simpler ranking.

Relatedly, KL measures output-distribution change, which need not track changes in task accuracy. I would interpret it as a proxy for preserving model behavior, with its value established by the downstream results. Those results currently leave me unconvinced that the extra ranking procedure offers a sufficiently reliable advantage.

Questions for authors
  1. Which removal regime best demonstrates the practical advantage of CausalGate? In particular, how should we weigh the quality loss in Table 1 against the measured speedups in Table 2 and ShortGPT's better perplexity at low removal ratios?

  2. Given Appendix E, what benefit do you see as the main justification for gate distillation over direct EMA ranking? Please clarify whether the advantage is intended mainly for low-to-moderate removal budgets.

Reviewer_96MKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses LLM submodule pruning. The authors point out that previous methods rely on observational proxies, such as activation magnitudes or hidden-state similarity, which may fail to identify important modules. To address this limitation, the paper proposes CausalGate, which directly intervenes in individual modules rather than simply observing their behavior. Specifically, the method zeros out the outputs of attention and MLP modules and measures the resulting changes in the token output distribution.

The intervention scores are then used as targets to train learnable gates that estimate module importance and preserve the relative ranking of the scores. Based on the learned gate values, the method selects which modules to prune.

The proposed method achieves lower perplexity than several observation-based and learnable gating methods at certain pruning ratios.

Critical strengths
  • Unlike methods that rely on observational proxies, the proposed method directly measures the impact of removing individual attention and MLP modules on the model's output distribution. This provides a more direct estimate of module importance.
Critical weaknesses
  • The technical novelty and motivation for gate distillation are unclear. The idea of measuring module importance by removing individual modules and observing changes in the model's output has already been explored in previous works, such as FinerCut[1] and BlockPruner[2]. Therefore, the technical contribution of this importance estimation approach is limited. While distilling the intervention scores into learnable gates could be considered a novel component, the paper does not sufficiently explain why this additional training step is necessary. In particular, there is no comparison with directly using the intervention scores for module selection.

  • Comparisons with gradient-based importance estimation methods are missing. The paper emphasizes the limitations of observational proxies but does not discuss related approaches such as EAP-IG[3], which uses gradients to identify important model components.

  • The choice of sparsity regularization is insufficiently justified. The paper uses L2 regularization to encourage sparsity, although L1 regularization generally promotes sparse solutions more directly.

  • The proposed method does not consistently outperform previous methods on downstream tasks, even though it achieves lower perplexity at some pruning ratios. Furthermore, Table 1 only evaluates a single backbone model. Additional experiments on other backbones are needed to assess the generalizability of the proposed method.

  • The presentation needs improvement. The "Repeat for 44 modules" in Figure 1(b) is too small to read clearly, and the arrows overlap with the "Original Logits", making the figure difficult to follow.

[1] Y. Zhang et al., FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models, NeurIPS Workshop on Machine Learning and Compression, 2024

[2] L. Zhong et al., BlockPruner: Fine-grained Pruning for Large Language Models, Findings of ACL, 2025

[3] M. Hanna et al., Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms, COLM, 2024

Questions for authors
  • Is there a specific reason for using L2 regularization to encourage sparsity instead of L1 regularization?
  • This paper assumes that combining the intervention scores obtained by evaluating each module independently can lead to an optimal pruning solution. This assumption is common in layer pruning, but is there another way to optimize the module selection jointly, such as using beam search?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.