- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CausalGate, a method for pruning attention and MLP submodules of pretrained transformers. It quantifies importance by dropping individual modules and computing the KL divergence of the resulting predictive distribution. These scores are used to supervise static scalar gates via EMA smoothing and a pairwise ranking objective, along with language-modeling and sparsity losses. The learned ranking decides which modules are dropped at inference time. Experiments We mainly evaluate TinyLlama-1.1B and also provide additional perplexity results for Qwen2.5-3B and Llama-3.1-8B models and inference measurements for 1.1B and 3B models.
Critical strengths
- The method assesses the importance of modules by analyzing changes in the final predictions, hence capturing downstream effects that local activation-based metrics may not capture using the KL metric.
- Static module selection avoids input-dependent routing and online importance estimation.
- The paper demonstrates measured inference speedups on TinyLlama-1.1B and Qwen2.5-3B and evaluates both language-modeling perplexity and downstream task accuracy.
Critical weaknesses
-
The novelty is insufficiently distinguished from prior work. FinerCut [1] already evaluates separate attention/FFN removals using output discrepancies, including Jensen–Shannon divergence. BlockPruner [2] and SLEB [3] evaluate candidate removals using perplexity or negative log-likelihood and iteratively update their choices. KL-based approaches also exist: ACDC [4] evaluates edge ablations through output KL, Edge Pruning [5] optimizes differentiable masks using a KL-based objective, and task-specific circuit extraction [6] uses component ablations and KL changes to guide pruning. Although these methods differ in granularity and application, they undermine the broad characterization of previous work as relying on observational proxies. Separate attention/MLP pruning and heterogeneous redundancy across these modules were also studied in [7].
-
The calibration cost and scalability are not well evaluated. Section 4.1 describes the training steps and hardware. Calibration wall-clock time, GPU-hours, and peak memory usage are not reported. Algorithms 1–2 appear to recompute all single-module interventions at each training iteration, requiring roughly forward evaluations per intervention sweep, plus gradient computation. I can see this process getting more and more expensive as model size and number of modules increase. The paper shows perplexity experiments for 3B and 8B, but no evaluation beyond 8B or calibration-cost metrics to show practical scalability. This cost of calibration is not justified by speedups in inference alone.
-
It is not clear whether learning at the gate adds any benefit. Aggregate intervention scores directly over ranks. This produces a static mask with no routing at runtime. Appendix E shows a useful comparison. EMA-only ranking preserves perplexity better at 30-40% removal rate. Also, its EMA targets are from the gate-training run, so this does not test a stand-alone calibration procedure without gate optimization. Comparison against averaged-KL ranking and iterative removal under matched calibration budgets would better isolate the contribution.
-
The empirical gains are not consistent. ShortGPT achieves better WikiText-2 and C4 perplexity, HellaSwag accuracy, and WinoGrande accuracy at 10% removal. At 20% CausalGate reduces perplexity compared to ShortGPT, but its perplexity on WikiText-2 increases from 13.58 for the unpruned model to 67.00. These results support a conditional trade-off rather than a general superiority. Matching module removal counts does not necessarily translate to computational savings because the attention and MLP modules have different costs. It would be more informative to compare at matched measured latencies.
[1] Yang Zhang et al. FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models. 2024. arXiv:2405.18218. [2] Longguang Zhong et al. BlockPruner: Fine-grained Pruning for Large Language Models. Findings of ACL, 2025. ACL Anthology. [3] Jiwon Song et al. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks.ICML, 2024. arXiv:2402.09025. [4] Arthur Conmy et al. Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS, 2023. arXiv:2304.14997. [5] Adithya Bhaskar et al. Finding Transformer Circuits with Edge Pruning. NeurIPS, 2024. arXiv:2406.16778. [6] Jorge García-Carrasco, Alejandro Maté, and Juan Trujillo. Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference. arXiv preprint, 2024. arXiv:2412.15750. [7] Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What Matters in Transformers? Not All Attention is Needed. 2024. arXiv:2406.15786.
Questions for authors
- The table-1 there is no clear advantage of CausalGate compared to the baseline methods, especially on QA tasks where the metric is given as accuracy. Could the authors deliver more details on how their method yields any benefit over these baselines?
- What is the precise advance over prior intervention-based approaches, and how does CausalGate compare with FinerCut, BlockPruner, and SLEB?
- What are the calibration time, peak memory usage, and intervention frequency for each model? How do these costs scale beyond 8B?
- Does gate learning consistently outperform standalone averaged-KL ranking and iterative removal under matched calibration budgets?