- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ACM, a training-free attention-head pruning method motivated by “Support Knowledge” for domain-specific predictions. It combines attention-logit margins and activation magnitudes with probing-based layer-wise budget allocation. Experiments evaluate domain performance, calibration stability, and corpus transfer.
Critical strengths
-
The paper addresses domain-preserving head pruning without retraining. Its combination of head scoring and layer-wise budget allocation shows encouraging language-modeling and Qwen downstream results.
-
Component ablations, magnitude-controlled experiments, and calibration analyses help assess the method.
Critical weaknesses
-
Comparative advantages and cross-model robustness remain uncertain. FLAP is the only prior structured-pruning baseline; Appendix H omits its bias compensation, while its main-experiment configuration is unclear. Existing ablations do not compare scores under a common allocator or matched per-layer budgets.
-
Practical efficiency and total pruning costs are insufficiently evaluated. The paper reports modest input-processing speedups and parameter reductions on one model, without end-to-end generation latency, decoding throughput, or timing variability. The complete pipeline’s wall-clock time and peak memory remain unreported, leaving the overall resource tradeoff unclear.
Questions for authors
-
Baseline coverage and attribution: Could the authors include comparisons with recent structured-pruning methods, such as SlimLLM [1] or Týr-the-Pruner [2], or explain their applicability to this setting? Comparisons should account for differences in pruning units, calibration/search budgets, and weight reconstruction or compensation. Is FLAP’s original compensation used in the main experiments? How much of ACM’s advantage remains when alternative scores receive the same allocator or identical per-layer pruning budgets?
-
Cross-model robustness: What explains the substantial downstream reversals in Table 8, particularly the FormationEval result for Llama-3.1-8B at 30% pruning? Do the existing results support a narrower conclusion about the model families or pruning regimes where ACM is effective?
-
Efficiency: Can the authors report absolute prefill and decoding latency, end-to-end generation time at specified input/output lengths, and the hardware and measurement protocol? What are the wall-clock time and peak memory of the complete scoring-and-allocation pipeline relative to FLAP?
-
Calibration protocol and overhead: Please specify calibration/probing sizes and sample selection, and clarify whether QA evaluation questions and their answers or explanations were excluded. Appendix C describes language-modeling splits, but item-level separation for QA remains unclear. Please also report the wall-clock time, peak memory, and storage requirements of calibration and probing for ACM and competing baselines. Accounting for these upfront costs, under what deployment workload do the inference savings yield a net efficiency benefit?
[1] Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. “SlimLLM: Accurate Structured Pruning for Large Language Models.” ICML, 2025. https://proceedings.mlr.press/v267/guo25a.html
[2] Guanchen Li et al. “Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization.” NeurIPS, 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/db2cbf43a349bc866111e791b58c7bf4-Abstract-Conference.html
2 comments
Thank you for their time and effort. However, my main concerns remain insufficiently addressed; I therefore maintain my current rating.
Thank you for your careful review and helpful suggestions.
Method comparison and downstream performance. In the main experiments, FLAP uses a query-head adaptation of its fluctuation score, without the bias compensation used in the original method. As described in Section 4.1 (ref to line 307 of our manuscript), ACM* retains ACM scores but globally prunes the lowest-scoring heads to decouple head scoring from budget allocation. To compare the scoring methods, we select heads for pruning by globally ranking the scores from ACM* and FLAP separately. ACM* achieved both a lower PPL ratio and higher top-1 prediction agreement in 30 of 48 settings.
On FormationEval, Llama-3.1-8B achieved accuracies of 0.403 with our ACM and 0.564 with FLAP when 30% of query heads were pruned. On PetroBench, the corresponding accuracies were 0.497 and 0.290 for the same model and pruning rate. We have not separately analysed the cause of this difference. We thank the reviewer for recommending SlimLLM and Týr-the-Pruner, and will consider including them in comparisons in a future version.
Efficiency and calibration settings. We evaluated Llama-3.2-3B on an A800 with 20% of query heads pruned, a batch size of 1, and an input length of 256 tokens. Generation was fixed at 128 tokens. Each condition used 2 warm-up runs and 7 measured runs. The median generation times for the original model before and after the pruning experiments were 2.143 and 2.135 seconds, respectively. The corresponding times were 2.204 seconds for FLAP and 2.173 seconds for our ACM. The appendix also reports prefill speedups.
For the downstream QA, calibration questions were separated from evaluation questions. Calibration text was extracted only from the questions.