- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposed a finetune pipeline that prunes FFN hidden channels first and then finetunes on the downstream task. The proposed pruning method needs only one forward pass. The proposed "Select, Then Train (STT)" pipeline demonstrates a decent performance-efficiency trade-off curves.
Critical strengths
-
The proposed FFN pruning is very simple to implemented fast.
-
The paper demonstrate that for some downstream tasks, a small ratio of parameters is enough.
Critical weaknesses
-
The pruning method is basically under the same framework of some previous pruning methods, like KCM and FLAP, while the importance metric in this work is much simpler. The novelty of method is weak.
-
There is a lack of baselines in the experiments. More pruning methods, like KCM, FLAP and P-pruning, should be added. And Quantization-Aware Training (QAT), like QA-LoRA, EfficientQAT and GPLQ, can also improve efficiency for downstream tasks, which is lacked in comparison.
Gradient-Free Structured Pruning with Unlabeled DataPruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models
Fluctuation-Based Adaptive Structured Pruning for Large Language Models
Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models
Questions for authors
Are parameters in attention blocks trained after pruning?
3 comments
Reference:
[1] Azade Nova, Hanjun Dai, and Dale Schuurmans. Gradient-Free Structured Pruning with Unlabeled Data. Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 26326–26341, 2023.
[2] Pingjie Wang, Hongcheng Liu, Yanfeng Wang, and Yu Wang. Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13279–13289, 2024.
[3] Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based Adaptive Structured Pruning for Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[4] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2309.14717, 2023.
[5] Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10081–10100, 2025.
[6] Guang Liang, Xinyao Liu, and Jianxin Wu. GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers. Advances in Neural Information Processing Systems, volume 38, 2025.
Thank your for sharing results of comparing P-pruning, but what about KCM and FLAP?
STT selects a narrower FFN architecture from task inputs and fine-tunes the retained channels at the original precision. The narrower network can also be quantized, making these methods complementary in principle.
The above claim needs experiments to support. Both STT and quantization reduce the information of parameters. It is unclear that whether these methods could used together in practice, which needs experiments.
Thank you for highlighting these related methods and asking about the trainable parameters. We clarify the distinctions and add some direct comparisons below.
Comparison with prior work. STT selects FFN channels through the union of mean-activation and activation-frequency rankings (§3.2). This retains frequently activated channels even when their mean-activation rank falls below the cutoff. KCM [1] combines activation rankings with a kernel-based measure of weight representativeness. P-pruning [2] shares our task-specific, prune-before-fine-tuning workflow. It performs clustering on attention heads and FFN neurons by their activation similarity on task inputs, then retains the member with the highest average absolute activation in each cluster. STT instead ranks FFN channels directly, without clustering or restricting selection to one representative per cluster. A frequently activated channel can therefore survive through its frequency ranking even if it would not be selected as its cluster’s representative. FLAP [3] uses weight-scaled activation variance, standardizes scores to allocate pruning across layers and modules. STT’s distinguishing criterion is activation frequency combined with mean activation.
Quantization and empirical comparisons. QA-LoRA [4] combines low-rank adaptation with quantization, using group-wise operators to balance quantization and adaptation. EfficientQAT [5] trains all parameters block by block, then tunes quantization parameters end to end. GPLQ [6] quantizes ViT activations before weights. These methods reduce numerical precision, while STT reduces FFN width. STT selects a narrower FFN architecture from task inputs and fine-tunes the retained channels at the original precision. The narrower network can also be quantized, making these methods complementary in principle.
Trainable parameters. STT freezes attention and trains the retained FFNs and task head (Algorithm 1, line 9).
Additional Evaluation. We adapted P-pruning’s clustering and representative-selection criterion [2] to the gated FFNs in Qwen2.5-3B and matched STT’s width in every layer. Across 120 evaluations (two methods, four tasks, three sparsity levels, five seeds), STT had higher mean exact match in 8 of 12 task–sparsity settings. At sparsity level 0.6, STT scored higher in all five seeds on both ARC-C and OBQA, with mean gains of 5.53 and 9.12 percentage points, respectively.
Qwen2.5-3B
P-pruning comparison
Exact match (%, mean±SE, five seeds).
On full ImageNet-1k, STT had higher three-seed mean Top-1 accuracy than both Wanda Adapt and Magnitude Pruning at all four tested sparsity levels. Its lead over the stronger baseline at each level ranged from 2.17 to 7.40 percentage points.
CLIP ViT-B/32
ImageNet-1k
Top-1 validation accuracy (%, mean±SE, three seeds).
On GSM8K, STT had higher three-seed mean accuracy than both matched-width baselines at all nine tested sparsity levels. On HellaSwag, WinoGrande, SocialIQA, and PIQA, we have results for all three seeds in 31 of 36 settings. STT scored higher on average than both baselines in 29 of those 31.
Qwen2.5-1.5B
HellaSwag, WinoGrande, SocialIQA, and PIQA
Selected results at sparsity level 0.6 for all four tasks. Official validation accuracy (%, mean±SE, three seeds).
GSM8K (selected results)
Generated-answer test accuracy (%, mean±SE, three seeds).