acceptodds
Under review as a conference paper at ICLR 2027

Select, Then Train: Efficient Transformer Adaptation Via Neuron Selection

Abstract

As pretrained transformers continue to scale, reducing both training and inference cost is critical for efficient downstream adaptation. On one hand, parameter-efficient fine-tuning methods reduce training overhead, but they generally do not reduce inference cost. On the other, most pruning methods focus on preserving task-agnostic pretrained behavior rather than post-adaptation performance. We study structured pruning (i.e., pruning neurons, not just weights) for downstream adaptation and show that simple activation statistics measured on a small unlabeled target-task calibration set provide an effective signal for identifying task-relevant neurons. In particular, we observe strong layer-wise complementarity between activation magnitude and activation frequency. Based on this observation, we propose Select, Then Train (STT), a lightweight select–then–train framework that removes under-utilized Feedforward Neural Network (FFN) neurons prior to adaptation, producing compact subnetworks optimized for task-specific performance and efficient inference. Across multiple Vision Transformers and Large Language Models, STT achieves better accuracy–efficiency trade-offs than magnitude-based and weight–activation pruning baselines, yielding average inference speedups of approximately 1.2× on LLMs and 1.7× on Vision Transformers across diverse configurations. These results demonstrate that activation-driven structured pruning is an effective and practical approach for task-specialized transformer adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
transformer adaptation, structured pruning, efficient fine-tuning, inference efficiency, neuron selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8FBKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a finetune pipeline that prunes FFN hidden channels first and then finetunes on the downstream task. The proposed pruning method needs only one forward pass. The proposed "Select, Then Train (STT)" pipeline demonstrates a decent performance-efficiency trade-off curves.

Critical strengths
  1. The proposed FFN pruning is very simple to implemented fast.

  2. The paper demonstrate that for some downstream tasks, a small ratio of parameters is enough.

Critical weaknesses
  1. The pruning method is basically under the same framework of some previous pruning methods, like KCM and FLAP, while the importance metric in this work is much simpler. The novelty of method is weak.

  2. There is a lack of baselines in the experiments. More pruning methods, like KCM, FLAP and P-pruning, should be added. And Quantization-Aware Training (QAT), like QA-LoRA, EfficientQAT and GPLQ, can also improve efficiency for downstream tasks, which is lacked in comparison.

Gradient-Free Structured Pruning with Unlabeled DataPruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models

Fluctuation-Based Adaptive Structured Pruning for Large Language Models

Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models

Questions for authors

Are parameters in attention blocks trained after pruning?

3 comments

Thank you for highlighting these related methods and asking about the trainable parameters. We clarify the distinctions and add some direct comparisons below.

Comparison with prior work. STT selects FFN channels through the union of mean-activation and activation-frequency rankings (§3.2). This retains frequently activated channels even when their mean-activation rank falls below the cutoff. KCM [1] combines activation rankings with a kernel-based measure of weight representativeness. P-pruning [2] shares our task-specific, prune-before-fine-tuning workflow. It performs clustering on attention heads and FFN neurons by their activation similarity on task inputs, then retains the member with the highest average absolute activation in each cluster. STT instead ranks FFN channels directly, without clustering or restricting selection to one representative per cluster. A frequently activated channel can therefore survive through its frequency ranking even if it would not be selected as its cluster’s representative. FLAP [3] uses weight-scaled activation variance, standardizes scores to allocate pruning across layers and modules. STT’s distinguishing criterion is activation frequency combined with mean activation.

Quantization and empirical comparisons. QA-LoRA [4] combines low-rank adaptation with quantization, using group-wise operators to balance quantization and adaptation. EfficientQAT [5] trains all parameters block by block, then tunes quantization parameters end to end. GPLQ [6] quantizes ViT activations before weights. These methods reduce numerical precision, while STT reduces FFN width. STT selects a narrower FFN architecture from task inputs and fine-tunes the retained channels at the original precision. The narrower network can also be quantized, making these methods complementary in principle.

Trainable parameters. STT freezes attention and trains the retained FFNs and task head (Algorithm 1, line 9).

Additional Evaluation. We adapted P-pruning’s clustering and representative-selection criterion [2] to the gated FFNs in Qwen2.5-3B and matched STT’s width in every layer. Across 120 evaluations (two methods, four tasks, three sparsity levels, five seeds), STT had higher mean exact match in 8 of 12 task–sparsity settings. At sparsity level 0.6, STT scored higher in all five seeds on both ARC-C and OBQA, with mean gains of 5.53 and 9.12 percentage points, respectively.

Qwen2.5-3B

P-pruning comparison

Exact match (%, mean±SE, five seeds).

TaskSparsity levelSTTP-pruning
ARC-C0.330.49±1.9131.54±2.79
ARC-C0.668.40±0.5162.87±0.54
ARC-C0.980.78±0.3379.39±0.25
OBQA0.349.96±9.0947.76±8.63
OBQA0.680.20±0.9571.08±0.72
OBQA0.986.52±0.3185.52±0.36
ARC-E0.362.66±2.9051.94±7.70
ARC-E0.673.88±12.0180.08±0.50
ARC-E0.978.06±13.0191.59±0.17
BoolQ0.364.67±0.3065.50±0.54
BoolQ0.680.84±0.1379.59±4.31
BoolQ0.986.76±0.1183.63±1.64

On full ImageNet-1k, STT had higher three-seed mean Top-1 accuracy than both Wanda Adapt and Magnitude Pruning at all four tested sparsity levels. Its lead over the stronger baseline at each level ranged from 2.17 to 7.40 percentage points.

CLIP ViT-B/32

ImageNet-1k

Top-1 validation accuracy (%, mean±SE, three seeds).

Sparsity levelSTTWanda AdaptMagnitude Pruning
0.153.30±0.0746.16±0.2445.58±0.18
0.260.70±0.0552.19±0.3653.30±0.34
0.364.39±0.1059.14±0.3060.45±0.11
0.466.98±0.0764.39±0.1564.81±0.13

On GSM8K, STT had higher three-seed mean accuracy than both matched-width baselines at all nine tested sparsity levels. On HellaSwag, WinoGrande, SocialIQA, and PIQA, we have results for all three seeds in 31 of 36 settings. STT scored higher on average than both baselines in 29 of those 31.

Qwen2.5-1.5B

HellaSwag, WinoGrande, SocialIQA, and PIQA

Selected results at sparsity level 0.6 for all four tasks. Official validation accuracy (%, mean±SE, three seeds).

DatasetSparsity levelSTTWanda AdaptMagnitude Pruning
HellaSwag0.679.01±0.0736.48±8.6225.29±0.24
WinoGrande0.671.82±0.6451.17±0.6148.93±0.33
SocialIQA0.673.47±0.4365.64±0.8563.75±0.25
PIQA0.677.86±0.9250.31±0.6249.75±0.07

GSM8K (selected results)

Generated-answer test accuracy (%, mean±SE, three seeds).

Sparsity levelSTTWanda AdaptMagnitude Pruning
0.321.13±0.5016.78±0.7216.48±0.22
0.529.37±0.8126.84±0.8526.26±0.14
0.739.10±0.4434.87±0.1832.65±0.07

Reference:

[1] Azade Nova, Hanjun Dai, and Dale Schuurmans. Gradient-Free Structured Pruning with Unlabeled Data. Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 26326–26341, 2023.

[2] Pingjie Wang, Hongcheng Liu, Yanfeng Wang, and Yu Wang. Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13279–13289, 2024.

[3] Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based Adaptive Structured Pruning for Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.

[4] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2309.14717, 2023.

[5] Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10081–10100, 2025.

[6] Guang Liang, Xinyao Liu, and Jianxin Wu. GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers. Advances in Neural Information Processing Systems, volume 38, 2025.

Thank your for sharing results of comparing P-pruning, but what about KCM and FLAP?

STT selects a narrower FFN architecture from task inputs and fine-tunes the retained channels at the original precision. The narrower network can also be quantized, making these methods complementary in principle.

The above claim needs experiments to support. Both STT and quantization reduce the information of parameters. It is unclear that whether these methods could used together in practice, which needs experiments.

Reviewer_woFdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Select, Then Train (STT), a finetuning framework combining with structured pruning for task specific adaptation of foundation models. STT will first run a single forward on a small calibration from a target task, then select the task specific weight columns/rows in FFN, prune the unselected parameters, and only finetune the selected parameters. The experiments are conduct on BoolQ, ARC-E/C and OBQA with Qwen2.5 1.5B/3B/7B (plus Gemma-3-1B in the appendix), EuroSAT, GTSRB and CLEVR-Count with ViTs (CLIP and SigLIP), and GLUE tasks in the appendix with BERTs. The paper compare STT with Wanda and Magnitude Pruning and demonstrate the accuracy–efficiency trade-off.

Critical strengths
  1. The paper presents the 95% CI across 5 seeds and conducts several ablation studies. Also, the paper reports the real wall-clock speedup, not just counting FLOPs.

  2. Instead of relying on Wanda style criteria, they derive an activation rate to measure importance of weight column and row in FFN for capturing target task information. I consider this to be the main new aspect of this paper.

Critical weaknesses
  1. My main concerns are novelty, positioning, and significance of this paper.

1.1 Starting with the motivation, CoFi[1] and Movement pruning have already explored the task specific concept and the paper doesn't cite and discuss them.

1.2 Then about the method itself, there are several works already studied the similar concept, such as [3], [4], [5], and [6]. Although some of them are working on finetuning instead of pruning, the idea behind the methods are quite similar. For example, [3] and [4] also run on a small calibration set to evaluate the important row/columns for some layers and finetune only the selected parameters. Moreover, the core idea behind [5] is very similar to STT, even the criterion is also similar.

1.3 The results are also not promising, in Figure 2, pruning the larger model to a smaller size performs significantly worse than full finetuning the smaller model. For instance, prune 7B to 3B (about 35%/40%/70%/40%) performs significantly worse than full finetuning 3B (about 80%/90%/80%/85%) model. Pruning 3B to 1.5B also shows the similar pattern. Given the results, why not just finetuning the smaller foundation models.

  1. My second concern is on evidence.

The evaluation benchmarks are limited and a little outdated. The paper only evaluates on 4 of 8 commonsense reasoning datasets and 3 vision benchmarks that are not sufficiently representative of broader vision adaptation tasks. Nowadays, evaluating only on these datasets is not sufficient.

For vision tasks, please at least include ImageNet-1k[7] or some other classical alternatives.

For language tasks, please at least include all of 8 commonsense reasoning tasks (HellaSwag[8], WinoGrande[9], SocialIQA[10], PIQA[11]) and some other settings. For instance, (1) math reasoning such as GSM8k[12] and AQuA[13], (2) instruction following such as MT-Bench[14] and IFEval[15], and (3) coding tasks such as MBPP[16], HumanEval[17], and EvalPlus (HumanEval+ / MBPP+) [18] (LiveCodeBench[19][20] can be more challenging alternative evaluation).

[1] Xia, Mengzhou, Zexuan Zhong, and Danqi Chen. "Structured pruning learns compact and accurate models." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

[2] Sanh, Victor, Thomas Wolf, and Alexander Rush. "Movement pruning: Adaptive sparsity by fine-tuning." Advances in neural information processing systems 33 (2020): 20378-20389.

[3] Yang, Xinyu, et al. "S $^{2} $ FT: Efficient, scalable and generalizable LLM fine-tuning by structured sparsity." Advances in Neural Information Processing Systems 37 (2024): 59912-59947.

[4] Li, Cen-Jhih, and Aditya Bhaskara. "An efficient sparse fine-tuning with low quantization error via neural network pruning." Transactions on Machine Learning Research (2025).

[5] Muralidharan, Saurav, et al. "Compact language models via pruning and knowledge distillation." Advances in Neural Information Processing Systems 37 (2024): 41076-41102.

[6] Chen, Tianlong, et al. "The lottery ticket hypothesis for pre-trained bert networks." Advances in neural information processing systems 33 (2020): 15834-15846.

[7] Russakovsky, Olga, et al. "Imagenet large scale visual recognition challenge." International journal of computer vision 115.3 (2015): 211-252.

[8] Zellers, Rowan, et al. "Hellaswag: Can a machine really finish your sentence?." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019.

[9] Sakaguchi, Keisuke, et al. "Winogrande: An adversarial winograd schema challenge at scale." Communications of the ACM 64.9 (2021): 99-106.

[10] Sap, Maarten, et al. "Social IQa: Commonsense reasoning about social interactions." Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019.

[11] Bisk, Yonatan, et al. "Piqa: Reasoning about physical commonsense in natural language." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020.

[12] Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021).

[13] Ling, Wang, et al. "Program induction by rationale generation: Learning to solve and explain algebraic word problems." Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers). 2017.

[14] Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in neural information processing systems 36 (2023): 46595-46623.

[15] Zhou, Jeffrey, et al. "Instruction-following evaluation for large language models." arXiv preprint arXiv:2311.07911 (2023).

[16] Austin, Jacob, et al. "Program synthesis with large language models." arXiv preprint arXiv:2108.07732 (2021).

[17] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

[18] Liu, Jiawei, et al. "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation." Advances in neural information processing systems 36 (2023): 21558-21572.

[19] Jain, Naman, et al. "Livecodebench: Holistic and contamination free evaluation of large language models for code." International Conference on Learning Representations. Vol. 2025. 2025.

[20] Zheng, Zihan, et al. "Livecodebench pro: How do olympiad medalists judge llms in competitive programming?." Advances in Neural Information Processing Systems 38 (2025).

Questions for authors
  1. In Figure 2, the pruned models seem to perform worse than fully finetuned smaller foundation models of a similar size. Did the authors study why? Could it be catastrophic forgetting[1][2]?
  2. Prior work[3][4] has shown that removing a large fraction of neurons may cause substantial capability loss, even at moderate sparsity, and such capability loss is hard to recover. Recovering it typically requires much more data than a downstream task dataset, e.g., 50B tokens of continued pre-training in [5]. In practice, quantization is often preferred over pruning [3][4]. Can the authors show in which situations STT has an advantage over these alternatives?

[1] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.

[2] Luo, Yun, et al. "An empirical study of catastrophic forgetting in large language models during continual fine-tuning." IEEE Transactions on Audio, Speech and Language Processing (2025).

[3] Jaiswal, Ajay, et al. "Compressing llms: The truth is rarely pure and never simple." International Conference on Learning Representations. Vol. 2024. 2024.

[4] Kuzmin, Andrey, et al. "Pruning vs quantization: Which is better?." Advances in neural information processing systems 36 (2023): 62414-62427.

[5] Xia, Mengzhou, et al. "Sheared llama: Accelerating language model pre-training via structured pruning." International Conference on Learning Representations. Vol. 2024. 2024.

4 comments

Thank you for pointing out the relevant prior work and suggesting broader evaluation. We address these points below.

Sparse updates and unstructured pruning. S²FT [1] and SPruFT [2] update selected parameters while retaining the full pretrained network; SPruFT uses W + MΔW (§4.1, Eq. (5)). Movement Pruning [3] scores weights using fine-tuning gradients, while BERT lottery tickets [4] use iterative magnitude pruning and reset surviving weights to their pretrained values. These latter methods produce weight-level sparse subnetworks. STT removes entire FFN channels, reducing the deployed network’s dimensions.

Structured pruning. CoFi [5] shares our task-specific objective, but uses gradient-based optimization to jointly learn pruning masks and model parameters, together with layerwise distillation. Minitron [6] combines activation-based width pruning with distillation to build smaller general-purpose LLMs. Sheared LLaMA [7] learns a specified target architecture and continues pretraining with dynamically adjusted data mixtures. STT fixes its FFN architecture before fine-tuning, using target-task inputs without mask optimization or teacher supervision. Its specific contribution is the union of mean-activation and activation-frequency selections: frequently activated channels survive even when their mean-activation rank falls below the cutoff. The retained FFNs are then adapted to the target task, consistent with our stated scope in the paper.

Comparison with other compression and efficiency methods. If a pretrained dense model is already available at the desired size, directly fine-tuning could be a strong alternative. Removing FFN channels can disrupt the pretrained network’s computational dynamics because the removed and retained channels were trained jointly. Jaiswal et al. [8] document capability losses after pruning, and Sheared LLaMA [7] uses 50B tokens of continued pretraining to recover general-purpose performance. Quantization [9] offers another way to reduce model size and can also be applied to the narrower FFNs.

STT studies what signals by examining the accuracy–efficiency trade-off when adapting a pretrained model to a specified downstream task under a fixed FFN width budget. We show why activation frequency could serve as an useful signal and complementary to mean activation value (Figure 5).

Additional Evaluation. We added GSM8K, HellaSwag, WinoGrande, SocialIQA, PIQA, and full ImageNet-1k. At matched per-layer widths, STT had higher three-seed mean accuracy than both Wanda Adapt and Magnitude Pruning at all nine sparsity levels on GSM8K and all four on ImageNet-1k. On the four new commonsense tasks, we have results for all three seeds in 31 of 36 settings. STT had higher mean accuracy than both baselines in 29 of those 31. On the four original benchmarks, STT had higher five-seed mean exact match in 8 of 12 settings compared with P-pruning [10], adapted to Qwen2.5-3B gated FFNs at matched widths.

Qwen2.5-1.5B

HellaSwag, WinoGrande, SocialIQA, and PIQA (%, mean±SE).

DatasetSparsity levelSTTWanda AdaptMagnitude Pruning
HellaSwag0.679.01±0.0736.48±8.6225.29±0.24
WinoGrande0.671.82±0.6451.17±0.6148.93±0.33
SocialIQA0.673.47±0.4365.64±0.8563.75±0.25
PIQA0.677.86±0.9250.31±0.6249.75±0.07

GSM8K (selected results)

Generated-answer test accuracy (%, mean±SE).

Sparsity levelSTTWanda AdaptMagnitude Pruning
0.321.13±0.5016.78±0.7216.48±0.22
0.529.37±0.8126.84±0.8526.26±0.14
0.739.10±0.4434.87±0.1832.65±0.07

CLIP ViT-B/32

ImageNet-1k

Top-1 validation accuracy (%, mean±SE).

Sparsity levelSTTWanda AdaptMagnitude Pruning
0.153.30±0.0746.16±0.2445.58±0.18
0.260.70±0.0552.19±0.3653.30±0.34
0.364.39±0.1059.14±0.3060.45±0.11
0.466.98±0.0764.39±0.1564.81±0.13

Qwen2.5-3B

P-pruning comparison on the original benchmarks

Exact match (%, mean±SE), with criterion [10] adapted to gated FFNs at matched sparsity.

TaskSparsity levelSTTP-pruning
ARC-C0.330.49±1.9131.54±2.79
ARC-C0.668.40±0.5162.87±0.54
ARC-C0.980.78±0.3379.39±0.25
OBQA0.349.96±9.0947.76±8.63
OBQA0.680.20±0.9571.08±0.72
OBQA0.986.52±0.3185.52±0.36
ARC-E0.362.66±2.9051.94±7.70
ARC-E0.673.88±12.0180.08±0.50
ARC-E0.978.06±13.0191.59±0.17
BoolQ0.364.67±0.3065.50±0.54
BoolQ0.680.84±0.1379.59±4.31
BoolQ0.986.76±0.1183.63±1.64

Thanks for the response and the additional experiment. Unfortunately, I think my main concern (the first concern) is still there.

I think maybe I need to clarify here. I agree that there are some differences between your work and some previous works, but my main concern is that the key contributions are a bit incremental, especially when the results are also not promising. Specifically, Figure 2 in the paper shows that full finetuning performs better than pruning a larger model to the same size. I would suggest that the paper demonstrates clearer and more concrete insights and contributions.

Also, I only see ARC-C/E, OBQA, and BoolQ instead of HellaSwag, WinoGrande, SocialIQA, and PIQA. Did the authors mistakenly report results on the same datasets already covered in the paper?

Reference:

[1] Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. S²FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity. arXiv preprint arXiv:2412.06289, 2024.

[2] Cen-Jhih Li and Aditya Bhaskara. An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning. Transactions on Machine Learning Research, 2025.

[3] Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement Pruning: Adaptive Sparsity by Fine-Tuning. arXiv preprint arXiv:2005.07683, 2020.

[4] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The Lottery Ticket Hypothesis for Pre-trained BERT Networks. arXiv preprint arXiv:2007.12223, 2020.

[5] Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured Pruning Learns Compact and Accurate Models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1513–1528, 2022.

[6] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact Language Models via Pruning and Knowledge Distillation. arXiv preprint arXiv:2407.14679, 2024.

[7] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. arXiv preprint arXiv:2310.06694, 2023.

[8] Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing LLMs: The Truth Is Rarely Pure and Never Simple. International Conference on Learning Representations, 2024.

[9] Andrey Kuzmin, Markus Nagel, Mart van Baalen, Arash Behboodi, and Tijmen Blankevoort. Pruning vs Quantization: Which Is Better? Advances in Neural Information Processing Systems, volume 36, pages 62414–62427, 2023.

[10] Pingjie Wang, Hongcheng Liu, Yanfeng Wang, and Yu Wang. Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13279–13289, 2024.

Reviewer_woFdOpenReview ↗

Response to the revised response

If the paper changes its claim to studying the signals for pruning a model under a fixed FFN budget, and restricts the pruning goal to only pruning the model to a target size where no pretrained model is available, then the paper may need to demonstrate curves similar to those in Figure 2, including the FFT results of the smaller pretrained models, for the broader evaluation on HellaSwag, WinoGrande, SocialIQA, PIQA, and GSM8K.

Also, I checked Figure 2 again. On the four tasks, the pruned 7B model only catches up with the FFT 3B model when it still has about 5–6B parameters, while the pruned 3B model only catches up with the FFT 1.5B model when it still has about 2.5–3B parameters. So even under the new claim, the evidence and significance are still concerning.

There is another issue in the new results. HellaSwag, WinoGrande, and PIQA are multiple-choice tasks with 4/2/2 choices, so accuracies of around 25/50/50% are close to random guessing. The baselines (Wanda Adapt and Magnitude Pruning) seem to crash on these tasks, so the paper may need to provide more informative baseline results. A similar issue also appears in Figure 2 for ARC-C/E and OBQA when the retention ratio kk is low.

Reviewer_7ru5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "select, then train" (STT), a structural pruning methods that could reduce both training and inference costs: combining the best of both world (training adaptation to different tasks, and inference cost optimization).

the authors have shown in STT could achieve average inference speedup of 1.2x on LLMS and 1.7x on Vision Transformers.

Critical strengths
  • the motivation of this paper is clear, and useful for current research and application of LLMs: current parameter-efficient finetuning methods fail to reduce inference cost, and regualar pruning preserves general rather than task-specific performance.

  • The proposed idea is highly practical and cheap.

  • the experimental results shown in the paper clearly show that SST could outperform baseline pruning methods with a strong accuracy-efficiency trade-offs. The generalization of SST is studied across a wide range of model sizes and different model families.

  • I particular like the fact that STT is robust in high sparsity regimes, without collapsing like many preuning methods.

Critical weaknesses

W1: the authors have shown pruning on FFNs is effective. Yet my concern is this framework exclusively targets on dense FFNs layers. this could be problematic and weakneaning the generalization claim: the experiments miss other core mechnisms of transformers such as attentions, and moe (which already utilize sparse FFNs and dynamic rounting).

W2: If i understand correctly, SST need to permanently prune activations before training, based on a small calibration set. This may make the resulting model highly prone to distribution shift. If the calibration subset fails to capture the edge-cases, the model will suffer capacity loss permanently.

W3: on evaluation, the authors chose s multiple-choice benchmarks like ARC-C and ARC-E, this could weaken the paper's claim since the most critical LLM capabilities is open-ended text generation (and coding and long-context reasoning).

Questions for authors

based on the weakness above, my main questions to authors are:

  1. how does STT translate to MoE arch?

  2. have you evaluated SST-pruned LLMs on open-ended generation tasks or mesure perplexity on held-out text?

  3. how sensitive is the final model's results to the size and composition of the calibration set?

  4. wonder if the author did similar analysis on attention module, as FFNs. for example ,to check out teh activation magnitude and frequency.

1 comment

Thank you for the questions on architectural scope, calibration and generation. We clarify each point below.

FFN scope and MoE. STT supports pruning different FFN architectures, such as SwiGLU [1], in both decoder-only and encoder-only Transformers through coupled channel slicing of standard linear layers (i.e., nn.Linear), as described in Eqs. (2)–(4). This operation preserves the FFN's input/output dimensions and can therefore be applied within MoE experts [2], making it structurally complementary to expert routing in principle.

Attention analysis. We focus on FFNs because their intermediate channels provide a common pruning unit across vision and language models: the same coupled slicing reduces model size while preserving each model's attention computation.

Calibration sensitivity. Appendix A.6 (Figs. 11–13) finds stable neuron selection and similar downstream accuracy when calibration size varies from 1% to 10%. This supports using a small unlabeled subset of the target-task data. STT targets task-specific adaptation, so calibration uses target-task inputs to select the FFN channels that are subsequently fine-tuned.

Additional results. On GSM8K [3], STT achieves higher mean generated-answer accuracy than both matched-width baselines across all nine sparsity levels below.

Qwen2.5-1.5B

GSM8K

Test accuracy (%, mean±SE, three seeds); matched widths.

Sparsity levelSTTWanda AdaptMagnitude Pruning
0.111.47±0.426.77±0.448.54±0.98
0.215.29±0.2212.89±0.2912.66±0.55
0.321.13±0.5016.78±0.7216.48±0.22
0.424.29±0.5622.64±0.3121.35±0.55
0.529.37±0.8126.84±0.8526.26±0.14
0.635.00±0.6530.68±0.3328.10±0.43
0.739.10±0.4434.87±0.1832.65±0.07
0.845.54±0.6139.65±0.3438.24±0.43
0.952.06±0.3147.18±0.5045.69±0.54

References:

[1] Noam Shazeer. GLU Variants Improve Transformer. arXiv preprint arXiv:2002.05202, 2020.

[2] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv preprint arXiv:1701.06538, 2017.

[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168, 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.