- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a retraining-free approach to making pretrained Transformers elastic, allowing a single model to adapt to different computational budgets through structured pruning. The method introduces differentiable soft pruning to jointly optimize the importance ranking of individual MLP neurons and attention heads across multiple sparsity targets, while keeping the pretrained weights frozen. Experiments across 27 vision and text Transformer encoders demonstrate improved accuracy–compute trade-offs over existing pruning methods, particularly at high sparsity levels. The authors further evaluate elastic retrieval with a fixed dense index and report practical GPU throughput, latency, and budget-switching overhead, demonstrating the potential of the proposed approach for flexible deployment.
Critical strengths
-
Flexible inference without retraining. The proposed method allows pretrained Transformers to dynamically adapt to different sparsity levels at inference time without requiring additional weight training or re-optimization of importance rankings.
-
Practical elastic deployment. The elastic deployment experiments demonstrate that using different model sparsity levels for offline data processing and online inference does not necessarily cause substantial performance degradation, further supporting the practical applicability of the proposed method.
-
Comprehensive efficiency evaluation. The authors provide concrete measurements of GPU throughput, inference latency, and the additional overhead associated with switching between different sparsity budgets.
Critical weaknesses
-
Limited methodological novelty. The proposed method largely builds upon established ideas in differentiable structured pruning and multi-budget optimization. Its algorithmic novelty appears relatively limited, with the primary contributions lying in applying these techniques at a finer granularity to pretrained Transformers and providing extensive empirical validation.
-
Sensitivity to calibration data. The optimized importance rankings appear sensitive to the calibration distribution, as evidenced by the substantial improvements achieved through task-aligned calibration. This raises concerns about the robustness of the learned rankings under distribution shifts, particularly when deployment workloads differ significantly from the calibration data.
Questions for authors
-
Optimization stability and convergence. I would appreciate additional analysis of the optimization process, including optimization loss curves, importance-ranking convergence trajectories, and training stability across different runs. In particular, the sigmoid-based soft gates may produce highly concentrated gradients near the pruning thresholds, potentially making the optimization sensitive to score changes and initialization. Could the authors provide further evidence that the optimization converges reliably across different models and random seeds?
-
Soft-to-hard pruning discrepancy. During importance-ranking optimization, soft gates allow neurons with low importance scores to retain nonzero contributions to the model output, whereas these neurons are completely removed by hard pruning during deployment. This introduces a potential mismatch between the optimization and inference settings. Could the authors quantify this soft-to-hard discretization gap? In particular, reporting benchmark results using soft masks alongside the corresponding hard-pruned models at matched sparsity levels would help clarify the impact of this discrepancy.
11 comments
Finally, we believe that a performance improvement due to task-aligned calibration is desirable: when the deployment domain is narrow and known in advance, in-domain calibration yields substantially stronger pruned models, e.g., +26.6 points on Flowers 102 at 40% sparsity, at the cost of a few minutes of re-optimization.
ImageNet-1k
| Method | Correction | Dense | 10% / 31.7 G | 20% / 28.2 G | 30% / 24.8 G | 40% / 21.2 G | 50% / 17.8 G | 60% / 14.2 G |
|---|---|---|---|---|---|---|---|---|
| SnapViT | ✗ | 83.6 | 82.2 | 77.4 | 70.0 | 57.7 | 28.4 | 17.7 |
| SparseGPT | ✗ | 83.6 | 78.8 | 64.2 | 36.5 | 20.3 | 14.1 | 10.2 |
| FPTP | ✗ | 83.6 | 82.0 | 78.5 | 71.3 | 44.2 | 17.2 | 12.6 |
| OPTIN | ✗ | 83.6 | 80.9 | 74.7 | 65.6 | 44.7 | 22.0 | 15.0 |
| VBP | ✓ | 83.6 | 82.6 | 81.1 | 74.3 | 51.4 | 25.5 | 13.2 |
| Ours | ✗ | 83.6 | 82.6 | 80.1 | 75.6 | 69.1 | 59.4 | 45.2 |
Top-1 accuracy in k-NN classification on ImageNet-1k using AugReg ViT-B/16 models pruned using our method and the baselines.
ImageNet-V2
| Method | Correction | Dense | 10% / 31.7 G | 20% / 28.2 G | 30% / 24.8 G | 40% / 21.2 G | 50% / 17.8 G | 60% / 14.2 G |
|---|---|---|---|---|---|---|---|---|
| SnapViT | ✗ | 73.0 | 71.8 | 65.8 | 56.8 | 43.4 | 18.9 | 10.7 |
| SparseGPT | ✗ | 73.0 | 66.7 | 49.9 | 24.6 | 12.8 | 7.8 | 5.8 |
| FPTP | ✗ | 73.0 | 71.0 | 66.8 | 58.8 | 32.3 | 10.6 | 7.2 |
| OPTIN | ✗ | 73.0 | 69.9 | 62.6 | 52.3 | 33.1 | 14.8 | 8.5 |
| VBP | ✓ | 73.0 | 71.9 | 70.1 | 62.1 | 39.9 | 18.0 | 8.7 |
| Ours | ✗ | 73.0 | 72.3 | 69.2 | 64.1 | 56.5 | 46.2 | 33.3 |
Top-1 accuracy in k-NN classification, with the index fitted on the ImageNet-1k training set, on ImageNet-V2 using AugReg ViT-B/16 models pruned using our method and the baselines.
ImageNet Sketch
| Method | Correction | Dense | 10% / 31.7 G | 20% / 28.2 G | 30% / 24.8 G | 40% / 21.2 G | 50% / 17.8 G | 60% / 14.2 G |
|---|---|---|---|---|---|---|---|---|
| SnapViT | ✗ | 43.1 | 40.1 | 32.6 | 22.3 | 11.6 | 4.1 | 1.6 |
| SparseGPT | ✗ | 43.1 | 29.8 | 12.4 | 3.9 | 1.6 | 1.0 | 0.8 |
| FPTP | ✗ | 43.1 | 38.5 | 33.7 | 29.5 | 11.3 | 2.0 | 1.1 |
| OPTIN | ✗ | 43.1 | 38.2 | 26.8 | 19.5 | 10.5 | 3.0 | 1.3 |
| VBP | ✓ | 43.1 | 41.3 | 39.7 | 31.5 | 14.6 | 5.0 | 1.4 |
| Ours | ✗ | 43.1 | 40.3 | 35.9 | 27.0 | 20.5 | 12.3 | 6.1 |
Top-1 accuracy in k-NN classification, with the index fitted on the ImageNet-1k training set, on ImageNet Sketch using AugReg ViT-B/16 models pruned using our method and the baselines.
ImageNet-R
| Method | Correction | Dense | 10% / 31.7 G | 20% / 28.2 G | 30% / 24.8 G | 40% / 21.2 G | 50% / 17.8 G | 60% / 14.2 G |
|---|---|---|---|---|---|---|---|---|
| SnapViT | ✗ | 55.3 | 52.5 | 41.4 | 31.0 | 22.2 | 8.9 | 5.6 |
| SparseGPT | ✗ | 55.3 | 42.5 | 24.5 | 10.5 | 6.2 | 4.5 | 3.8 |
| FPTP | ✗ | 55.3 | 49.0 | 43.7 | 39.3 | 20.0 | 6.6 | 4.5 |
| OPTIN | ✗ | 55.3 | 49.0 | 35.9 | 27.5 | 16.5 | 7.0 | 4.9 |
| VBP | ✓ | 55.3 | 53.2 | 51.9 | 42.8 | 27.0 | 12.2 | 5.2 |
| Ours | ✗ | 55.3 | 52.9 | 47.8 | 42.1 | 32.9 | 24.9 | 17.4 |
Top-1 accuracy in k-NN classification, with the index fitted on the ImageNet-1k training set, on ImageNet-R using AugReg ViT-B/16 models pruned using our method and the baselines. As ImageNet-R only contains 200 classes, we filter the ImageNet-1k training set to only those classes.
Optimization stability and convergence. I would appreciate additional analysis of the optimization process, including optimization loss curves, importance-ranking convergence trajectories, and training stability across different runs. In particular, the sigmoid-based soft gates may produce highly concentrated gradients near the pruning thresholds, potentially making the optimization sensitive to score changes and initialization. Could the authors provide further evidence that the optimization converges reliably across different models and random seeds?
We measured the optimization stability across vision and text models, and report, over 50 step intervals, the loss, the Spearman correlation of MLP neuron and attention head rankings between the current step and the 50 steps ago snapshot, and the percentage of kept units changed since the snapshot. We consider AugReg models of increasing sizes, namely ViT-S/16, ViT-B/16 and ViT-L/16, models from the AugReg, DINOv3 and SigLIPv2 families, and both vision and text encoders, including E5 B and the BGE S, B, and L. For each entry, we report the mean across three seeds and the sample standard deviation.
We observe that losses reliably converge across seeds, but the results suggest that some models could benefit from further optimization, e.g. AugReg ViT-L/16 and SigLIPv2, whose loss is still decreasing by step 300. Convergence is further supported by the Spearman correlation across past and current rankings, as by step 300 Spearman correlations are always above 0.99 for MLP neurons and 0.90 for attention heads. Interestingly, the attention head rankings for AugReg ViT-S/16 and ViT-L/16 seem to not have settled, further indicating more steps might lead to even better performance. Finally, the percentage of the keep set changing for a 40% sparse model over the optimization trajectory further corroborates convergence.
We also report the top-1 accuracy in k-NN on ImageNet-1k for the pruned vision models. We observe that pruned models perform similarly well across seeds, with the standard deviation increasing at high sparsity, to up to 1.2% for AugReg ViT-S/16 at 50% sparsity. Nonetheless, we find performance to be consistent across seeds. We do not report performance results for text encoders due to time constraints, as their evaluation is expensive.
Regarding concentrated gradients, optimizing multiple targets spreads them across the ranking: at initialization, 99.1% of the units in the sparsity range receive at least half of the maximum gradient from their closest sigmoid, as discussed in our response to reviewer v4ZC. Regarding initialization, Table 2 of the paper shows that our optimization recovers from poor initial scores, e.g., lifting Wanda from 7.2% to 77.8% at 20% sparsity, within 2.3 points of our default, although a strong initialization remains beneficial under deep pruning.
The batch size used to estimate the initial importance scores for DINOv3 ViT-B was 8, versus a default of 16 due to GPU availability.
We thank the reviewer for the insightful comments. We first summarize our responses to each raised feedback item, which we then discuss in detail:
1ffB, these components are complementary: their gains accumulate under deep pruning, with the full recipe improving top-1 accuracy at 50% sparsity from 28.0% to 59.4%.We appreciate the reviewer's comment. We agree that core ingredients for our method such as differentiable structured pruning and multi-budget optimization are established in the literature. We believe that our core contribution is a formulation that learns the ranking of functional units itself, rather than inheriting it from initial importance scores, enabling effective training-free elasticity across a broad range of transformers, spanning sizes, pretraining objectives, and modalities, with only small differences for the text and image modalities.
In terms of algorithmic contributions, our method proposes modeling scores for individual functional units, as also recognized by the reviewer, using differentiable soft pruning to optimize a ranking of functional units, rather than to produce a single sparse model, and ranking MLP neurons and attention heads on a common scale, enabling users to specify a target sparsity and obtain a model whose sparsity is automatically allocated across neurons and attention heads, rather than hand-tuning the allocation as in SnapViT or pruning layers uniformly as in SparseGPT.
We further discuss the novelty of our work in the shared response.
We appreciate the reviewer raising this point. To test this, we prune AugReg ViT-B/16 models using our method and the baselines, fit a k-NN index using the ImageNet-1k training split, and report the top-1 accuracy on the validation split of ImageNet-1k, V2, Sketch, and R. For ImageNet-R, we filter the k-NN index to the 200 represented classes.
Pruning degrades out of distribution performance more aggressively for both our method and the baselines. In terms of absolute accuracy, we find that we outperform every baseline from 40% sparsity onwards, and we are either the best or second best at every sparsity, except on ImageNet Sketch at 30% sparsity.
In addition, in the paper we evaluate ImageNet-calibrated rankings on nine transfer datasets spanning different domains and label spaces. On AugReg ViT-B/16, our method generally matches or outperforms the baselines under moderate and deep pruning, with the exception of EuroSAT. We observe similar trends for DINOv3 ViT-H+/16 and SigLIPv2 ViT-G/16 from 40% and 30% sparsity onwards, respectively.
Furthermore, in the paper, we optimize the unit ranking for text encoders using samples from MS MARCO, a passage retrieval dataset. Nonetheless, pruned encoders perform strongly in MTEB, whose 41 tasks cover classification, clustering, pair classification, retrieval, reranking, semantic textual similarity, and summarization.