acceptodds
Under review as a conference paper at ICLR 2027

Your Transformer Is Secretly Elastic

Abstract

Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once. Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining. At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to higher throughput and lower single-query latency on an A100 GPU. Elastic models are produced in minutes, with construction time scaling sub-linearly with parameter count.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
transformer, pruning, elastic, retrieval, classification, segmentation, inference, text embedding, text retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nZgcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a retraining-free approach to making pretrained Transformers elastic, allowing a single model to adapt to different computational budgets through structured pruning. The method introduces differentiable soft pruning to jointly optimize the importance ranking of individual MLP neurons and attention heads across multiple sparsity targets, while keeping the pretrained weights frozen. Experiments across 27 vision and text Transformer encoders demonstrate improved accuracy–compute trade-offs over existing pruning methods, particularly at high sparsity levels. The authors further evaluate elastic retrieval with a fixed dense index and report practical GPU throughput, latency, and budget-switching overhead, demonstrating the potential of the proposed approach for flexible deployment.

Critical strengths
  1. Flexible inference without retraining. The proposed method allows pretrained Transformers to dynamically adapt to different sparsity levels at inference time without requiring additional weight training or re-optimization of importance rankings.

  2. Practical elastic deployment. The elastic deployment experiments demonstrate that using different model sparsity levels for offline data processing and online inference does not necessarily cause substantial performance degradation, further supporting the practical applicability of the proposed method.

  3. Comprehensive efficiency evaluation. The authors provide concrete measurements of GPU throughput, inference latency, and the additional overhead associated with switching between different sparsity budgets.

Critical weaknesses
  1. Limited methodological novelty. The proposed method largely builds upon established ideas in differentiable structured pruning and multi-budget optimization. Its algorithmic novelty appears relatively limited, with the primary contributions lying in applying these techniques at a finer granularity to pretrained Transformers and providing extensive empirical validation.

  2. Sensitivity to calibration data. The optimized importance rankings appear sensitive to the calibration distribution, as evidenced by the substantial improvements achieved through task-aligned calibration. This raises concerns about the robustness of the learned rankings under distribution shifts, particularly when deployment workloads differ significantly from the calibration data.

Questions for authors
  1. Optimization stability and convergence. I would appreciate additional analysis of the optimization process, including optimization loss curves, importance-ranking convergence trajectories, and training stability across different runs. In particular, the sigmoid-based soft gates may produce highly concentrated gradients near the pruning thresholds, potentially making the optimization sensitive to score changes and initialization. Could the authors provide further evidence that the optimization converges reliably across different models and random seeds?

  2. Soft-to-hard pruning discrepancy. During importance-ranking optimization, soft gates allow neurons with low importance scores to retain nonzero contributions to the model output, whereas these neurons are completely removed by hard pruning during deployment. This introduces a potential mismatch between the optimization and inference settings. Could the authors quantify this soft-to-hard discretization gap? In particular, reporting benchmark results using soft masks alongside the corresponding hard-pruned models at matched sparsity levels would help clarify the impact of this discrepancy.

11 comments

We thank the reviewer for the insightful comments. We first summarize our responses to each raised feedback item, which we then discuss in detail:

  • Novelty. We agree that the individual ingredients of our method are established in the literature. Our contribution is a formulation that integrates them to learn the ranking of functional units itself, rather than rescale a fixed one: per-unit scores, a differentiable relaxation of the ranking that can be optimized with any differentiable loss, and a joint ranking of MLP neurons and attention heads on a common scale. As discussed in the shared response and in detail in our response to reviewer 1ffB, these components are complementary: their gains accumulate under deep pruning, with the full recipe improving top-1 accuracy at 50% sparsity from 28.0% to 59.4%.
  • Calibration data. We outperform every baseline on ImageNet V2, Sketch and R from 40% sparsity onwards, and are the best or second best at lower sparsity, except on Sketch at 30%. Furthermore, text encoders calibrated only on MS MARCO, a passage retrieval dataset, perform strongly on MTEB, whose 41 tasks also cover classification, clustering, semantic similarity, and summarization.
  • Stability. Losses and rankings reliably converge across three seeds for all nine models we tested. Performance is also consistent across seeds, with a standard deviation of at most 1.2% in top-1 accuracy.
  • Soft-to-hard gap. The soft relaxation closely matches the hard-pruned models used at deployment, with gaps of at most 2.7%.

Limited methodological novelty. The proposed method largely builds upon established ideas in differentiable structured pruning and multi-budget optimization. Its algorithmic novelty appears relatively limited, with the primary contributions lying in applying these techniques at a finer granularity to pretrained Transformers and providing extensive empirical validation.

We appreciate the reviewer's comment. We agree that core ingredients for our method such as differentiable structured pruning and multi-budget optimization are established in the literature. We believe that our core contribution is a formulation that learns the ranking of functional units itself, rather than inheriting it from initial importance scores, enabling effective training-free elasticity across a broad range of transformers, spanning sizes, pretraining objectives, and modalities, with only small differences for the text and image modalities.

In terms of algorithmic contributions, our method proposes modeling scores for individual functional units, as also recognized by the reviewer, using differentiable soft pruning to optimize a ranking of functional units, rather than to produce a single sparse model, and ranking MLP neurons and attention heads on a common scale, enabling users to specify a target sparsity and obtain a model whose sparsity is automatically allocated across neurons and attention heads, rather than hand-tuning the allocation as in SnapViT or pruning layers uniformly as in SparseGPT.

We further discuss the novelty of our work in the shared response.

Sensitivity to calibration data. The optimized importance rankings appear sensitive to the calibration distribution, as evidenced by the substantial improvements achieved through task-aligned calibration. This raises concerns about the robustness of the learned rankings under distribution shifts, particularly when deployment workloads differ significantly from the calibration data.

We appreciate the reviewer raising this point. To test this, we prune AugReg ViT-B/16 models using our method and the baselines, fit a k-NN index using the ImageNet-1k training split, and report the top-1 accuracy on the validation split of ImageNet-1k, V2, Sketch, and R. For ImageNet-R, we filter the k-NN index to the 200 represented classes.

Pruning degrades out of distribution performance more aggressively for both our method and the baselines. In terms of absolute accuracy, we find that we outperform every baseline from 40% sparsity onwards, and we are either the best or second best at every sparsity, except on ImageNet Sketch at 30% sparsity.

In addition, in the paper we evaluate ImageNet-calibrated rankings on nine transfer datasets spanning different domains and label spaces. On AugReg ViT-B/16, our method generally matches or outperforms the baselines under moderate and deep pruning, with the exception of EuroSAT. We observe similar trends for DINOv3 ViT-H+/16 and SigLIPv2 ViT-G/16 from 40% and 30% sparsity onwards, respectively.

Furthermore, in the paper, we optimize the unit ranking for text encoders using samples from MS MARCO, a passage retrieval dataset. Nonetheless, pruned encoders perform strongly in MTEB, whose 41 tasks cover classification, clustering, pair classification, retrieval, reranking, semantic textual similarity, and summarization.

Finally, we believe that a performance improvement due to task-aligned calibration is desirable: when the deployment domain is narrow and known in advance, in-domain calibration yields substantially stronger pruned models, e.g., +26.6 points on Flowers 102 at 40% sparsity, at the cost of a few minutes of re-optimization.

ImageNet-1k

MethodCorrectionDense10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
SnapViT✗83.682.277.470.057.728.417.7
SparseGPT✗83.678.864.236.520.314.110.2
FPTP✗83.682.078.571.344.217.212.6
OPTIN✗83.680.974.765.644.722.015.0
VBP✓83.682.681.174.351.425.513.2
Ours✗83.682.680.175.669.159.445.2

Top-1 accuracy in k-NN classification on ImageNet-1k using AugReg ViT-B/16 models pruned using our method and the baselines.

ImageNet-V2

MethodCorrectionDense10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
SnapViT✗73.071.865.856.843.418.910.7
SparseGPT✗73.066.749.924.612.87.85.8
FPTP✗73.071.066.858.832.310.67.2
OPTIN✗73.069.962.652.333.114.88.5
VBP✓73.071.970.162.139.918.08.7
Ours✗73.072.369.264.156.546.233.3

Top-1 accuracy in k-NN classification, with the index fitted on the ImageNet-1k training set, on ImageNet-V2 using AugReg ViT-B/16 models pruned using our method and the baselines.

ImageNet Sketch

MethodCorrectionDense10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
SnapViT✗43.140.132.622.311.64.11.6
SparseGPT✗43.129.812.43.91.61.00.8
FPTP✗43.138.533.729.511.32.01.1
OPTIN✗43.138.226.819.510.53.01.3
VBP✓43.141.339.731.514.65.01.4
Ours✗43.140.335.927.020.512.36.1

Top-1 accuracy in k-NN classification, with the index fitted on the ImageNet-1k training set, on ImageNet Sketch using AugReg ViT-B/16 models pruned using our method and the baselines.

ImageNet-R

MethodCorrectionDense10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
SnapViT✗55.352.541.431.022.28.95.6
SparseGPT✗55.342.524.510.56.24.53.8
FPTP✗55.349.043.739.320.06.64.5
OPTIN✗55.349.035.927.516.57.04.9
VBP✓55.353.251.942.827.012.25.2
Ours✗55.352.947.842.132.924.917.4

Top-1 accuracy in k-NN classification, with the index fitted on the ImageNet-1k training set, on ImageNet-R using AugReg ViT-B/16 models pruned using our method and the baselines. As ImageNet-R only contains 200 classes, we filter the ImageNet-1k training set to only those classes.

Optimization stability and convergence. I would appreciate additional analysis of the optimization process, including optimization loss curves, importance-ranking convergence trajectories, and training stability across different runs. In particular, the sigmoid-based soft gates may produce highly concentrated gradients near the pruning thresholds, potentially making the optimization sensitive to score changes and initialization. Could the authors provide further evidence that the optimization converges reliably across different models and random seeds?

We measured the optimization stability across vision and text models, and report, over 50 step intervals, the loss, the Spearman correlation of MLP neuron and attention head rankings between the current step and the 50 steps ago snapshot, and the percentage of kept units changed since the snapshot. We consider AugReg models of increasing sizes, namely ViT-S/16, ViT-B/16 and ViT-L/16, models from the AugReg, DINOv3 and SigLIPv2 families, and both vision and text encoders, including E5 B and the BGE S, B, and L. For each entry, we report the mean across three seeds and the sample standard deviation.

We observe that losses reliably converge across seeds, but the results suggest that some models could benefit from further optimization, e.g. AugReg ViT-L/16 and SigLIPv2, whose loss is still decreasing by step 300. Convergence is further supported by the Spearman correlation across past and current rankings, as by step 300 Spearman correlations are always above 0.99 for MLP neurons and 0.90 for attention heads. Interestingly, the attention head rankings for AugReg ViT-S/16 and ViT-L/16 seem to not have settled, further indicating more steps might lead to even better performance. Finally, the percentage of the keep set changing for a 40% sparse model over the optimization trajectory further corroborates convergence.

We also report the top-1 accuracy in k-NN on ImageNet-1k for the pruned vision models. We observe that pruned models perform similarly well across seeds, with the standard deviation increasing at high sparsity, to up to 1.2% for AugReg ViT-S/16 at 50% sparsity. Nonetheless, we find performance to be consistent across seeds. We do not report performance results for text encoders due to time constraints, as their evaluation is expensive.

Regarding concentrated gradients, optimizing multiple targets spreads them across the ranking: at initialization, 99.1% of the units in the [0.05,0.70][0.05, 0.70] sparsity range receive at least half of the maximum gradient from their closest sigmoid, as discussed in our response to reviewer v4ZC. Regarding initialization, Table 2 of the paper shows that our optimization recovers from poor initial scores, e.g., lifting Wanda from 7.2% to 77.8% at 20% sparsity, within 2.3 points of our default, although a strong initialization remains beneficial under deep pruning.

The batch size used to estimate the initial importance scores for DINOv3 ViT-B was 8, versus a default of 16 due to GPU availability.

Reviewer_1ffBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a retraining-free approach for converting pretrained Transformer encoders into elastic models that support multiple inference-time computational budgets. Building on SnapViT, the method introduces differentiable soft pruning to jointly optimize individual importance scores for MLP neurons and attention heads across multiple sparsity targets, while keeping the pretrained model weights frozen. The authors evaluate the approach on 27 vision and text encoders, reporting substantial improvements over existing pruning baselines, particularly at high sparsity levels. The paper also investigates dense-index compatibility, inference acceleration, and several important design choices through extensive ablations.

Critical strengths
  1. The method consistently outperforms the evaluated baselines at moderate and high sparsity levels. Results on large models, especially DINOv3 ViT-H+/16, are particularly impressive.
  2. Table 1 provides a valuable controlled comparison across different score parameterizations. The substantial improvement from SnapViT-style block-level scaling to per-neuron optimization supports the importance of fine-grained ranking flexibility.
  3. The paper covers multiple pretrained model families, vision and text tasks, model scales, and pruning budgets. The additional experiments on initialization, ranking granularity, calibration losses, weight correction, and robustness strengthen the empirical evidence.
  4. The dense-index retrieval setting and measured inference speedups go beyond conventional pruning evaluations and address meaningful deployment considerations.
Critical weaknesses
  1. Although the paper emphasizes adapting model computation dynamically to changing workloads, the actual latency and throughput measurements are limited to AugReg ViT-B/16. More importantly, Appendix A.4 shows that the reported inference improvements depend substantially on compilation and hardware-friendly pruning constraints. For example, at batch size 1, pruning provides essentially no speedup under standard compilation, while the reported 1.26× acceleration requires reduce-overhead compilation and GPU-friendly tiling. The latter also incurs an accuracy reduction at high sparsity. Furthermore, compiled graphs must be cached separately for different budgets, and compiling a previously unseen budget requires 12–49 seconds. The reported switching overhead excludes this compilation cost. These results suggest that the method provides an effective nested pruning ranking, but its ability to support flexible, low-overhead inference over a broad range of budgets remains less fully demonstrated. The paper would benefit from a more complete analysis of accuracy, latency, memory consumption, and compilation overhead across multiple budgets, ideally including larger text encoders.

  2. Table 3 suggests that joint optimization across 14 sparsity targets outperforms separate single-target optimization. However, each multi-budget optimization step evaluates 14 masked configurations, whereas single-target optimization requires only one. If both approaches are trained for the same number of steps, the multi-budget method receives substantially more optimization computation. Consequently, the reported improvements do not clearly distinguish the benefits of cross-budget optimization from those of additional compute.

  3. The paper primarily combines the global ranking-based elastic pruning formulation of SnapViT with established differentiable masking techniques, extending the optimization variables to individual neurons and attention heads. Although this change produces substantial empirical improvements, the underlying formulation is relatively straightforward. In particular, sigmoid-based masking, gradient optimization of importance scores, and averaging objectives over multiple budgets are natural extensions of existing ideas. The authors could strengthen the methodological contribution by providing a deeper analysis of why fine-grained multi-budget optimization works, beyond showing that finer parameterizations improve accuracy.

Questions for authors
  1. Does multi-budget optimization still outperform single-target optimization when controlling for total forward/backward compute?
  2. What is the total memory overhead of supporting multiple compiled sparsity configurations for large text encoders? How does the switching mechanism scale beyond ViT-B/16?
  3. Can the authors provide a more systematic comparison separating the effects of per-unit parameterization, calibration loss, and joint neuron/head ranking?
5 comments

We thank the reviewer for the insightful comments and the support for our work. We first summarize our responses to each raised feedback item, which we then discuss in detail:

  • Speedups. Larger models can be further accelerated, by up to 2.19x in throughput and, under reduce-overhead compilation, 1.98x in batch-1 latency. GPU-friendly tiling has effectively no performance impact on large encoders.
  • Compute-matched optimization. Under matched compute, optimizing 3 or 7 budgets still outperforms single-budget optimization, whose rankings degrade away from their target.
  • Why multi-budget optimization works. Rankings optimized for too few budgets over-prune attention heads away from them, degrading performance by up to 18.2%.
  • Memory and switching. Compared to the dense model, supporting 7 budgets adds at most 92MB of cached graphs, i.e., a 0.5-1.1% memory overhead for Qwen3 4B and 8B. Switching between them costs ~1ms, scaling with the number of blocks rather than model size, while compiling a new budget is a one-off cost that can be paid in advance.
  • Stacked ablation. Each ingredient of our recipe improves performance under deep pruning, by up to 9.8%, 8.4% and 7.3%.

Although the paper emphasizes adapting model computation dynamically to changing workloads, the actual latency and throughput measurements are limited to AugReg ViT-B/16. More importantly, Appendix A.4 shows that the reported inference improvements depend substantially on compilation and hardware-friendly pruning constraints. For example, at batch size 1, pruning provides essentially no speedup under standard compilation, while the reported 1.26× acceleration requires reduce-overhead compilation and GPU-friendly tiling. The latter also incurs an accuracy reduction at high sparsity. Furthermore, compiled graphs must be cached separately for different budgets, and compiling a previously unseen budget requires 12–49 seconds. The reported switching overhead excludes this compilation cost. These results suggest that the method provides an effective nested pruning ranking, but its ability to support flexible, low-overhead inference over a broad range of budgets remains less fully demonstrated. The paper would benefit from a more complete analysis of accuracy, latency, memory consumption, and compilation overhead across multiple budgets, ideally including larger text encoders.

We thank the reviewer for raising this point. We agree that speedup measurements should be qualified across a broader spectrum of models. We report throughput and latency results for DINOv3 ViT-H+/16 and Qwen3 4B and 8B in the shared response.

We find that larger models can be further accelerated, by up to 2.19x in terms of throughput for a compiled 60% sparse Qwen3 8B model at batch 32, A=64A=64 and using 256-token long inputs. Even without compilation, the same model can reach 1.84x throughput at A=8A=8. In terms of latency, larger models still require reduce-overhead compilation to realize speedups, but can be further accelerated compared to AugReg ViT-B/16, reaching 1.7x faster batch-1 inference at 60% sparsity for both Qwen models using 256-token long inputs, and up to 1.98x for Qwen3 8B using 2048-token long inputs.

We believe that reduce-overhead more closely mimics a production setup, as standard compilation only fuses kernels, and does not remove the overhead due to kernel launches. Furthermore, we find that cached graphs only add minimal overhead, split across a fixed graph pool from 71MB for Qwen3 8B to 86MB for AugReg ViT-B/16 and a per-sparsity graph, each of which costs between ~1.5MB and ~0.5MB for the aforementioned models. The main cost is the initial compilation, which may cost up to 76s in the worst case for large text encoders, but, in practice, we expect users to deploy a limited subset of sparsities, for which the compilation cost can be paid in advance and would not affect online inference. This cost is not unique to our method, and is shared by every weight-elastic approach.

Regarding the accuracy reduction at high sparsity under tiling, we believe this to be specific to AugReg, as large vision and text encoders, detailed in the common response, suffer effectively no penalty.

We further discuss memory and compilation overhead in the question below.

Table 3 suggests that joint optimization across 14 sparsity targets outperforms separate single-target optimization. However, each multi-budget optimization step evaluates 14 masked configurations, whereas single-target optimization requires only one. If both approaches are trained for the same number of steps, the multi-budget method receives substantially more optimization computation. Consequently, the reported improvements do not clearly distinguish the benefits of cross-budget optimization from those of additional compute. Does multi-budget optimization still outperform single-target optimization when controlling for total forward/backward compute?

We agree with the reviewer that a compute-matched optimization would be a fairer setup to evaluate single versus multi-budget optimization. We observe that scaling the optimization budget, measured as budgets ×\times steps, for single target optimization to match the elastic run leads to worse performance overall, as shown in the table below.

As this may simply be an effect of overfitting the calibration data, we further experiment with reducing the number of steps and budgets for our multi-budget optimization, ensuring matched compute with the single target optimization. We find that using 3 or 7 training budgets outperforms single budget optimization, while optimizing for 14 budgets at once performs worse. We hypothesize that this is due to the reduced number of steps, as we can only run 21 steps at 14 budgets to match single target optimization at 300 steps in terms of compute.

On the other hand, a single-target ranking only performs well near its target. Optimized for 40% sparsity, it reaches 47.6% at 50%, versus 59.2% for our 14-budget ranking. Covering kk budgets would thus require kk separate runs, i.e., kk times the compute, making the per-target matched comparison above a favorable setting for single-target optimization.

SparsitySingleSingle (Matched)Ours
10% / 31.7 G82.381.882.6
20% / 28.2 G79.778.780.1
30% / 24.8 G75.273.775.7
40% / 21.2 G67.266.969.1
50% / 17.8 G57.456.759.3
60% / 14.2 G41.639.445.2

Top-1 accuracy in ImageNet-1k k-NN for pruned AugReg ViT-B/16 models, whose ranking was optimized either via single or multi budget optimization. Matched indicates equivalent compute in terms of step ×\times budgets.

Method# Train budgetsTrain budgetsSteps20% / 28.2 G40% / 21.2 G60% / 14.2 G
Single1The target budget30079.767.241.6
Ours3[0.2,0.4,0.6][0.2, 0.4, 0.6]10080.1 (+0.4)69.2 (+2.0)43.2 (+1.6)
Ours7[0.1,0.70][0.1, 0.70], step 0.14379.9 (+0.2)68.0 (+0.8)43.0 (+1.4)
Ours14[0.05,0.70][0.05, 0.70], step 0.052179.8 (+0.1)66.8 (−0.4)40.4 (−1.2)

Top-1 accuracy in ImageNet-1k k-NN for pruned AugReg ViT-B/16 models, whose ranking was optimized either via single or multi budget optimization, under fixed compute in terms of step ×\times budgets.

The paper primarily combines the global ranking-based elastic pruning formulation of SnapViT with established differentiable masking techniques, extending the optimization variables to individual neurons and attention heads. Although this change produces substantial empirical improvements, the underlying formulation is relatively straightforward. In particular, sigmoid-based masking, gradient optimization of importance scores, and averaging objectives over multiple budgets are natural extensions of existing ideas. The authors could strengthen the methodological contribution by providing a deeper analysis of why fine-grained multi-budget optimization works, beyond showing that finer parameterizations improve accuracy.

We thank the reviewer for the question. We investigated the role of multi-budget optimization by varying the number of budgets. We find that halving the number of budgets has effectively no performance impact, and further halving them only suffers relatively minor performance degradation. On the other hand, using only two budgets at the extremes of the optimized sparsity band causes performance to collapse.

We believe that the cause of this collapse stems from the misallocation of sparsity across MLP neurons and attention heads. Specifically, when a ranking is used at a sparsity it was not optimized for, it removes far more attention heads than our default 14-target ranking at the same sparsity, e.g., 45.1% versus 19.4% of heads at 60% sparsity for a ranking optimized only for 40%.

This is particularly visible in the single target row: the 40% sparse model, i.e., the one optimized for, prunes heads to the exact same ratio as our 14-budget optimization does, while sparser budgets aggressively prune heads, leading to a performance degradation of up to 17.7% at 60% sparsity. We see similar effects in the 2-target run: the 60% sparse model, albeit degraded, performs better than single target, as we optimize for the 70% sparse model. On the other hand, mid-sparsity budgets suffer significant degradation, and aggressively prune heads compared to our default setup, indicated with ★.

In terms of novelty, we agree with the reviewer that the individual ingredients of our method have been previously studied in the literature. We believe that our main contribution is a formulation that learns the ranking of functional units itself, rather than rescaling a fixed one as in SnapViT, by repurposing differentiable pruning, traditionally used to produce a single sparse model, to optimize a ranking across multiple budgets. We believe that such a formulation, which works across models spanning different sizes, pretraining strategies, and modalities, with only some modality-specific differences, is a worthwhile contribution to the community. We further discuss novelty in the shared comment.

# TargetsTargets10%20%30%40%50%60%
14 ★[0.05,0.70][0.05, 0.70], step 0.050.0% / 82.50.7% / 79.83.5% / 75.54.9% / 69.110.4% / 59.219.4% / 41.8
7[0.1,0.70][0.1, 0.70], step 0.10.0% / 82.6 (+0.1)2.1% / 80.0 (+0.2)4.2% / 75.5 (−0.0)6.2% / 69.2 (+0.1)11.1% / 59.2 (+0.0)21.5% / 42.5 (+0.7)
30.1, 0.4, 0.70.0% / 82.6 (+0.1)4.2% / 79.6 (−0.2)6.9% / 75.7 (+0.2)8.3% / 68.8 (−0.3)22.9% / 56.3 (−2.9)33.3% / 40.5 (−1.2)
20.1, 0.70.7% / 82.3 (−0.2)7.6% / 78.4 (−1.4)20.1% / 68.0 (−7.5)25.0% / 54.4 (−14.7)29.9% / 41.0 (−18.2)36.1% / 30.1 (−11.7)
10.40.0% / 82.6 (+0.1)1.4% / 79.8 (−0.0)2.8% / 75.3 (−0.2)4.9% / 67.7 (−1.4)24.3% / 47.6 (−11.6)45.1% / 24.1 (−17.7)

Top-1 accuracy in ImageNet-1k k-NN for AugReg ViT-B/16 models pruned using our method, varying the number of target sparsities used during ranking optimization. Each cell reports the percentage of pruned heads and the top-1 k-NN accuracy.

What is the total memory overhead of supporting multiple compiled sparsity configurations for large text encoders? How does the switching mechanism scale beyond ViT-B/16?

Our elastic models have two types of memory overhead. The first is the QKV buffer, which holds the truncated QKV weights for the current sparsity as a contiguous block. This is required as removing entire heads results in a ragged QKV matrix, which cannot be expressed as a tensor view in PyTorch. On the other hand, using such a buffer is merely a convenience to remain compatible with existing attention implementation as in e.g., timm, but can be removed by laying out the QKV matrix as [qkv,qkv,...,qkv], where qkv represents the query, key, and value projections for a single head, instead of the standard [QKV] layout, where Q, K, V represent the projections for every head. Using the first layout, the scratch buffer is not needed, and its memory overhead drops to 0. As a side effect, this also approximately halves the switching cost, from 0.46ms for AugReg ViT-B/16 to 0.23ms, and from 1.04ms to 0.59ms for DINOv3 ViT-H+/16. We will make such a layout the default.

In the current implementation, the fused QKV buffer is only required for models which use a fused QKV projection, and accounts for ~85MB for AugReg ViT-B/16 and ~629MB for DINOv3 ViT-H+/16. In the context of text encoders, this buffer ranges from ~21MB for the 33M params BGE Small to ~302MB for the 334M BGE Large. As for large decoder-based text encoders, e.g., the Qwen3 family, they implement grouped-query attention, or GQA, which means they do not use fused QKV projections, and hence no buffer is required.

The second memory overhead is only relevant for models compiled in reduce-overhead mode, as this requires PyTorch to record the sequence of kernel launches. Assuming 7 compiled models, this overhead is ~89MB for AugReg ViT-B/16, ~92MB for DINOv3 ViT-H+/16, ~92MB for Qwen3 4B and ~80MB for Qwen3 8B. In the context of large text encoders this means a relative memory overhead of 1.1-0.5%. The reduce-overhead memory overhead scales linearly for every additional budget.

Regarding the switching mechanism, the switching cost scales with the number of blocks rather than model size:

ModelBlocksParametersMean switching costPer blockCompilation cost per budget
AugReg ViT-B/161287M0.46 ms39 µs12–49 s
DINOv3 ViT-H+/1632840M1.04 ms32 µs40–74 s
Qwen3 4B364.0B0.83 ms23 µs45–76 s
Qwen3 8B367.6B0.82 ms23 µs47–75 s
Reviewer_v4ZCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a retraining-free method for making pre-trained transformer encoders elastic, so that a single model can be pruned to arbitrary compute budgets at inference time. The authors assign an importance score to every MLP neuron and attention head, initialized from a Fisher-based approximation, and then refine the resulting global ranking with sigmoid soft masks .

Critical strengths

–The method is simple and practical. A single frozen model can be pruned to any budget at inference time, with no retraining and negligible switching cost –The evaluation is broad, covering 27 vision and text encoders from 22M to 8B parameters, multiple tasks. –The result is striking. Beyond 30% sparsity, most baselines degrade sharply while the proposed method degrades smoothly

Critical weaknesses

–The comparison may not be fully fair. All baselines are run without weight correction. –Lack of more speedup measurement. Speedup results are reported for only one model. –The methodological novelty is somewhat limited. The work builds directly on SnapViT’s retraining-free ranking formulation, and soft masking for differentiable pruning is well established.

Questions for authors

How do the baselines perform with weight correction enabled? Can the authors report measured throughput and latency on more models? How sensitive is the method to the temperature, the set of sparsity targets, and the choice of calibration data? What is the optimization cost of the method? Beyond per-unit scores, what is the main technical contribution over SnapViT?

7 comments

We thank the reviewer for the insightful comments and the support for our work. We first summarize our responses to each raised feedback item, which we then discuss in detail:

  • Weight correction. Our correction-free method matches or outperforms most corrected baselines. Corrected SnapViT outperforms our method on AugReg ViT-B/16, by up to 4 points, but requires 2.4x the weight memory to serve six budgets. On DINOv3 ViT-H+/16, we outperform it under deep pruning, by up to 18.2%.
  • Speedups. Larger models can be further accelerated, by up to 2.19x in throughput and, under reduce-overhead compilation, 1.98x in batch-1 latency.
  • Novelty. Beyond per-unit scores, our main contribution is a differentiable relaxation of the ranking, which lets us learn the ranking itself, rather than rescale a fixed one, with any differentiable loss, including the model's native objective. We further rank MLP neurons and attention heads jointly on a common scale, so that users can simply specify a target sparsity instead of hand-tuning its allocation across neurons and heads.
  • Sensitivity. Temperatures in [0.05,0.2][0.05, 0.2] work well, 7 targets match 14, and more or task-aligned calibration data generally improves performance.
  • Cost. Our method runs in minutes and scales sub-linearly with model size, versus hours for OPTIN and VBP.

The comparison may not be fully fair. All baselines are run without weight correction. How do the baselines perform with weight correction enabled?

The reviewer is right. In the main paper, we only report correction-free results, as using weight correction either requires storing a bank of corrected weights, whose size grows linearly as more budgets have to be supported, or breaks the elasticity property. Where possible, e.g., in the case of VBP with its mean compensation, we report results for corrected models.

In the appendix, we compare with weight corrected baselines in Tables 11 and 12. We replicate the results here for reviewer convenience, include VBP for completeness, and also measure the total weight memory required to serve 6 pruned models as a multiple of the dense weights. The tables below illustrate the ImageNet-1k top-1 accuracy in k-nn classification for pruned AugReg ViT-B/16 and DINOv3 ViT-H+/16 models. We find that our correction-free method matches or outperforms most baselines which make use of weight correction. SnapViT with weight correction outperforms our method on AugReg ViT-B/16, but we outperform it on DINOv3 ViT-H+/16 from 30% sparsity. Furthermore, corrected SnapViT requires 2.4x the memory used for the dense model to serve six sparse models, while our method does not add any weight memory overhead.

We further find that our method can also benefit from weight correction, and when we use weight correction, we match or outperform SnapViT on AugReg ViT-B/16, especially at higher sparsity levels. With weight correction, our method has the same weight memory overhead as SnapViT.

Finally, in Table 13 of the appendix, replicated here for reviewer convenience, we report the performance of our correction-free method versus corrected SlimGPT. We find that our method outperforms corrected SlimGPT at every evaluated sparsity target on BGE-B, BGE-L, Qwen3 0.6B, and Qwen3 8B, and on seven of nine encoders between 40% and 60% sparsity.

MethodCorrectionWeight memory10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
SparseGPT✗≈\approx 1.0x78.864.236.520.314.110.2
SparseGPT✓4.9x82.479.372.259.942.429.1
FPTP✗≈\approx 1.0x82.078.571.344.317.212.6
FPTP✓≈\approx 1.0x82.179.576.257.019.313.3
SnapViT✗1.0x82.277.570.157.728.417.7
SnapViT✓2.4x82.881.078.473.063.347.5
VBP✓≈\approx 1.0x82.681.174.351.425.513.2
Ours✗1.0x82.680.175.769.159.345.2

Top-1 accuracy in k-NN classification on ImageNet-1k for pruned AugReg ViT-B/16 models, using our method and the baseline, with and without weight correction. Our correction-free method largely outperforms corrected SparseGPT and FPTP, improves over VBP at high sparsity, and is within 4 points from corrected SnapViT, but only requires a single copy of the weights to serve any budget, while SnapViT requires 2.4x the dense weights memory to serve 6 sparse budgets.

MethodCorrectionWeight memory10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
SparseGPT✓4.9x82.479.372.259.942.429.1
FPTP✓≈\approx 1.0x82.179.576.257.019.313.3
SnapViT✓2.4x82.881.078.473.063.347.5
VBP✓≈\approx 1.0x82.681.174.351.425.513.2
Ours✓2.4x82.881.078.172.664.354.0

Top-1 accuracy in k-NN classification on ImageNet-1k for pruned AugReg ViT-B/16 models, using our method and the baseline with weight correction. We match SnapViT at shallow to mid sparsity, and outperform both VBP and SnapViT under deep pruning.

MethodCorrectionWeight memory10% / 398.6 G20% / 354.4 G30% / 310.0 G40% / 265.8 G50% / 221.7 G60% / 177.5 G
SparseGPT✓4.9x85.285.180.669.655.438.3
SnapViT✓2.2x85.285.285.083.576.861.2
VBP✓≈\approx 1.0x83.879.454.513.45.06.2
Ours✗1.0x85.185.185.184.783.379.4

Top-1 accuracy in k-NN classification on ImageNet-1k for pruned DINOv3 ViT-H+/16 models, using our correction-free method and the baselines. Our method matches corrected baselines at shallow sparsity, and significantly improves over them under deep pruning. FPTP is not compatible with DINOv3, as it requires a classification head.

BackboneMethodCorrectionWeight memoryDense10%20%30%40%50%60%
BGE-SSlimGPT✗1.0x64.363.261.258.454.748.941.8
SlimGPT✓2.0x63.662.560.458.354.048.3
Ours✗1.0x63.762.360.358.456.754.3
BGE-BSlimGPT✗1.0x65.263.861.257.153.440.940.6
SlimGPT✓2.3x64.163.161.658.155.232.2
Ours✗1.0x64.864.263.261.859.556.4
BGE-LSlimGPT✗1.0x65.859.151.850.149.647.140.0
SlimGPT✓2.5x63.658.056.153.353.045.7
Ours✗1.0x65.564.863.661.259.557.1
E5-BSlimGPT✗1.0x62.562.459.954.155.651.948.0
SlimGPT✓2.3x62.261.659.657.353.548.2
Ours✗1.0x62.162.060.859.857.755.6
E5-LSlimGPT✗1.0x62.454.755.654.652.046.143.5
SlimGPT✓2.5x63.362.661.359.256.053.1
Ours✗1.0x62.461.761.160.359.157.3
E5-Mistral 7BSlimGPT✗1.0x68.367.565.662.156.348.936.7
SlimGPT✓2.3x67.967.165.463.260.757.2
Ours✗1.0x68.167.365.662.759.456.8
Qwen3 0.6BSlimGPT✗1.0x70.150.937.421.018.314.114.1
SlimGPT✓2.0x69.162.056.653.247.442.0
Ours✗1.0x69.566.861.958.254.452.1
Qwen3 4BSlimGPT✗1.0x74.168.759.844.434.022.214.6
SlimGPT✓2.3x73.071.668.664.959.956.1
Ours✗1.0x72.670.868.063.658.654.6
Qwen3 8BSlimGPT✗1.0x74.671.867.461.250.339.426.1
SlimGPT✓2.3x73.672.069.465.760.456.7
Ours✗1.0x74.272.670.367.363.858.9

MTEB(eng, v2) score for the nine text encoders, comparing our correction-free method with SlimGPT with and without correction. Our method outperforms corrected SlimGPT at every evaluated sparsity target on BGE-B, BGE-L, Qwen3 0.6B, and Qwen3 8B, and on seven of nine encoders between 40% and 60% sparsity.

Comments on the paper

4 comments

We thank the reviewers for their time and suggestions, which will definitely improve the paper. We will tackle points raised by multiple reviewers in this comment, namely the concerns about novelty and a more thorough characterization of speedups.

Novelty

We thank all reviewers for the opportunity to clarify our novelty. We agree that differentiable pruning, multi-budget optimization, and unit ranking have individually been studied, and we do not claim them as novel in isolation. We are well aware that SnapViT already showed that pretrained vision transformers can be made elastic by ranking their units, without updating the weights. However, SnapViT does not learn this ranking: it inherits the order of units from Hessian-diagonal scores, and its search only rescales them per MLP block and attention head, so the order of neurons within each block never changes. Our contribution is a formulation that learns the ranking itself.

Existing methods cannot directly learn such a ranking. Differentiable pruning methods learn one mask per sparsity target, typically while updating the weights. Applying them to k budgets thus yields k independent masks, which are generally not nested, and k sets of weights, which cannot be served from a single model. On the other hand, SnapViT's black-box search does not scale to one score per MLP neuron and attention head, i.e., about 37k scores for a ViT-B/16 and 444k for Qwen3 8B, which is why it only optimizes one scale per MLP block and attention head, i.e., 156 for a ViT-B/16.

Our key technical idea is a differentiable relaxation of the ranking. At every budget, we replace the hard cutoff in the ranking with a soft mask, so that the loss of the pruned model can be backpropagated to the score of each individual unit, letting every unit move freely across the ranking. We further median-center head and neuron scores and optimize them jointly in one ranking, so that the optimization, rather than a hand-tuned schedule, decides how sparsity is allocated between them, and users only need to specify a target sparsity.

Since our method learns the ranking rather than rescaling a fixed one, it can also recover from poor initial scores, e.g., improving the accuracy of AugReg ViT-B/16 at 40% sparsity from 14.7% to 63.2% when starting from random scores. More broadly, our results suggest that pretrained transformers are more elastic than previously shown, and that the ranking, rather than the weights, is the main bottleneck: on the same model, optimizing the ranking improves accuracy by up to 28.2 points at 60% sparsity, while weight correction adds at most 8.8 points on top. The stacked ablation in our response to reviewer 1ffB further shows that the components of our recipe are complementary, with the full recipe improving accuracy at 50% sparsity from 28.0% to 59.4%.

Overall, we believe that this formulation, and its broad applicability across transformers spanning sizes, pre-training strategies, and modalities, would be a worthwhile contribution to the community.

Speedups

We extended our analysis of speedups to DINOv3-H+/16, Qwen3 4B, and Qwen3 8B, for which we report both throughput and latency in the tables below. In terms of throughput, we find that larger models can be further accelerated compared to AugReg ViT-B/16, by up to 2.19x over the dense model for a 60% sparse Qwen3 8B model at batch 32 and using 256-token long inputs. As for latency, we find that kernel launch dominates the batch-1 inference cost for every model, and practical acceleration requires both reduce-overhead compilation and GPU-friendly MLP tiling. Nonetheless, we find that under such a setup batch-1 inference can be up to 1.7x faster at 60% sparsity for both Qwen3 4B and 8B at 256 tokens.

We further find that, under reduce-overhead compilation and A=8 tiling, batch-1 latency speedups depend on the sequence length. For Qwen3 4B, processing a 16-token sequence through a 60% sparse model only results in a 1.27x latency speedup, while processing a 512-token sequence results in a 1.85x speedup. As for the 8B variants, we observe speedups for short and long sequences alike, with 2048-token sequences being processed up to 1.98x faster using a 60% sparse model.

While standard compilation does not incur any additional memory overhead, reduce-overhead uses a small buffer to record the sequence of kernel launches for every target sparsity. This overhead is split across a fixed graph pool, between 71MB for Qwen3 8B to 86MB for AugReg ViT-B/16, and per-sparsity graphs, each costing between ~1.5MB and ~0.5MB for the aforementioned models. For example, supporting 7 budgets from 0 to 60% sparsity incurs an 80MB memory overhead for Qwen3 8B and 89MB memory overhead for AugReg ViT-B/16.

Finally, we measure the accuracy impact of GPU-friendly tiling for DINOv3 ViT-H+/16 and Qwen3 4B and 8B. For the former, we report the top-1 accuracy in k-NN classification on ImageNet-1k and the transfer average, and for the large text encoders we report the 15-dataset BEIR nDCG@10 average, in the elastic retrieval with a frozen index setup.

We find that GPU-friendly tiling has effectively no performance impact on large vision and text encoders, with deltas being within 0.2 points from the ragged-width performance.

DINOv3 ViT-H+/16

EagerCompiled
Sparsity ↓A=1A=8A=64A=1A=8A=64
Dense (442.6 G)235 samples/s319 samples/s
10% / 398.6 G0.79×1.05×1.08×1.09×1.09×1.10×
20% / 354.4 G0.83×1.16×1.19×1.21×1.24×1.25×
30% / 310.0 G0.95×1.26×1.29×1.33×1.38×1.39×
40% / 265.8 G1.02×1.39×1.42×1.53×1.56×1.57×
50% / 221.7 G1.25×1.53×1.56×1.75×1.76×1.78×
60% / 177.5 G1.43×1.73×1.75×2.04×2.05×2.08×

Throughput for pruned DINOv3 ViT-H+/16 models, at batch size 128.

EagerCompiled
Sparsity ↓A=1A=8A=64A=1A=8A=64
Dense (442.6 G)28.5 ms10.7 ms
10% / 398.6 G1.03×1.02×1.04×0.82×1.08×1.09×
20% / 354.4 G1.04×1.02×1.01×0.87×1.18×1.18×
30% / 310.0 G1.05×1.05×1.04×0.97×1.28×1.28×
40% / 265.8 G1.04×1.06×1.05×1.08×1.40×1.41×
50% / 221.7 G1.07×1.08×1.07×1.27×1.54×1.54×
60% / 177.5 G1.03×1.06×1.07×1.43×1.69×1.69×

Batch-1 latency for pruned DINOv3 ViT-H+/16 models, eager and compiled with reduce-overhead.

Sparsity / GFLOPsA=1A=8A=64
10% / 399 G85.185.1 (+0.0)85.0 (−0.1)
20% / 354 G85.185.1 (+0.0)85.1 (+0.0)
30% / 310 G85.085.1 (+0.1)85.1 (+0.1)
40% / 266 G84.784.7 (+0.0)84.7 (+0.0)
50% / 222 G83.383.3 (+0.0)83.3 (+0.0)
60% / 178 G79.479.4 (+0.0)79.4 (+0.0)

Top-1 accuracy in ImageNet-1k k-NN classification for pruned DINOv3 ViT-H+/16 models, using ragged MLP widths (A=1) or GPU-friendly tiling (A=8,64). GPU-friendly tiling has effectively no performance impact.

Sparsity / GFLOPsA=1A=8A=64
10% / 399 G87.087.0 (+0.0)87.0 (+0.0)
20% / 354 G86.986.8 (−0.1)86.8 (−0.1)
30% / 310 G86.686.6 (+0.0)86.5 (−0.1)
40% / 266 G86.186.0 (−0.1)86.0 (−0.1)
50% / 222 G84.884.9 (+0.1)84.9 (+0.1)
60% / 178 G80.580.5 (+0.0)80.3 (−0.2)

Average top-1 accuracy on the transfer datasets in k-NN classification for pruned DINOv3 ViT-H+/16 models, using ragged MLP widths (A=1) or GPU-friendly tiling (A=8,64). GPU-friendly tiling has effectively no performance impact.

Qwen3 4B

EagerCompiled
Sparsity ↓A=1A=8A=64A=1A=8A=64
Dense (1899.1 G)78 samples/s110 samples/s
10% / 1713.6 G0.59×1.07×1.07×1.07×1.09×1.09×
20% / 1524.4 G0.65×1.15×1.15×1.18×1.20×1.21×
30% / 1338.9 G0.79×1.25×1.25×1.32×1.34×1.35×
40% / 1154.6 G0.91×1.37×1.37×1.50×1.53×1.53×
50% / 972.6 G1.00×1.53×1.53×1.74×1.78×1.78×
60% / 788.9 G1.28×1.75×1.76×2.09×2.12×2.12×

Throughput for pruned Qwen3 4B models, at batch size 32 and 256 tokens.

EagerCompiled
Sparsity ↓A=1A=8A=64A=1A=8A=64
Dense (1899.1 G)40.1 ms14.5 ms
10% / 1713.6 G1.00×1.01×1.00×0.55×1.04×1.05×
20% / 1524.4 G0.99×1.03×1.00×0.61×1.13×1.13×
30% / 1338.9 G0.99×1.02×1.00×0.76×1.27×1.28×
40% / 1154.6 G1.00×1.03×1.02×0.84×1.34×1.36×
50% / 972.6 G0.99×1.02×1.02×0.90×1.55×1.55×
60% / 788.9 G0.99×1.02×1.01×1.11×1.69×1.70×

Batch-1 latency for pruned Qwen3 4B models at 256 tokens, eager and compiled with reduce-overhead.

TokensDense10%20%30%40%50%60%
168.9 ms1.03×1.11×1.18×1.15×1.22×1.27×
328.8 ms1.05×1.12×1.22×1.20×1.33×1.42×
649.4 ms1.06×1.14×1.22×1.32×1.44×1.38×
12810.8 ms1.05×1.11×1.21×1.30×1.46×1.51×
25614.5 ms1.04×1.13×1.27×1.34×1.55×1.69×
51226.5 ms1.03×1.16×1.37×1.47×1.69×1.85×
102448.4 ms1.04×1.12×1.25×1.34×1.56×1.77×
2048105.3 ms1.07×1.13×1.21×1.34×1.55×1.77×

Batch-1 latency of pruned Qwen3 4B models, with A=8, across sequence lengths, under reduce-overhead compilation.

Sparsity / GFLOPsA=1A=8A=64
10% / 1714 G59.959.9 (+0.0)59.9 (+0.0)
20% / 1524 G58.458.4 (+0.0)58.4 (+0.0)
30% / 1339 G56.256.3 (+0.1)56.1 (−0.1)
40% / 1155 G53.353.3 (+0.0)53.4 (+0.1)
50% / 973 G47.847.9 (+0.1)47.7 (−0.1)
60% / 789 G41.541.6 (+0.1)41.3 (−0.2)

15-dataset BEIR nDCG@10 average for pruned Qwen3 4B models, using ragged MLP widths (A=1) or GPU-friendly tiling (A=8,64). GPU-friendly tiling has effectively no performance impact.

Qwen3 8B

EagerCompiled
Sparsity ↓A=1A=8A=64A=1A=8A=64
Dense (3595.1 G)49 samples/s63 samples/s
10% / 3239.3 G0.56×1.07×1.08×1.07×1.09×1.10×
20% / 2882.5 G0.58×1.17×1.17×1.19×1.22×1.22×
30% / 2526.5 G0.74×1.27×1.28×1.34×1.36×1.37×
40% / 2177.6 G0.87×1.42×1.42×1.53×1.56×1.56×
50% / 1833.3 G0.93×1.59×1.60×1.78×1.81×1.82×
60% / 1482.9 G1.13×1.84×1.85×2.14×2.18×2.19×

Throughput for pruned Qwen3 8B models, at batch size 32 and 256 tokens.

EagerCompiled
Sparsity ↓A=1A=8A=64A=1A=8A=64
Dense (3595.1 G)39.0 ms21.2 ms
10% / 3239.3 G0.78×0.96×0.98×0.54×1.04×1.04×
20% / 2882.5 G0.79×0.98×0.96×0.52×1.10×1.09×
30% / 2526.5 G0.90×0.98×0.97×0.68×1.17×1.17×
40% / 2177.6 G0.98×1.00×1.00×0.74×1.32×1.32×
50% / 1833.3 G0.98×0.95×0.99×0.84×1.46×1.46×
60% / 1482.9 G0.96×0.99×0.99×1.00×1.70×1.70×

Batch-1 latency for pruned Qwen3 8B models at 256 tokens, eager and compiled with reduce-overhead.

TokensDense10%20%30%40%50%60%
1613.2 ms1.08×1.17×1.27×1.39×1.56×1.75×
3213.6 ms1.07×1.15×1.26×1.40×1.57×1.80×
6413.9 ms1.06×1.15×1.26×1.40×1.57×1.75×
12816.8 ms1.08×1.18×1.26×1.38×1.54×1.75×
25621.2 ms1.05×1.10×1.18×1.31×1.47×1.70×
51239.0 ms1.04×1.10×1.20×1.36×1.57×1.87×
102479.9 ms1.08×1.15×1.25×1.40×1.58×1.90×
2048163.9 ms1.09×1.18×1.27×1.45×1.66×1.98×

Batch-1 latency of pruned Qwen3 8B models, with A=8, across sequence lengths, under reduce-overhead compilation.

Sparsity / GFLOPsA=1A=8A=64
10% / 3239 G62.262.2 (+0.0)62.2 (+0.0)
20% / 2883 G61.261.2 (+0.0)61.3 (+0.1)
30% / 2526 G59.259.2 (+0.0)59.2 (+0.0)
40% / 2178 G56.256.3 (+0.1)56.3 (+0.1)
50% / 1833 G52.552.5 (+0.0)52.5 (+0.0)
60% / 1483 G43.643.6 (+0.0)43.6 (+0.0)

15-dataset BEIR nDCG@10 average for pruned Qwen3 8B models, using ragged MLP widths (A=1) or GPU-friendly tiling (A=8,64). GPU-friendly tiling has effectively no performance impact.

Additionally, we summarize below the insights that surfaced from individual reviewer comments:

  • Multi-budget optimization (reviewer 1ffB). Rankings optimized for a single budget over-prune attention heads away from it, leading to a performance degradation of up to 17.7% at 60% sparsity. We further find that, under matched compute, optimizing 3 or 7 budgets still outperforms single-budget optimization.
  • Weight correction (reviewer v4ZC). Without weight correction, our method outperforms corrected SlimGPT on seven of nine encoders between 40% and 60% sparsity, and corrected SnapViT on DINOv3 ViT-H+/16 by up to 18.2%. On AugReg ViT-B/16, we are within 4 points from corrected SnapViT, which requires 2.4x the weight memory to serve the six budgets evaluated in the paper.
  • Robustness (reviewer nZgc). We outperform every baseline on ImageNet V2, Sketch and R from 40% sparsity onwards. Furthermore, text encoders calibrated only on MS MARCO, a passage retrieval dataset, perform strongly on MTEB, whose 41 tasks also cover classification, clustering, semantic similarity, and summarization. Losses and rankings reliably converge across seeds, and the soft relaxation closely matches the hard-pruned models used at deployment, with gaps of at most 2.7%.
  • Sensitivity and cost (reviewer v4ZC). Sharp temperatures in [0.05,0.2][0.05, 0.2] work well, 7 targets match 14, and task-aligned calibration improves performance by up to 26.6%. Our method runs in minutes, versus hours for OPTIN and VBP.

Paper revisions

We will update the paper with the results and insights from this rebuttal, namely:

  • A revised introduction and related work that more clearly state our contribution and its relation to prior work on differentiable pruning and elastic inference.
  • Speedups for DINOv3 ViT-H+/16, Qwen3 4B, and Qwen3 8B, including latency across sequence lengths and the memory and compilation overhead of serving multiple budgets (reviewers v4ZC, 1ffB).
  • The stacked ablation, the compute-matched comparison of single- and multi-budget optimization, and the analysis of attention head allocation (reviewer 1ffB).
  • A more prominent discussion in the main text of the comparison with weight-corrected baselines, including the weight memory required to serve multiple budgets (reviewer v4ZC).
  • The sensitivity to the number of sparsity targets (reviewer v4ZC).
  • The robustness to distribution shift, the optimization stability across seeds, and the soft-to-hard gap (reviewer nZgc).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.