acceptodds
Under review as a conference paper at ICLR 2027

GradSelect: Output-Targeted Patch Ranking under Exact-K Vision Transformer Pruning

Abstract

How should a frozen vision transformer choose K patches when the pruning layer and token budget are fixed? This paper isolates patch ranking from system choices such as pruning schedules, routing modules, and classifier changes. GradSelect scores each image by first running the full model to obtain its own predicted class, backpropagating that output to the cut layer, keeping positive first-order support for each patch, and fusing this signal with CLS attention after per-image z-scoring. The fusion weight is selected on a lock split, and all final comparisons use the same exact-K intervention. At 25% retention after block 8, GradSelect has the highest point estimate among primary selectors in all nine ImageNet-1K, Food-101, and SUN397 cells across CLIP, DINOv2, and DeiT, including five cells not used to choose the recipe. Against the stronger per-cell opponent drawn from either the primary suite or the attribution/removal stress suite, it wins 6/9 cells with +2.24-point mean and +2.53-point median accuracy margins; exact single-token removal remains stronger on supervised DeiT. On Caltech-101, GradSelect is best on CLIP and DINOv2 and 0.22 points below DivPrune on DeiT. Because scoring requires a full forward pass and one backward pass, GradSelect is best understood as an offline output-aware patch selector rather than an online accelerator.

Primary area:
interpretability and explainable AI
Keywords:
visual token pruning, vision transformers, gradient-based attribution, exact-K selection, model compression, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_c8m7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GradSelect, an output-aware patch ranking method for frozen Vision Transformers under a fixed cut and an exact-K token budget. GradSelect computes the gradient of the full model’s predicted-class log probability with respect to patch representations, uses the resulting activation-gradient product as a positive-support score, and fuses it with CLS-to-patch attention through per-image normalization and a learned coefficient. The paper evaluates this ranking rule under a controlled exact-K intervention across ImageNet-1K, Food-101, SUN397, and three ViT backbones, together with stress tests, transfer evaluation, fidelity analysis, and computational-cost diagnostics. The main results show that GradSelect achieves the best point estimate in all nine primary cells, although its advantage is weaker against stronger attribution/removal baselines in several settings.

Critical strengths

Well-controlled experimental protocol. The paper carefully fixes the cut, token budget, suffix, classifier/probe, and evaluation split, allowing a relatively clean comparison of patch-ranking rules rather than confounding ranking quality with different pruning systems.

Broad and systematic evaluation. The experiments cover three datasets and three different frozen ViT representations, and additionally include stress tests, transfer experiments, retention/cut-depth analyses, and exact-removal diagnostics. This provides useful evidence about both the strengths and limitations of the proposed selector.

Reasonable output-aware formulation and transparent limitations. Using the full model’s own predicted class as the target is a natural model-fidelity objective, and the paper is relatively careful to distinguish the proposed first-order score from true causal token removal and to acknowledge that the method is an offline selector rather than an online accelerator.

Critical weaknesses

Limited methodological novelty and insufficient positioning against closely related token-attribution pruning methods. My main concern is that the core GradSelect construction appears to be a relatively straightforward recombination of established token-importance mechanisms, and the paper does not yet make a sufficiently strong case for what fundamentally new principle is introduced.

A particularly relevant prior work is STAR (^^^), which already proposes synergistic patch pruning by combining two complementary views of patch importance: an online intra-layer importance derived from the CLS-to-patch attention map and an offline inter-layer relevance score obtained through backward Layer-wise Relevance Propagation (LRP). These scores are fused and used for top-k patch pruning across layers. Thus, the high-level idea of combining a forward attention-based signal with a backward output-relevance signal to rank patches is already present in the literature.

GradSelect replaces STAR's backward relevance signal with an instance-specific first-order score, $ \left\langle z_i,\nabla_{z_i}\log p(\hat y_{\mathrm{full}}\mid x)\right\rangle, $ applies positive clipping and per-image normalization, and linearly fuses it with CLS attention before exact-K Top-K selection. I agree that this is a reasonable and potentially useful refinement, but it is not yet clear that it constitutes a substantially new pruning principle rather than a new choice of backward attribution operator within an already established attention+relevance framework.

This concern is reinforced by other prior work. Gradient-based saliency has already been used for token contribution estimation and adaptive token removal in Transformers(^). More directly, recent ViT pruning work has explicitly combined attention with class-output gradient sensitivity for token importance: the paper(^^) defines an attention-gradient importance score by combining normalized attention weights with the magnitude of the gradient of the class output with respect to token representations, and uses the resulting scores for top-k/fusion token pruning in DeiT. Finally, systematic analysis has shown that simple Top-K pruning itself is already a strong baseline for ViT token reduction.

Consequently, the remaining design choices in GradSelect currently look more like heuristic refinements and recombinations of established components than a fundamentally new algorithmic idea. The empirical finding that this particular recipe performs well under the authors' controlled exact-K protocol may be interesting, but empirical superiority of one combination of saliency signals does not by itself establish strong methodological novelty.

I therefore think the paper needs a much stronger argument for what is fundamentally new beyond replacing one backward relevance mechanism with another.

At the same time, I have some questions regarding the mathematics. Please refer to the "Questions For Authors" section.

(^) Liu Y, Wang T, Zhang X, et al. Multi-dimensional collaborative pruning: optimizing vision transformers via token, attention head and FFN neuron[J]. Journal of King Saud University Computer and Information Sciences, 2026, 38(5): 197.

(^^) Modarressi A, Mohebbi H, Pilehvar M T. Adapler: Speeding up inference by adaptive length reduction[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 1-15.

(^^^) Zhang Y, Wei L, Freris N. Synergistic patch pruning for vision transformer: Unifying intra-& inter-layer patch importance[C]//International Conference on Learning Representations. 2024, 2024: 34095-34113.

Questions for authors
  • How does the proposed positive-support score follow from the stated first-order Taylor formulation under an exact-K constraint? The paper defines $ u_i=\left\langle z_i,\nabla_{z_i}t(x)\right\rangle $ and then uses gi+=max⁡(0,ui)g_i^+=\max(0,u_i) for Top-K selection. However, for scalar gates mim_i with an exact cardinality constraint ∑imi=K\sum_i m_i=K, the first-order approximation is $ t(m)\approx t(\mathbf 1)+\sum_i(m_i-1)u_i, $ for which the corresponding linear exact-K selection problem would rank tokens by the signed $u_i$ not by max⁡(0,ui)\max(0,u_i). In particular, when fewer than KK tokens have positive support, the proposed clipping makes all remaining candidates tied at zero, although the Taylor objective would still distinguish among negative-support tokens. What is the theoretical justification for discarding the sign of negative uiu_i, and should the positive clipping therefore be viewed as an empirical heuristic rather than the solution to the first-order exact-K optimization problem?

  • How is GradSelect fundamentally different from existing attention–relevance or attention–gradient token pruning methods? Could the authors clarify what new principle or property is introduced beyond this replacement and the particular score fusion? As a paper focused on interpretability and explainable AI, I would like to see insightful analysis rather than purely empirical experiments.

Reviewer_Gk2AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GradSelect, a patch selection method for vision transformers. Unlike prior approaches that rank patches through pruning schedules or routing modules, GradSelect first runs the full model and then uses backpropagation to rank patches by whether they preserve first-order support. The authors further combine this gradient-based ranking with CLS attention to improve performance. Experiments on ImageNet-1K, Food-101, and SUN397 with CLIP, DINOv2, and DeiT backbones show that GradSelect outperforms existing methods.

Critical strengths

The proposed approach propagates gradients from the predicted class and selects tokens based on first-order gate sensitivity and positive support. This appears to be a different approach from existing token selection methods. In addition, the weight λ\lambda for combining this gradient-based evidence with CLS attention is determined on images not used for testing and then fixed, which avoids tuning on the test set. The experimental results show clear performance improvements over existing methods. In particular, Table 2 shows that applying the proposed method to CLIP and DINOv2 improves performance over existing ranking methods.

Critical weaknesses
  • The proposed method requires a full forward pass and one backward pass. As shown in Table 5, this increases the computational cost by about 1.5–1.6 times or more. Table 1 shows a clear improvement over existing methods, but the gain over methods that use CLS attention is relatively small. Combining existing token selection methods with CLS attention might yield a similar effect. In addition, beyond RepShift, DivPrune, and TRIO-NMS, comparisons with a wider range of recent token selection methods are needed.
  • As shown in Table 2, the proposed method does not consistently improve performance on DeiT. This suggests that the method may be effective only for certain foundation models. Additional experiments are needed to clarify this.
Questions for authors
  1. The proposed method requires a full forward pass and one backward pass, which increases the computational cost by about 1.5–1.6 times. In what practical scenarios does the accuracy gain justify this additional cost? Could the authors compare accuracy against existing methods under the same computational budget?
  2. The improvement over methods that use CLS attention appears relatively small. If existing token selection methods were also combined with CLS attention, would the proposed method still show a clear advantage? Could the authors also compare with a wider range of recent token selection methods?
  3. In Table 2, the proposed method does not consistently improve performance on DeiT. What do the authors think causes this difference across backbones? Could they provide additional experiments on other backbones to show whether the method generalizes beyond specific foundation models?
Reviewer_yLiNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GradSelect, a method for deciding which image patches to keep in a ViT. In prior work, pruning location and the number of kept patches differ across methods, making it hard to fairly compare patch selection criteria alone. The authors fix the setting so that every method keeps the same number of patches at the same location. GradSelect computes how much each patch contributes to the model’s final prediction using backpropagation, and combines this with CLS attention, which reflects how much the model attends to each patch, to rank the patches. It achieves the best result in all 9 main settings, but when additional strong baselines are included, it is best in 6 of 9. However, it requires a full forward and backward pass before selecting patches, so it does not speed up actual inference.

Critical strengths
  1. All methods keep the same number of patches at the same location, so the patch selection criteria themselves can be compared.
  2. The evaluation covers 9 settings, combining CLIP, DINOv2, and DeiT with three image datasets.
  3. The method is compared against a range of criteria, including attention-only methods, gradient-based methods, and methods that measure importance by actually removing patches one at a time.
  4. The authors openly report the DeiT results where GradSelect does not perform well, as well as the actual computational cost.
  5. Some experiments use statistical tests to check whether small accuracy differences are due to chance.
Critical weaknesses
  1. A full forward and backward pass is required before selecting patches. In its current form, the method therefore cannot reduce total computation or speed up inference.
  2. The core method is a relatively simple combination of gradient-based contribution and CLS attention. It is hard to see it as a fundamentally new principle for patch selection.
  3. The details of the scoring method were chosen while looking at the final results of 4 of the 9 main settings. Therefore, not all results can be seen as fully independent validation.
  4. GradSelect’s advantage is not consistent across all ViT models. In particular, on all three datasets with DeiT, the method that measures importance by removing patches one at a time outperforms GradSelect.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.