- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GradSelect, an output-aware patch ranking method for frozen Vision Transformers under a fixed cut and an exact-K token budget. GradSelect computes the gradient of the full model’s predicted-class log probability with respect to patch representations, uses the resulting activation-gradient product as a positive-support score, and fuses it with CLS-to-patch attention through per-image normalization and a learned coefficient. The paper evaluates this ranking rule under a controlled exact-K intervention across ImageNet-1K, Food-101, SUN397, and three ViT backbones, together with stress tests, transfer evaluation, fidelity analysis, and computational-cost diagnostics. The main results show that GradSelect achieves the best point estimate in all nine primary cells, although its advantage is weaker against stronger attribution/removal baselines in several settings.
Critical strengths
Well-controlled experimental protocol. The paper carefully fixes the cut, token budget, suffix, classifier/probe, and evaluation split, allowing a relatively clean comparison of patch-ranking rules rather than confounding ranking quality with different pruning systems.
Broad and systematic evaluation. The experiments cover three datasets and three different frozen ViT representations, and additionally include stress tests, transfer experiments, retention/cut-depth analyses, and exact-removal diagnostics. This provides useful evidence about both the strengths and limitations of the proposed selector.
Reasonable output-aware formulation and transparent limitations. Using the full model’s own predicted class as the target is a natural model-fidelity objective, and the paper is relatively careful to distinguish the proposed first-order score from true causal token removal and to acknowledge that the method is an offline selector rather than an online accelerator.
Critical weaknesses
Limited methodological novelty and insufficient positioning against closely related token-attribution pruning methods. My main concern is that the core GradSelect construction appears to be a relatively straightforward recombination of established token-importance mechanisms, and the paper does not yet make a sufficiently strong case for what fundamentally new principle is introduced.
A particularly relevant prior work is STAR (^^^), which already proposes synergistic patch pruning by combining two complementary views of patch importance: an online intra-layer importance derived from the CLS-to-patch attention map and an offline inter-layer relevance score obtained through backward Layer-wise Relevance Propagation (LRP). These scores are fused and used for top-k patch pruning across layers. Thus, the high-level idea of combining a forward attention-based signal with a backward output-relevance signal to rank patches is already present in the literature.
GradSelect replaces STAR's backward relevance signal with an instance-specific first-order score, $ \left\langle z_i,\nabla_{z_i}\log p(\hat y_{\mathrm{full}}\mid x)\right\rangle, $ applies positive clipping and per-image normalization, and linearly fuses it with CLS attention before exact-K Top-K selection. I agree that this is a reasonable and potentially useful refinement, but it is not yet clear that it constitutes a substantially new pruning principle rather than a new choice of backward attribution operator within an already established attention+relevance framework.
This concern is reinforced by other prior work. Gradient-based saliency has already been used for token contribution estimation and adaptive token removal in Transformers(^). More directly, recent ViT pruning work has explicitly combined attention with class-output gradient sensitivity for token importance: the paper(^^) defines an attention-gradient importance score by combining normalized attention weights with the magnitude of the gradient of the class output with respect to token representations, and uses the resulting scores for top-k/fusion token pruning in DeiT. Finally, systematic analysis has shown that simple Top-K pruning itself is already a strong baseline for ViT token reduction.
Consequently, the remaining design choices in GradSelect currently look more like heuristic refinements and recombinations of established components than a fundamentally new algorithmic idea. The empirical finding that this particular recipe performs well under the authors' controlled exact-K protocol may be interesting, but empirical superiority of one combination of saliency signals does not by itself establish strong methodological novelty.
I therefore think the paper needs a much stronger argument for what is fundamentally new beyond replacing one backward relevance mechanism with another.
At the same time, I have some questions regarding the mathematics. Please refer to the "Questions For Authors" section.
(^) Liu Y, Wang T, Zhang X, et al. Multi-dimensional collaborative pruning: optimizing vision transformers via token, attention head and FFN neuron[J]. Journal of King Saud University Computer and Information Sciences, 2026, 38(5): 197.
(^^) Modarressi A, Mohebbi H, Pilehvar M T. Adapler: Speeding up inference by adaptive length reduction[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 1-15.
(^^^) Zhang Y, Wei L, Freris N. Synergistic patch pruning for vision transformer: Unifying intra-& inter-layer patch importance[C]//International Conference on Learning Representations. 2024, 2024: 34095-34113.
Questions for authors
-
How does the proposed positive-support score follow from the stated first-order Taylor formulation under an exact-K constraint? The paper defines $ u_i=\left\langle z_i,\nabla_{z_i}t(x)\right\rangle $ and then uses for Top-K selection. However, for scalar gates with an exact cardinality constraint , the first-order approximation is $ t(m)\approx t(\mathbf 1)+\sum_i(m_i-1)u_i, $ for which the corresponding linear exact-K selection problem would rank tokens by the signed $u_i$ not by . In particular, when fewer than tokens have positive support, the proposed clipping makes all remaining candidates tied at zero, although the Taylor objective would still distinguish among negative-support tokens. What is the theoretical justification for discarding the sign of negative , and should the positive clipping therefore be viewed as an empirical heuristic rather than the solution to the first-order exact-K optimization problem?
-
How is GradSelect fundamentally different from existing attention–relevance or attention–gradient token pruning methods? Could the authors clarify what new principle or property is introduced beyond this replacement and the particular score fusion? As a paper focused on interpretability and explainable AI, I would like to see insightful analysis rather than purely empirical experiments.