acceptodds
Withdrawn from ICLR 2027

RAJNI: Importance-Aware, Dispersion-Scheduled Training-Free Token Pruning for Vision Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision Transformers incur inference cost that scales quadratically in the number of patch tokens, making token pruning an attractive training-free accelerator. We present RAJNI, built on two components: a forward-pass importance score that is exactly a token's contribution to its layer's CLS aggregate under ablation, and that we show upper-bounds (though does not guarantee) the token's effect on the model's output, under an alignment condition we validate empirically; and a closed-form scheduler that sets each layer's keep-ratio from the dispersion of that layer's own importance distribution, requiring no learned parameters, auxiliary networks, or fine-tuning. We validate the method with three kinds of evidence rather than a single accuracy table: agreement with exact gradient sensitivity across depth (up to Spearman correlation), a controlled ablation showing the score's two factors are individually insufficient, and a large-scale () analysis showing the scheduler's adaptivity compounds systematically with depth. On ImageNet-1K, RAJNI reaches an approximate throughput gain of 40% at an accuracy cost of 0.7% on ViT-Base, remaining close to the Pareto front of stronger training-free baselines (ToMe, SAINT) in the moderate-compression regime, and consistently ahead of GTP-ViT. Main-text throughput numbers use batch-aggregated dispersion statistics for GPU efficiency; we state this explicitly, since per-image dynamic sequence lengths would require custom sparse-attention kernels in a batched production deployment.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
vision transformers, token pruning, training-free inference, model efficiency, adaptive computation, attention mechanisms, inference acceleration, image classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zZieOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAJNI, a training-free token pruning method for ViTs. The method uses attention (that it gotten from CLS token) and value information to estimate token importance, and then uses an adaptive scheduler to decide how many tokens to keep at each layer. The results are reasonable at moderate compression rates, but I have some concerns regarding the novelty and experimental evaluation.

Critical strengths
  1. The method is simple and does not require any additional training.
Critical weaknesses
  1. I am not quite convinced about the novelty of the approach. The importance score seems very similar to what was already proposed in ATS, and adaptive token pruning has been explored quite extensively before. So ultimately, the main contribution seems to be the particular heuristic used for scheduling, which I don't think is substantial enough on its own.

  2. The experimental evaluation is quite limited. The authors only consider ImageNet classification with two relatively small backbones. Given how much work has already been done in this space, I would expect experiments on larger backbones, different architectures, and potentially other tasks such as dense prediction or VLMs. Moreover, the improvements over ToMe and SAINT are not particularly convincing, especially at higher compression rates.

  3. One of the main motivations of the paper is to adapt the number of tokens based on the input image. However, the throughput experiments are done using batch-level statistics. So I am not sure how much of the proposed per-image adaptivity is actually reflected in the reported results. I think this needs to be clarified, and ideally both settings should be evaluated.

  4. I just think that this idea is quite outdated for ICLR 2027. Most of the literature and backbones in this setup are from around 2021-2023. That's when these setups were valuable to study, and given that such methods are supposed to be practical inference methods, I do not think it is a good fit as they have lost their market.

Questions for authors
  1. Why are the authors still working on models like DeiT-S or basic ViTs? why not more interesting setups like VLMs or even generative models or unified ones -- if compute is a challenge, maybe only focus on the very small models in those setups
Reviewer_p4xbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAJNI, a training-free token pruning method for ViTs. Each token is scored by the product of its CLS attention and a normalized value magnitude, and the keep-ratio at each layer is set by how spread out the scores are. On ImageNet-1K, it reports 1.39x throughput at -0.73pp accuracy on ViT-B/16 and 1.36x at -0.97pp on DeiT-S/16.

Critical strengths
  1. The method is lightweight and can be applied directly to pretrained models without changing the weights.
  2. Adapting the keep-ratio to the image and to layer depth is a reasonable idea, and the results are competitive in the moderate-compression range on ViT-B.
Critical weaknesses
  1. The theory does not match the score actually used. Proposition 1 is a simple identity, and Proposition 2 is a first-order approximation, not a bound. The score actually used (head-averaged values, deviation from the mean, sigmoid) is not A*||V|| and does not preserve its ranking. Neither proposition therefore applies to the method as implemented.
  2. The speed and adaptivity results come from different settings. Throughput is measured by pooling statistics over the batch, so every image in a batch gets the same keep-ratio, while the per-image behavior in Fig. 6 can only come from a per-image setting. It is not stated which setting produced Tables 1 and 2.
  3. Novelty is limited and related work is missing. Measuring token contribution by attention weight times value norm was already proposed by Kobayashi et al. (EMNLP 2020), which is not cited. The gain over attention-only is about 0.3pp (Fig. 5), and there is no ablation of the scheduler’s design choices (log, the 75th percentile, dividing by |q|). DiffRate (ICCV 2023), which addresses the same layer-wise compression rate problem, is neither cited nor compared. In addition, token reduction leaves much of the FFN cost in place, and discussing orthogonal approaches that reduce the FFN channel dimension, such as ToaSt (ICML 2026), would help explain RAJNI’s limited speedup range.
Questions for authors

See Weaknesses

Reviewer_WckyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes post-hoc token pruning for vision transformers using a CLS-attention score multiplied by a transformed value magnitude, together with a dispersion-based rule for choosing the retained token count at each layer. It reports ImageNet accuracy-throughput trade-offs, score and scheduler ablations, and correlations with gradient sensitivity and image complexity. The intended contribution is a training-free adaptive pruning rule with an interpretable importance score.

Critical strengths
  1. The moderate-compression results are useful: Table 1 reports 80.87% top-1 accuracy at 1.24x throughput versus an 81.10% unpruned ViT-B baseline. Table 3 isolates a 0.60-point benefit from the adaptive scheduler at approximately matched speed, using the same importance score.
  2. The paper explicitly acknowledges the batch-aggregated deployment setting and the loss of competitiveness at stronger compression. The component ablation and full-validation-set complexity analysis provide evidence beyond a single operating point.
Critical weaknesses
  1. The relation to ATS is materially mischaracterized (Sections 2 and 6, pp. 2 and 9). ATS already uses CLS attention multiplied by the value-vector norm (Fayyaz et al., ECCV 2022, Eq. 3), provides a deterministic inverse-CDF sampling scheme, and explicitly evaluates use without further training (Appendix C). Thus the 49.63% accuracy / 0.16x throughput result cannot by itself support the explanation that ATS fails because fine-tuning is intrinsically required. The paper needs to establish what the transformed score and dispersion scheduler add over a faithful training-free ATS baseline.
  2. The theoretical interpretation does not carry through to the implemented score (pp. 3-5). Proposition 1 is valid for an unnormalized soft ablation with score A_t times ||V_t||. Equations 7-9 instead average heads, center values across tokens, and apply a sigmoid; these operations do not generally preserve the ranking of A_t times ||V_t||, contrary to the statement below Eq. 6. Also, the hard-removal correction in Section 3.1.5 is generally first order in the removed attention mass, not second order. For a normalized single-head aggregate z and removed weight a, the exact change is a(v-z)/(1-a), rather than a*v. Proposition 2 bounds the linearized effect, without controlling the Taylor remainder. Correlation with input-gradient norms does not resolve these distinct gaps.
  3. The evidence for adaptive deployment is difficult to reproduce from the PDF. Sections 1 and 7 use batch-aggregated dispersion for throughput, whereas Section 5.6 discusses per-image retention; the aggregation and evaluation modes need to be connected explicitly. Section 4 and the reproducibility statement refer to one another for exact hyperparameters and calibration details, but the appendix provides retained sweep ranges, not the exact operating-point configurations, calibration split, or fixed-schedule specification (pp. 5, 8-9, 11-12). These details affect whether Table 3 isolates adaptivity and whether the reported speed/accuracy pairs describe the same procedure.
Questions for authors
  1. Please reconcile the ATS description with its Eq. 3, deterministic sampling rule, and training-free experiments. What checkpoint, insertion layers, sampling settings, and implementation produce the Section 6 result?
  2. Can the claims about ranking preservation and the hard-removal correction be corrected, and can the actual gated score be compared with A_t times ||V_t|| under the same scheduler and token budget?
  3. For Tables 1-3 and Figure 6, please specify the calibration split, exact gamma/warmup/minimum-retention settings, fixed schedule, batch aggregation rule, and whether accuracy and speed are measured under the same pruning mode.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.