acceptodds
Under review as a conference paper at ICLR 2027

Fast Adaptive Token Selection via Efficient Reinforcement Learning

Abstract

Vision Transformers (ViTs) have achieved remarkable success in a wide range of visual tasks due to their strong global modeling capability. However, the continuous processing of full token sequences in deep Transformer layers introduces substantial inference overhead. Existing token pruning methods often struggle to balance efficiency and task adaptability: retraining-free methods rely on fixed importance heuristics, while retraining-based methods typically require additional optimization of the full model. Therefore, a key challenge is: how to learn token selection policies with both task awareness and long-term decision-making capability without updating large-scale pretrained ViTs. To address this challenge, we propose FASTER, a lightweight reinforcement learning(RL) framework for rapid task-adaptive ViT token pruning. FASTER freezes the pretrained ViT backbone and task head, and only optimizes a lightweight Actor–Critic policy network. It introduces Task-aware Sensitivity Encoding (TSE) to transform task-related knowledge embedded in the pretrained model into a reinforcement learning state representation, and learns instance-adaptive token retention policies by jointly considering token representations, spatial information, and budget constraints. Experimental results demonstrate that FASTER effectively reduces computational cost and inference latency on ViT-L, DeiT-B, DeiT-S, and DeiT-T, achieving a competitive trade-off between accuracy and

Primary area:
reinforcement learning
Keywords:
Vision Transformer, token pruning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wLkwOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FASTER, a lightweight actor-critic policy for adaptive token pruning in ViTs. The backbone and task head stay frozen, while task-aware sensitivity features and long-horizon rewards are used to learn layer-wise retention decisions. Results on DeiT models show a reasonable accuracy-latency trade-off, and the transfer experiments suggest that the learned policy is not limited to ImageNet classification, which is important for ViT acceleration methods.

Critical strengths
  • This work investigates plenty of related works.
  • A clear motivation for treating progressive token pruning as a sequential decision problem.
  • For evaluation. this paper reports actual latency with physical token compaction, rather than only FLOPs or masked execution. Evaluation covers several backbone sizes and downstream classification and segmentation tasks.
Critical weaknesses
  • The abstract in the openreview platform is incomplete. Please avoid this mistake in future submissions.
  • The framework figure (fig.2) requires large revision efforts. Do not give the framework with such tiny text fonts. Meanwhile, it is suggested to maintain the similar font style across all figures.
  • The method still needs a calibration set, gradient-based sensitivity computation, and RL training for each pretrained backbone. The practical adaptation cost is not compared with fine-tuning-based pruning methods. Limitations should be more directly given.
  • The factorized Bernoulli policy makes independent token decisions, while the final inference procedure uses deterministic top-k selection. The gap between the training distribution and deployment rule deserves a stronger analysis. Perhaps I do not fully understand this point in the real implementation.
Questions for authors

See above.

Reviewer_ADPjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper puts forward FASTER, a reinforcement-learning framework for progressive token pruning in frozen Vision Transformers. From gradient responses on a calibration set, it computes a fixed Task-aware Sensitivity Encoding (TSE) that depends on layer and position. A lightweight actor then combines this encoding with token features, positional information, and budget cues to predict token-retention probabilities. Training relies on factorized Bernoulli actions and A3C, with rewards for representation consistency, budget consistency, and the terminal task; at inference, tokens are chosen through budget-constrained Top-k selection and physically compacted. The paper reports accuracy–efficiency improvements on DeiT backbones, comparisons with alternative RL optimizers, component ablations, and transfer of the frozen policy to downstream classification and segmentation tasks.

Critical strengths
  1. A practically relevant adaptation setting with a clear deployment path. Learning a token-selection policy while both the backbone and the task head stay frozen fits an important constraint in deployment.

  2. Broad empirical coverage and explicit data separation. The manuscript tests several model scales and budgets, compares RL optimizers, and includes component and downstream-transfer experiments.

Critical weaknesses
  1. The efficiency gains need clearer support. A few speedups in Table 4 do not follow from the latencies given: 2.15/1.41 comes to roughly 1.52× rather than 1.55×, and 9.80/8.93 to roughly 1.10× rather than 1.06×. These values should be corrected. The paper should also state how long it takes in total to prepare and train the pruning policy, since otherwise readers cannot judge the claimed efficiency.

  2. The ablations still do not make clear where the gains come from. TSE gives patch positions fixed sensitivity scores, yet no comparison with a simple positional prior is provided. Table 3 also does not explain how the policy is trained once the reward components are removed. Better descriptions and the right controls would make the benefit of TSE and of reinforcement learning easier to establish.

Questions for authors
  1. Efficiency measurements: Could you account for the speedup discrepancies in Table 4, and say whether the token-selection network is included in the reported computation?
  2. Training cost and fair comparison: How much time does it take, in total, to prepare TSE and train the policy for each backbone? Were the RL optimizers given comparable training and tuning budgets?
  3. Ablation settings: What exactly is removed in each ablation in Table 3, in particular “w/o Reward Components,” and which learning signal is left?
  4. TSE’s contribution: Is there any evidence that TSE beats a simple positional prior when the computation budget is held fixed?
Reviewer_y6M9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FASTER learns a token-selection policy while keeping the vision transformer frozen. It combines calibrated sensitivity features with actor–critic training, then uses top-k selection and physical token gathering at inference. The experiments cover ImageNet classification and downstream transfer.

Critical strengths
  1. Adapting a small policy without retraining the backbone is a useful setting. The reported latency includes the actor, selection, and gathering costs, and Table 4 shows that physical compaction produces actual speed gains.
  2. The accuracy–latency trade-off is promising across the three DeiT sizes. On DeiT-S, accuracy drops from 79.89% to 79.42% while latency falls from 5.20 to 3.30 ms; the component ablations help explain this result.
Critical weaknesses
  1. TSE reduces task sensitivity to a fixed layer–position prior (Section 3.2). Averaging squared gradients across calibration images removes instance-specific sensitivity, and the resulting guidance does not update as other tokens disappear. The actor still receives content features, but TSE itself cannot track these changes. Its ablation shows usefulness, without distinguishing task-sensitive guidance from a helpful spatial prior.
  2. Training and deployment optimize different selection rules (Section 3.3). Training samples independent Bernoulli masks with a soft count penalty; inference forces a fixed-size top-k subset. Top-k maximizes mask probability under that constraint, not task return. Appendix C explains how the penalty reduces count mismatch, but does not establish that the deployed subset preserves the benefit learned from stochastic trajectories.
  3. The reconstruction reward measures error only on surviving tokens (Eq. 5), so the action also determines what is evaluated. This can favor tokens whose features are easy to preserve, without directly measuring information lost through deletion. The terminal task reward mitigates this risk, and Table 15 supports the combined reward empirically, but the trade-off between feature consistency and task utility remains underexplained.
Questions for authors
  1. Can you distinguish TSE's contribution from a matched positional prior, especially when object locations or surviving token sets change?
  2. How does training with fixed-cardinality actions compare with Bernoulli training when both use the same top-k deployment budget?
  3. When does survivor-only reconstruction error disagree with task loss, and how is this trade-off controlled by the reward weights?
2 comments

Thank you for the detailed rebuttal. The matched positional controls clarify what task calibration adds, and fixed-cardinality training achieves comparable accuracy at the same top-k deployment budgets. The trajectory analysis also clarifies where survivor-only reconstruction diverges from task utility. These results address my main concerns, so I’m raising Soundness to 3 and my overall rating to 3

AuthorsOpenReview ↗

Thanks to Reviewer y6M9 for the Support

We sincerely thank you for your careful evaluation, constructive comments, and for the detailed assessment of our rebuttal. We are very grateful that the matched positional controls clarified the contribution of task calibration, that the fixed-cardinality training comparison addressed the training–inference consistency concern, and that the trajectory analysis clarified where survivor-only reconstruction diverges from task utility.

We especially appreciate your recognition that these results address your main concerns, and we thank you for raising Soundness to 3 and the overall rating to 3. Your support and positive assessment of the practical value of our work mean a great deal to us.

We will retain and further strengthen these clarifications in the revised manuscript, including the matched positional controls, the fixed-cardinality training comparison, and the trajectory-level analysis of reconstruction versus task utility. We will also make the corresponding limitations and design choices clearer in the final version.

Thank you again for your time, careful review, and support for our work.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.