- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FASTER, a lightweight actor-critic policy for adaptive token pruning in ViTs. The backbone and task head stay frozen, while task-aware sensitivity features and long-horizon rewards are used to learn layer-wise retention decisions. Results on DeiT models show a reasonable accuracy-latency trade-off, and the transfer experiments suggest that the learned policy is not limited to ImageNet classification, which is important for ViT acceleration methods.
Critical strengths
- This work investigates plenty of related works.
- A clear motivation for treating progressive token pruning as a sequential decision problem.
- For evaluation. this paper reports actual latency with physical token compaction, rather than only FLOPs or masked execution. Evaluation covers several backbone sizes and downstream classification and segmentation tasks.
Critical weaknesses
- The abstract in the openreview platform is incomplete. Please avoid this mistake in future submissions.
- The framework figure (fig.2) requires large revision efforts. Do not give the framework with such tiny text fonts. Meanwhile, it is suggested to maintain the similar font style across all figures.
- The method still needs a calibration set, gradient-based sensitivity computation, and RL training for each pretrained backbone. The practical adaptation cost is not compared with fine-tuning-based pruning methods. Limitations should be more directly given.
- The factorized Bernoulli policy makes independent token decisions, while the final inference procedure uses deterministic top-k selection. The gap between the training distribution and deployment rule deserves a stronger analysis. Perhaps I do not fully understand this point in the real implementation.
Questions for authors
See above.
Thank you for the detailed rebuttal. The matched positional controls clarify what task calibration adds, and fixed-cardinality training achieves comparable accuracy at the same top-k deployment budgets. The trajectory analysis also clarifies where survivor-only reconstruction diverges from task utility. These results address my main concerns, so I’m raising Soundness to 3 and my overall rating to 3