- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RAJNI, a training-free token pruning method for ViTs. The method uses attention (that it gotten from CLS token) and value information to estimate token importance, and then uses an adaptive scheduler to decide how many tokens to keep at each layer. The results are reasonable at moderate compression rates, but I have some concerns regarding the novelty and experimental evaluation.
Critical strengths
- The method is simple and does not require any additional training.
Critical weaknesses
-
I am not quite convinced about the novelty of the approach. The importance score seems very similar to what was already proposed in ATS, and adaptive token pruning has been explored quite extensively before. So ultimately, the main contribution seems to be the particular heuristic used for scheduling, which I don't think is substantial enough on its own.
-
The experimental evaluation is quite limited. The authors only consider ImageNet classification with two relatively small backbones. Given how much work has already been done in this space, I would expect experiments on larger backbones, different architectures, and potentially other tasks such as dense prediction or VLMs. Moreover, the improvements over ToMe and SAINT are not particularly convincing, especially at higher compression rates.
-
One of the main motivations of the paper is to adapt the number of tokens based on the input image. However, the throughput experiments are done using batch-level statistics. So I am not sure how much of the proposed per-image adaptivity is actually reflected in the reported results. I think this needs to be clarified, and ideally both settings should be evaluated.
-
I just think that this idea is quite outdated for ICLR 2027. Most of the literature and backbones in this setup are from around 2021-2023. That's when these setups were valuable to study, and given that such methods are supposed to be practical inference methods, I do not think it is a good fit as they have lost their market.
Questions for authors
- Why are the authors still working on models like DeiT-S or basic ViTs? why not more interesting setups like VLMs or even generative models or unified ones -- if compute is a challenge, maybe only focus on the very small models in those setups