- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes SAVOR, a video token reduction method. SAVOR combines similarity based merging, importance-based token selection and adaptive compute allocation across samples and layers. SAVOR uses a variable-length backbone that removes tokens during training and inference. Experiments on Epic Kitchens and Kinetics shows reduced FLOPs with relatively small performance losses.
Critical strengths
- On Epic Kitchens in particular, SAVOR shows strong accuracy-compute tradeoff. At 0.7 target, SAVOR slightly outperforms the dense baseline. There are also competitive results on Kinetics and EGTEA as well as additional baselines.
- The variable length execution is a practical way to implement sample-dependent token reduction without having to keep removed tokens in the deeper transformer blocks. The method also closely tracks its requested FLOPs budget supporting transferring a trained model to nearby compute budgets without retraining. The ablation study provides evidence for the importance of the proposed token scorer. Removing the attention oracle reduces mAP from 60.02 to 58.14 on Epic Kitchens. The authors also report training throughout improvements, reaching about 1.25 x at about half the encoder FLOPs.
Critical weaknesses
- One big concern I have is the lack of comparison to highly related works. For example, LITE [1] and Flux [2] have been completely omitted despite solving the same task and also in a very similar fashion. These are particularly important omissions as LITE already uses learned token importance and LITE++ uses adaptive compute budget based on the video. LITE also reports comparable performance with the dense backbone used at very similar ratios to the ones reported by SAVOR. Flux studies flexible video-token training and deployment under different compute budgets which also follows similar core principles. Whilst LITE uses VideoMAE and Flux uses VideoIntern, direct comparisons results wise is not possible. So, an additional potential experiment to try is using VideoMAE as a backbone for SAVOR and compare against LITE. The bare minimum would be at least a clear distinction from both LITE and Flux.
- Another point that is currently not very clear to me is the gains from the proposed components. For example, the contribution of the adaptive budget allocator is not sufficiently isolated. The main fixed-anchor ablation removes both sample-dependent and layer-dependent budget variation at once. Although the fixed-anchor performs worse, this does not show whether the improvement comes from adapting budgets across samples or layers or both. The paper also lacks comparisons with simpler budget-allocation strategies such as fixed layer schedules combined with sample-dependent retention. Also, the evaluation of the temporal understanding remains limited. Most experiments focus on Epic Kitchens and Kinetics and additional EGTEA results on only one official split. Sth-Sth v2 [3] is missing and is generally a standard benchmark for methods that need to preserve fine-grained temporal information. LITE and other related methods compare on it. Since SAVOR explicitly merges tokens across neighbor frames, testing subtle motion and temporal order information is important. Whilst I do not expect experiments on a completely new dataset in time for the rebuttal, I hope this can be incorporated in the eventual final version.
- The practical efficiency gains are much smaller than the claimed FLOP reductions. For example, at 0.7 target SAVOR reduces encoder FLOPs by 30.3% but actual training throughput improves by only 6% (18.6 to 19.7 vids/sec). More importantly, the paper does not provide actual inference latency even though it is mentioned as one of the main motivations. Existing works like ToMe, TempMe, LITE etc all provide this. Also, the FLOPs calculation leaves out parts like the matching, repacking and the allocator which is a bit unfair to compare against. Another minor point in terms of the numbers itself is that the improvements are too small for a meaningful conclusion. For example, on Kinetics-400 under the 5x3 at 0.7 target SAVOR achieves 83.43% compared with 83.37% for Pollard and Wray at nearly identical estimated compute. There isn't enough repeated runs to establish statistical significance.
[1] Hao, X., Li, G., Gowda, S.N., Fisher, R.B., Huang, J., Arnab, A. and Sevilla-Lara, L., 2025, October. Principles of visual tokens for efficient video understanding. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 21254-21264). IEEE. [2] Wang, C., Li, K., Jiang, T., Zeng, X., Wang, Y. and Wang, L., 2025, October. Make your training flexible: Towards deployment-efficient video models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 23880-23891). IEEE. [3] Goyal, R., Kahou, S.E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M. and Hoppe, F., 2017, October. The “something something” video database for learning and evaluating visual common sense. In 2017 IEEE international conference on computer vision (ICCV) (pp. 5843-5851). IEEE.
Questions for authors
- Can the authors compare against LITE/LITE++ using the same backbone and compute budget? What improvement does SAVOR provide? This is not a defining question for me, but would help me in changing my rating.
- More importantly, can the authors distinguish their work with LITE/LITE++ and Flux?
- What is SAVOR's inference latency, throughput and peak allocated memory in comparison to ToMe, TempMe and other baselines using the same hardware?
- Can the authors separately ablate sample-dependent allocation and layer-dependent allocation and compare their allocator against simpler budget schedules?
- Are the small improvements consistent across multiple seeds?
- How does SAVOR perform on fine-grained temporal reasoning tasks? Again not expecting results for this given the short rebuttal response time, but adding it here as a potential useful question to think about.