acceptodds
Under review as a conference paper at ICLR 2027

SAVOR: Sample-Aware Video Token Reduction with Native Variable-Length Execution

Abstract

Video Transformers spend much of their computation on temporally redundant tokens: nearby frames are often very similar, and static backgrounds do not change across frames. Many token-reduction methods either keep the same number of tokens for every sample, or keep masked tokens in the computational graph during training. The former ignores per-sample redundancy, while the latter does not shorten the actual sequences that are fed to attention layers. To address these issues, we propose Sample-Aware Video tOken Reduction (SAVOR), which operates on a variable-length transformer backbone. It flattens all clips in a batch into one sequence with cumulative lengths and uses a per-sample attention mask to prevent cross-sample attention, so that the removed tokens can be physically deleted during both training and inference. SAVOR combines three components: a deterministic PreMerge that merges redundant temporally adjacent tokens, a CLS-conditioned scorer that keeps the highest-scoring tokens, merges low-scoring neighbors, and drops the rest, and a Hierarchical FLOPs Allocator that turns a global FLOPs target into per-sample, per-layer retention ratios. We evaluate SAVOR on two video benchmarks, i.e., EPIC-KITCHENS-100 (EK-100) and Kinetics-400 (K400). On EK-100 multi-instance retrieval, SAVOR reduces the analytical encoder FLOPs proxy by 30.3%, achieves 60.74 mAP versus 60.71 for the dense backbone, and improves nDCG by 0.37 percentage points. Across the EK-100 target sweep, the realized proxy ratio differs from the target by less than 2% in relative terms. Code is released at https://anonymous.4open.science/r/SAVOR.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Deep Learning, Computer Vision, Video Understanding, Token Pruning, Model Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UUGgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SAVOR, a video token reduction method. SAVOR combines similarity based merging, importance-based token selection and adaptive compute allocation across samples and layers. SAVOR uses a variable-length backbone that removes tokens during training and inference. Experiments on Epic Kitchens and Kinetics shows reduced FLOPs with relatively small performance losses.

Critical strengths
  1. On Epic Kitchens in particular, SAVOR shows strong accuracy-compute tradeoff. At 0.7 target, SAVOR slightly outperforms the dense baseline. There are also competitive results on Kinetics and EGTEA as well as additional baselines.
  2. The variable length execution is a practical way to implement sample-dependent token reduction without having to keep removed tokens in the deeper transformer blocks. The method also closely tracks its requested FLOPs budget supporting transferring a trained model to nearby compute budgets without retraining. The ablation study provides evidence for the importance of the proposed token scorer. Removing the attention oracle reduces mAP from 60.02 to 58.14 on Epic Kitchens. The authors also report training throughout improvements, reaching about 1.25 x at about half the encoder FLOPs.
Critical weaknesses
  1. One big concern I have is the lack of comparison to highly related works. For example, LITE [1] and Flux [2] have been completely omitted despite solving the same task and also in a very similar fashion. These are particularly important omissions as LITE already uses learned token importance and LITE++ uses adaptive compute budget based on the video. LITE also reports comparable performance with the dense backbone used at very similar ratios to the ones reported by SAVOR. Flux studies flexible video-token training and deployment under different compute budgets which also follows similar core principles. Whilst LITE uses VideoMAE and Flux uses VideoIntern, direct comparisons results wise is not possible. So, an additional potential experiment to try is using VideoMAE as a backbone for SAVOR and compare against LITE. The bare minimum would be at least a clear distinction from both LITE and Flux.
  2. Another point that is currently not very clear to me is the gains from the proposed components. For example, the contribution of the adaptive budget allocator is not sufficiently isolated. The main fixed-anchor ablation removes both sample-dependent and layer-dependent budget variation at once. Although the fixed-anchor performs worse, this does not show whether the improvement comes from adapting budgets across samples or layers or both. The paper also lacks comparisons with simpler budget-allocation strategies such as fixed layer schedules combined with sample-dependent retention. Also, the evaluation of the temporal understanding remains limited. Most experiments focus on Epic Kitchens and Kinetics and additional EGTEA results on only one official split. Sth-Sth v2 [3] is missing and is generally a standard benchmark for methods that need to preserve fine-grained temporal information. LITE and other related methods compare on it. Since SAVOR explicitly merges tokens across neighbor frames, testing subtle motion and temporal order information is important. Whilst I do not expect experiments on a completely new dataset in time for the rebuttal, I hope this can be incorporated in the eventual final version.
  3. The practical efficiency gains are much smaller than the claimed FLOP reductions. For example, at 0.7 target SAVOR reduces encoder FLOPs by 30.3% but actual training throughput improves by only 6% (18.6 to 19.7 vids/sec). More importantly, the paper does not provide actual inference latency even though it is mentioned as one of the main motivations. Existing works like ToMe, TempMe, LITE etc all provide this. Also, the FLOPs calculation leaves out parts like the matching, repacking and the allocator which is a bit unfair to compare against. Another minor point in terms of the numbers itself is that the improvements are too small for a meaningful conclusion. For example, on Kinetics-400 under the 5x3 at 0.7 target SAVOR achieves 83.43% compared with 83.37% for Pollard and Wray at nearly identical estimated compute. There isn't enough repeated runs to establish statistical significance.

[1] Hao, X., Li, G., Gowda, S.N., Fisher, R.B., Huang, J., Arnab, A. and Sevilla-Lara, L., 2025, October. Principles of visual tokens for efficient video understanding. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 21254-21264). IEEE. [2] Wang, C., Li, K., Jiang, T., Zeng, X., Wang, Y. and Wang, L., 2025, October. Make your training flexible: Towards deployment-efficient video models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 23880-23891). IEEE. [3] Goyal, R., Kahou, S.E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M. and Hoppe, F., 2017, October. The “something something” video database for learning and evaluating visual common sense. In 2017 IEEE international conference on computer vision (ICCV) (pp. 5843-5851). IEEE.

Questions for authors
  1. Can the authors compare against LITE/LITE++ using the same backbone and compute budget? What improvement does SAVOR provide? This is not a defining question for me, but would help me in changing my rating.
  2. More importantly, can the authors distinguish their work with LITE/LITE++ and Flux?
  3. What is SAVOR's inference latency, throughput and peak allocated memory in comparison to ToMe, TempMe and other baselines using the same hardware?
  4. Can the authors separately ablate sample-dependent allocation and layer-dependent allocation and compare their allocator against simpler budget schedules?
  5. Are the small improvements consistent across multiple seeds?
  6. How does SAVOR perform on fine-grained temporal reasoning tasks? Again not expecting results for this given the short rebuttal response time, but adding it here as a potential useful question to think about.
Reviewer_XbUfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a method called SAVOR. To be honest, the design of SAVOR is a bit complex. Based on my understanding, it merges similar temporal neighbors, scores the remaining tokens and chooses retention rates. The authors also emphasize that they "physically" process the tokens for encoder blocks.

Critical strengths
  1. The authors add token removal to both training and inference, so the model can be accelerated in both.
  2. Across the several benchmarks, SAVOR keeps accuracy level.
Critical weaknesses

I suggest that the authors enhance the clarity of the writing, because the paper is not easy to follow and still contains mistakes:

  1. The authors said DynamicViT is a one-time selection. This is incorrect because DynamicViT uses hierarchical pruning at multiple layers. DynamicViT also has a learned prediction module. The oversimplified description is not suitable.
  2. The work didn't mention or compare with many recent token reduction works. For example, Dycoke from CVPR 2025, SURGE from ICLR 2026, and especially Earlytom from CVPR 2026, which performs training-free compression inside the vision encoder. The missing discussion and comparison make your novelty unclear.
  3. I don't think dropping the softmax in Eq. 11 "does not affect ranking". Have you considered multiple-head scenarios? For example, head one logits are (10, 0, 11) and head two logits are (0, 2, 0), the rank with and without softmax are reversed. I think this is a critical issue since parts of the method are based on this.
  4. Another inconsistency I found is in table 2 and table 8. Initially, I thought the attention cost is quadratic and your mean token counts may underestimate the actual average cost. But when I substitute the table 8 token counts into your table 13 equation, I get 0.502 but table 2 gives 0.4946. Please clarify where this difference comes from.
Questions for authors

Please answer and address the questions I raised in the weaknesses.

Reviewer_XkotOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAVOR reduces video-encoder computation through sample-aware token reduction and packed variable-length execution during both training and inference. It combines temporal similarity-based merging, a learned importance scorer for token retention and reduction, and a hierarchical allocator that assigns per-sample, per-layer budgets around a global FLOPs target. Experiments evaluate retrieval and classification performance, analytical FLOPs, and training efficiency, with additional backbone and dataset evaluations in the appendix.

Critical strengths
  1. The packed variable-length implementation addresses a practical challenge in adaptive token reduction. Physically removing tokens during training and inference allows different samples to retain different sequence lengths while avoiding computation on discarded tokens.

  2. The paper provides useful empirical coverage, including comparisons with several token-reduction methods, component ablations, budget-transfer experiments, and an additional backbone. On EK-100, the method maintains retrieval performance near the dense baseline at approximately 30% lower analytical encoder FLOPs.

Critical weaknesses
  1. The additional methodological contribution over established token-reduction approaches is not sufficiently demonstrated. Similarity-based merging, importance-based pruning, and training-time token reduction already have substantial precedent. SAVOR adds a sample- and layer-dependent allocator and feedback mechanism, but their independent benefits appear modest: removing GRU feedback changes EK-100 mAP by 0.12 points, while the fixed-anchor control differs from the full method by 0.34 points. These comparisons also have different realized FLOPs ratios, making the contribution of the learned allocation harder to quantify. Stronger controlled evidence is needed to justify the additional complexity.

  2. The practical efficiency advantage over competing methods remains insufficiently established. The reported FLOPs proxy excludes several components of the reduction pipeline. At the approximately 30% FLOPs-reduction setting, training throughput increases from 18.6 to 19.7 videos/s, while reserved GPU memory decreases from 14,750 to 14,234 MiB. These are useful but relatively modest measured gains. The main efficiency table compares SAVOR with the dense backbone rather than providing matched-hardware throughput and memory comparisons against competing reducers. Consequently, the accuracy–FLOPs results do not fully establish a superior practical accuracy–cost trade-off.

Overall, I lean toward weak rejection. The implementation has practical merit, but the independent value of the additional adaptive mechanisms and their efficiency advantage over simpler alternatives require stronger evidence.

Questions for authors
  1. Could the full method and the fixed-anchor control be compared at matched realized FLOPs, with repeated-training variability reported? Their current proxy ratios are 0.4946 and 0.5147, respectively. This would help assess whether the 0.34-point mAP difference reliably supports the learned allocator.

  2. Could the authors provide training and inference throughput, together with memory measurements, for representative competing reducers on the same hardware and batch configuration? Please include the runtime costs of scoring, matching, and repacking, as well as the training-only oracle where applicable.

  3. How much of the benefit comes specifically from sample-dependent allocation, rather than from learning a sample-independent schedule across layers? The existing fixed-anchor control removes both forms of budget adaptation. A controlled comparison separating these effects would help establish the contribution beyond existing training-time token-reduction approaches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.