acceptodds
Under review as a conference paper at ICLR 2027

Saevia: Spatiotemporal Token Compression with Attention Compensation for Efficient Video Understanding

Abstract

Video large language models (VideoLLMs) have demonstrated strong capabilities in video understanding, yet processing numerous visual tokens incurs high inference latency and limits practical deployment. Existing token compression methods reduce temporal and spatial redundancy but often overlook compression-induced attention distortion, which can impair accuracy. We introduce Saevia, a training-free framework that combines spatiotemporal token compression with attention compensation. Specifically, position-wise temporal merging reduces temporal redundancy at each spatial position. Request-aware spatial pruning selects tokens by jointly considering visual salience and request relevance. During LLM prefill, sparsity-aware attention compensation mitigates the attention distortion caused by compression. Across three VideoLLMs and five video understanding benchmarks, Saevia preserves 95.3–99.2% of full-token average accuracy at 10% visual token retention. With dedicated Triton kernels and a FlashAttention-compatible attention compensation design, Saevia achieves 2.6–3.2× time-to-first-token (TTFT) speedups over full-token baselines on one NVIDIA A100 80GB GPU and up to 1.25× end-to-end throughput with 1K-token outputs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video LLMs, Inference Acceleration, Token Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Jn6DOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free spatiotemporal token compression framework for video understanding named Saevia. Specifically, Saevia consists of three core components: Position-wise Temporal Merging, Request-aware Spatial Pruning, and Sparsity-aware Attention Compensation. The Position-wise Temporal Merging and Request-aware Spatial Pruning modules make compression decisions on which visual tokens to merge, retain, or discard via similarity-based temporal grouping and spatial selection guided by visual salience. The Sparsity-aware Attention Compensation module dynamically adapts the prefill attention weights to mitigate distortion.

The experimental evaluation in this paper assesses the effectiveness of Saevia across multiple benchmark datasets, including MVBench, Video-MME, EgoSchema, LongVideoBench, and MLVU. Results demonstrate that Saevia achieves substantially improved inference efficiency and high accuracy retention while preserving the backbone's general video understanding ability, outperforming several training-free token compression methods.

Critical strengths
  • The idea of introducing compensation within the LLM for information lost during token compression is considered novel.
  • The authors’ concern that compression alters the relative representation of different types of content in attention is reasonable.
Critical weaknesses
  • Could saliency-guided pruning compromise semantic completeness? Focusing only on the most salient regions may omit background information or interaction cues. This could affect scenarios in which the main objects interact with surroundings or when considering the relative positions of different objects.

  • The authors partition temporal sequences into segments based on the similarity between adjacent tokens at the same spatial position, using local continuity as a metric. However, the method imposes no further constraints on within-group variance, the maximum pairwise distance, or the similarity between each original token and the final one. It is quite possible for each individual step to involve only a small change while the segment as a whole undergoes a substantial state change.

  • The paper adjusts LLM attention logits using the number of merged tokens and the total attention mass before and after pruning. This appears to rely on the assumption that visual attention can approximate the contributions of discarded tokens across different LLM queries. However, attention reweighting can only influence the retained content; it cannot recover information that is absent. Does this imply that the strategy can only mitigate compression-induced shifts in attention weights, without necessarily matching the actual LLM attention that needs to be corrected?

  • Related to the previous concern, in Eq. (11), aia_i appears to be derived solely from the vision encoder, and all retained tokens assigned to the same frame share this factor. This again seems to assume that visual attention alone can model the contributions of discarded tokens across different LLM queries. While in practice, the actual contribution of a visual token also depends on the query, attention head, context, and hidden states.

Questions for authors
  • How do the authors handle positional encoding after compression? Could token merging and reordering affect the positional encoding?
  • Have the authors considered comparing their method with additional training-free approaches, such as LLaVA-Scissor [1], DyCoke [2], and PruneVID [3]?

[1] LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

[2] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models.

[3] PruneVid: Visual Token Pruning for Efficient Video Large Language Models.

Reviewer_FjctOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Saevia is a training-free framework for reducing video-LLM inference cost. It first averages similar tokens at corresponding spatial positions across adjacent frames, subject to temporal-span and token-budget constraints. It then selects tokens using vision attention and relevance to the question and answer choices. During LLM prefill, it adds logarithmic temporal-count and spatial-attention-mass corrections through a FlashAttention-compatible construction. Evaluation covers three backbones and five benchmarks. At 10% visual-token retention, the authors report 95.3–99.2% preservation of full-token average accuracy and 2.6–3.2× TTFT improvements on a fixed A100 workload.

Critical strengths
  1. Broad accuracy evaluation under aggressive compression. Tables 1–2 show 99.22%, 96.48%, and 95.31% retained average accuracy at 10% retention. The complete five-budget results, duration breakdowns, and matched-budget component ablations provide substantive evidence. The appendix also acknowledges settings where Saevia does not lead.

  2. A practical implementation with measured latency gains. Table 3 includes vision encoding, token processing, and LLM prefill. Appendix B explains how compensation preserves FlashAttention compatibility without materializing the full attention matrix. The reported TTFT gains are meaningful for the stated workload. Appendix E.3 appropriately limits them to a fixed video rather than benchmark-wide latency estimates.

Critical weaknesses
  1. The distinctive compensation contribution remains insufficiently isolated. Tables 4 and 16 show positive joint-compensation gains of 0.56, 0.59, and 0.19 accuracy points at 15% retention. However, both temporal and spatial corrections are disabled together. This does not establish the contribution of the proposed spatial attention-mass ratio beyond the temporal multiplicity correction already attributed to ToMe. Equation (11) rescales a vision-attention proxy; it does not establish recovery of query-, head-, or layer-dependent LLM attention. A temporal-only control on identical compressed sequences would directly test the additional contribution.

  2. The comparative evaluation leaves consequential gaps. UniComp (CVPR 2026) evaluates pre-LLM compression on the same LLaVA backbones at overlapping frame counts and retention budgets. EarlyTom (CVPR 2026), already cited, directly reduces vision-encoder latency. Both are relevant to assessing the complete system’s contribution. Their published scores cannot be imported directly because protocols differ. A matched comparison matters particularly because the 10% average-accuracy gains over FlashVID are 0.87, 0.40, and 0.13 points, while custom implementation optimizations also contribute to the latency advantage.

Questions for authors
  1. Evidence, With compressed features and retained token indices fixed, how does temporal-only compensation compare with joint compensation? Does the attention-mass spatial factor improve over a simpler count-based spatial correction at the headline compression budget?

  2. Evidence, Can you compare Saevia with UniComp and EarlyTom on one shared LLaVA backbone using identical frame samples, resolution, prompts, precision, compatible attention backends, and timing boundaries? Accuracy–TTFT results across a small budget range would clarify the practical contribution.

  3. Clarification, How were configurations C1 and C2 selected? This matters for Qwen3-VL, where C1 gives 65.43 average accuracy at 10% retention, versus 65.79 for the reported C2 configuration and 65.66 for FlashVID. Please also clarify the interpretation of the Qwen3-VL accuracy–latency comparison, given the disclosed 128-frame accuracy and 64-frame timing workloads.

Reviewer_7qxmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The number of visual tokens in video LLMs makes inference costly. To tackle this, token compression methods have shown interesting results; however, most of these techniques overlook the attention distortion caused by compression, and many rely on attention scores whose extraction may require switching away from FlashAttention. The authors introduce a plug-and-play spatiotemporal method with attention compensation that remains compatible with FlashAttention. First, position-wise temporal merging merges, for each position, similar tokens along the temporal axis, with a maximum-span constraint in order to avoid distorting the temporal distribution; cuts are then added to this merging in case the video is too "static", which would lead to a harder compression than the fixed ratio. Second, request-aware spatial pruning computes the average cosine similarity between a visual token and the text tokens (excluding the system prompt) and constructs a vector q_n for a frame across all the spatial positions, taking into account the sum of the attention of the tokens in the same frame. It then performs a request-aware re-ranking by combining the attention score and the request-relevance score of each candidate, giving more weight to attention for the most salient tokens and progressively more to request relevance toward the selection boundary. The final budget is allocated across frames according to their remaining candidate counts after temporal merging, and each frame keeps its top candidates according to this combined score.

Finally, sparsity-aware attention compensation uses multiplicities (the cardinality of the merged unit for temporal merging, and the ratio of total to retained attention mass in the frame for spatial pruning) to reattribute to the retained tokens the attention lost through compression, by adding the logarithms of these multiplicities to the attention logits before the softmax. In the evaluation, the authors claim the top average accuracy among the compared compression methods (FastV, VisionZip, SparseVLM, FlashVID) across 5 benchmarks and 3 models, as well as an acceleration of up to 3.17 times (LLaVA-OneVision-7B) compared to the full-token model.

Critical strengths

Critical Strengths

  • The methode is well situed in the litterature, the attention distortion in the token compression litterature needs probably more comprehensive studies and methodoligical solutions -> extention of ToMe’s proportional attention and the effect is confirmed in the ablation table 4.

  • Good results across benchmarks and models, both in accuracy and acceleration: at 10% retention ratio, Saevia keeps 99.22% (LLaVA-OneVision-7B), 96.48% (LLaVA-Video-7B) and 95.31% (Qwen3-VL-8B) of the full-token accuracy, with the best TTFT among the compared methods on the three models (Table 3: 199.95 ms / 3.17×, 461.97 ms / 2.62× and 1873.54 ms / 2.91×, respectively).

  • Hyperparameter analysis shows good stability, with at most 1.08 percentage points across the 3 models and the different retention ratios (reinforcing the plug-and-play argument).

  • Convenient to use: training-free, without breaking FlashAttention compatibility.

  • Method comparison seems fair (Appendix D).

Critical weaknesses
  • Gains are moderate. Without a naive baseline such as random token selection, it is difficult to assess whether this added complexity is fully justified.

  • Even if the method seems to work well, its conceptual novelty remains rather limited: it builds on elements already well known in token compression (proportional attention from ToMe, attention-based token selection; request relevance from text tokens, as in SparseVLM). The contribution lies mainly in their combination and improvement.

  • Limit of the position-wise temporal merging: The merging is performed at a fixed spatial position, an approach that FlashVID, used as a baseline, calls into question for videos where objects/camera are moving. The paper could discuss this limitation and its possible effect on the balance between temporal merging and spatial pruning.

  • Code unavailable, mitigated by the presence of the pseudocode, hyperparameters, prompt and other implementation details.

Questions for authors
  • Q1. Wen et al. (2025), “Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?”, show that random token selection can outperform carefully designed pruning methods on image benchmarks, notably FastV and SparseVLM. Could you report random token selection (uniformly sampled across the sequence) as a before-LLM baseline on Qwen3-VL-8B-Instruct, at a 20% retention ratio, across the five benchmarks, with both accuracy and efficiency?

  • Q2. Part of the method consists of merging tokens at the same position over time based on their similarity: how does the method behave on dynamic videos (where objects are moving a lot or the camera angle is dynamic, for example)? Could the temporal merging become very poor and place the burden on the spatial pruning, instead of tracking the movement (as FlashVID is trying to do)? Could you please look at the 20% of MVBench videos with the lowest average inter-frame similarity, then compare your method’s accuracy on this subset to the uncompressed model and FlashVID (it would be particularly relevant with your most aggressive retention ratio: 10%)?

  • Q3. Does the random seed have an impact on the results in your pipeline (reset to 3407 for each request, Appendix D)? Could you rerun the evaluation with seed=42 on MVBench, for Saevia and FlashVID at 10% retention ratio, to check the robustness of the results?

  • Q4. How do you explain that the visual token processing time advantage disappear on Qwen3-VL-8B (“Token Proc.”, Table 3: 122.14 ms for Saevia vs 120.78 ms for FlashVID, whereas Saevia is much faster on the LLaVA models)?

If the authors address these concerns, I would be willing to increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.