- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free spatiotemporal token compression framework for video understanding named Saevia. Specifically, Saevia consists of three core components: Position-wise Temporal Merging, Request-aware Spatial Pruning, and Sparsity-aware Attention Compensation. The Position-wise Temporal Merging and Request-aware Spatial Pruning modules make compression decisions on which visual tokens to merge, retain, or discard via similarity-based temporal grouping and spatial selection guided by visual salience. The Sparsity-aware Attention Compensation module dynamically adapts the prefill attention weights to mitigate distortion.
The experimental evaluation in this paper assesses the effectiveness of Saevia across multiple benchmark datasets, including MVBench, Video-MME, EgoSchema, LongVideoBench, and MLVU. Results demonstrate that Saevia achieves substantially improved inference efficiency and high accuracy retention while preserving the backbone's general video understanding ability, outperforming several training-free token compression methods.
Critical strengths
- The idea of introducing compensation within the LLM for information lost during token compression is considered novel.
- The authors’ concern that compression alters the relative representation of different types of content in attention is reasonable.
Critical weaknesses
-
Could saliency-guided pruning compromise semantic completeness? Focusing only on the most salient regions may omit background information or interaction cues. This could affect scenarios in which the main objects interact with surroundings or when considering the relative positions of different objects.
-
The authors partition temporal sequences into segments based on the similarity between adjacent tokens at the same spatial position, using local continuity as a metric. However, the method imposes no further constraints on within-group variance, the maximum pairwise distance, or the similarity between each original token and the final one. It is quite possible for each individual step to involve only a small change while the segment as a whole undergoes a substantial state change.
-
The paper adjusts LLM attention logits using the number of merged tokens and the total attention mass before and after pruning. This appears to rely on the assumption that visual attention can approximate the contributions of discarded tokens across different LLM queries. However, attention reweighting can only influence the retained content; it cannot recover information that is absent. Does this imply that the strategy can only mitigate compression-induced shifts in attention weights, without necessarily matching the actual LLM attention that needs to be corrected?
-
Related to the previous concern, in Eq. (11), appears to be derived solely from the vision encoder, and all retained tokens assigned to the same frame share this factor. This again seems to assume that visual attention alone can model the contributions of discarded tokens across different LLM queries. While in practice, the actual contribution of a visual token also depends on the query, attention head, context, and hidden states.
Questions for authors
- How do the authors handle positional encoding after compression? Could token merging and reordering affect the positional encoding?
- Have the authors considered comparing their method with additional training-free approaches, such as LLaVA-Scissor [1], DyCoke [2], and PruneVID [3]?
[1] LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
[2] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models.
[3] PruneVid: Visual Token Pruning for Efficient Video Large Language Models.