acceptodds
Under review as a conference paper at ICLR 2027

STORS: Spatio-Temporal Orthogonal Residual Selection for Video Token Compression

Abstract

Efficient video understanding requires compressing highly redundant visual tokens across space and time. Existing geometric selection methods typically operate on decoder-facing representations, overlooking how the visual projector may alter the geometry on which their selection criteria depend. We identify a substantial change in representation geometry across the projector in Qwen2.5-VL: encoder-side features exhibit extreme anisotropy and near-saturated pairwise similarities, whereas decoder-facing features are substantially more dispersed. Motivated by this finding, we propose STORS, a training-free method that scores tokens by their unexplained residual energy over a unified spatio-temporal pool while retaining their projected features for the language model. An explicit temporal redundancy weight controls cross-frame redundancy under a fixed token budget. Controlled experiments reveal that the effectiveness of residual selection depends strongly on the scoring space: on Video-MME, the same criterion underperforms uniform grid sampling in all nine settings when applied after the projector, but outperforms it in eight of nine when applied before the projector. In contrast, no statistically detectable scoring-space benefit is observed in LLaVA-OneVision, whose representations exhibit similar geometry. Across five video benchmarks and two backbones, STORS achieves competitive performance under aggressive compression. On Qwen2.5-VL-7B, retaining 10% of visual tokens preserves 94.4% of the uncompressed average accuracy while achieving a speedup in time to first token.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Token Compression, Vision-Language Models, Scoring-Space Decoupling, Spatio-Temporal Redundancy, Orthogonal Residual Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TYbuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

STORS proposes a training-free method for video visual-token compression. It selects tokens by orthogonal residuals over a unified spatio-temporal candidate pool using vision-encoder-side features, then merges the post-projector features of unselected tokens into the kept tokens. The paper's central observation is that representation geometry differs substantially across the visual projector in Qwen2.5-VL, making pre-projector scoring more effective than post-projector scoring; no comparable scoring-space benefit is statistically detectable in LLaVA-OneVision.

Critical strengths
  1. The scoring-space comparison is valuable. On Video-MME with Qwen2.5-VL, holding the selection criterion fixed and changing only the scoring space, pre-projector scoring yields higher accuracy in all nine settings; eight meet the paper's paired-test significance threshold. The authors also report the absence of a detectable benefit on LLaVA-OneVision.
  2. The compression results have practical potential. At 10% visual-token retention, Qwen2.5-VL achieves an average score of 58.7, or 94.4% of the uncompressed score of 62.2. With 32 frames, time to first token (TTFT) decreases from 1.10 s to 0.60 s.
Critical weaknesses
  1. The scope and cause of the scoring-space benefit remain unclear. For Qwen, the default pre-projector scoring feature is the mean of the four sub-tokens within a 2×22\times2 cell, whereas the actual projector input is their concatenation. The current results do not adequately separate the effects of feature averaging, the projector's joint spatial transformation, and other architectural differences between backbones. I suggest comparing mean, concatenated, and post-projector scoring features on the same Qwen backbone under matched settings, and qualifying the cross-backbone mechanistic interpretation accordingly.
  2. The main LLaVA-OneVision results are less complete, limiting cross-backbone conclusions. Table 1 reports three retention ratios (5%/10%/20%), Video-MME duration splits, MLVU, and a broader set of compression baselines for Qwen. Table 2 reports only 10%/20%, average scores on three benchmarks, and a subset of the baselines for LLaVA. Although supplementary Table 21 includes STORS and scoring-space comparisons at 5% on LLaVA, it does not provide a full matched-budget, multi-method comparison analogous to Table 1. Please add the 5% baselines, Video-MME duration splits, MLVU, and missing methods where feasible, and explain any inapplicable settings.
  3. Efficiency gains should be distinguished from those over a matched-budget baseline. At 32 frames and 10% retention, TTFT is 0.60 s for STORS, 0.64 s for VisionZip, and 1.10 s without compression. The approximately 1.82× speedup over the uncompressed model is valid, but the latency improvement over VisionZip at a matched token budget is about 6%. Please break down the time spent on visual encoding, token selection and merging, and LLM prefill. The current budget-matched comparison disables FlashVID's second, in-LLM compression stage; I also recommend reporting the quality–latency results and per-layer token counts for the original two-stage configuration.
Questions for authors
  1. On the same Qwen backbone, data, and token budget, can you compare accuracy, feature geometry, coverage-fill fraction, and computational cost when the scoring features are the four-sub-token mean, concatenation, or post-projector features? Can these experiments separate the effects of scoring-feature construction from those of the projector itself?
  2. Can you provide LLaVA-OneVision comparisons aligned with Table 1 as closely as possible? If some baselines or datasets cannot be included, please explain the limitations.
  3. What is the relationship between the optimization objectives in Eqs. (2) and (1)? Please also check the explanation of the stopping criterion following Eq. (4).
  4. Please report a stage-wise breakdown of TTFT, end-to-end inference latency, peak memory, and the quality–latency results and per-layer token-retention profile of the original two-stage FlashVID.
Reviewer_DrWeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STORS is a training-free video token compression method that performs global spatiotemporal orthogonal residual selection on pre-projector features and uses the corresponding post-projector features, optionally augmented through merging, for decoding.

Critical strengths
  • Separating the scoring space from the decoding representation is an interesting design choice. The paper demonstrates that the same residual-based selection criterion can behave substantially differently before and after the projector. This observation provides a useful perspective on geometry-based token compression.

  • The method is supported by extensive ablations and shows practical potential. The experiments examine scoring spaces, temporal weighting, feature merging, and coverage sampling across multiple benchmarks. The reported accuracy and TTFT results suggest a useful efficiency–performance trade-off at low token retention ratios.

Critical weaknesses
  • The backbone coverage is dated, leaving generalization to newer architectures unclear. The evaluation is limited to Qwen2.5-VL and LLaVA-OneVision, without experiments on newer models such as Qwen3-VL-8B. Since the central argument depends on projector-induced changes in representation geometry, validation on a newer architecture is particularly important.

  • The baseline comparison does not sufficiently cover recent progress. Despite including FlashVID, the evaluation omits relevant methods such as EarlyTom and Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention from CVPR 2026, as well as S3-Prune and SToP from ECCV 2026. Comparisons under matched backbones, frame counts, and token budgets, together with end-to-end latency measurements, would better establish the method's competitiveness.

  • The runtime accounting for online orthogonalization needs clarification. Algorithm 1 requires iterative QR decomposition and residual updates during selection, followed by another QR decomposition in the post-projector space, but Table 5 does not explicitly define whether its TTFT measurements include all these operations. The authors should clarify the timing boundaries and provide a runtime breakdown for the reported 32- and 64-frame settings to quantify the compression overhead.

  • A minor reporting inconsistency between the main text and appendix should be clarified. For LLaVA-OneVision on Video-MME Medium at 5% retention, Table 4 reports a score of 52.00 and discordant counts of 41/46, whereas Table 21 reports 52.06 and 41/45. Table 21 explicitly explains the exclusion of video-decoding failures, so the authors should clarify whether Table 4 uses the same valid sample set; this discrepancy does not change the nonsignificant conclusion in this setting.

Questions for authors

See weakness above

Reviewer_mLHJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes STORS, a training-free method for video token compression. The central empirical claim is that the visual projector in Qwen2.5-VL substantially changes token geometry: pre-projector encoder features are highly anisotropic with near-saturated pairwise cosine similarities, whereas post-projector decoder-facing features are more dispersed. Motivated by this observation, STORS separates a score space (pre-projector features, used for selection) from a reserve space (post-projector features, passed to the LLM). Selection is performed by greedy orthogonal residual (column-pivoted QR) over a unified spatio-temporal token pool, with a temporal redundancy weight controlling the contribution of cross-slice explained energy. Experiments on Qwen2.5-VL-7B and LLaVA-OneVision-7B across five video benchmarks report competitive accuracy under aggressive compression (10% retention preserves 94.4% of uncompressed accuracy on Qwen2.5-VL-7B, with 1.82× TTFT speedup).

Critical strengths

S1. The geometric diagnostic is real, reproducible, and cross-validated. Table 12 reports anisotropy, cosine medians, and >0.9 fractions for both spaces, and the measurements are reproduced across two independent benchmarks with an independent feature-extraction implementation, agreeing to within 0.01. This is a genuine empirical contribution: the near-rank-one character of the score space (99.98% of pairs with cosine >0.9) versus the reserve space (0.01%) is a large, stable effect that the community should know about.

S2. Table 14 is a well-designed controlled experiment. Holding scores, candidates, budget, and pruning step fixed while varying only the scoring space, the same residual criterion is worse than a uniform grid in the reserve space (nine cells negative, up to −6.11) and better in the score space (nine cells positive). The sign flip is the paper's most compelling evidence that criterion validity is a property of the representation, not of the criterion itself.

S3. The falsifiable prediction in Appendix N is unusually rigorous. The mechanism predicts that on a backbone whose projector acts per-token (LLaVA-OneVision), the scoring-space effect should vanish entirely. Table 21 confirms this.

Critical weaknesses

W1. Neither of the two self-declared contributions holds. §3 claims to fill two gaps: "video treated as a longer image" and "in which representation residuals should be measured." The first is handled by λT\lambda_T, but §3.2 itself states that at λT=1\lambda_T = 1, si≡ris_i \equiv r_i and the criterion degenerates to "plain column subset selection over one global pool." Yet §4.1/§4.3 set λT=1\lambda_T = 1 as the default, and all main results use it — so the patch is switched off in the default configuration. The second gap also does not exist: the Introduction states that geometric criteria "typically operate on decoder-facing features," but ToMe [1] applies a geometric criterion inside the ViT (as the authors themselves write in §2), and VisionZip [2] / PruMerge [3] also score on the encoder side.

With both removed, the method = SPARE's criterion [4] (§2's description of it matches Eqs. (1)–(2) exactly; arXiv-only, so discussion is limited) run on Du et al.'s global pool [6], with a greedy sweep equivalent to CDPruner [5]. **CDPruner (NeurIPS 2025) and Du et al. (CVPR 2026) are both formally published, and the paper never compares against them under the same backbone and same budget.

W2. Appendix H cites a non-existent ablation. Appendix H invokes "the mechanism ablation in the main paper" to argue that the geometric discontinuity comes from the MLP rather than from 2×2 averaging, but the paper contains no accuracy result for score (concat) anywhere, and Table 6 has no such row. Table 12 in fact shows concat statistics falling between the two spaces (cosine median 0.685 vs 0.265/0.997), pointing toward averaging also contributing.

W3. Mismatch between framing and evidence. The abstract and introduction frame STORS as a methodological contribution ("we propose STORS, a training-free method"). However, the supporting evidence points instead to an analytical contribution: geometric observations (Table 12—reproducible and stable across datasets), controlled experiments (Table 14—accounting for the confounding factor of the greedy ratio), and falsifiable predictions (Appendix N—which were empirically verified). The method itself (see W1) is a reconfiguration of previously published components, and its "headline" configuration actually disables the very novelty it claims to possess. The framing should be adjusted: the geometric analysis is the primary contribution, while the method serves as an instantiation of that analysis.

[1] Bolya et al. Token Merging: Your ViT but Faster. ICLR 2023. [2] Yang et al. VisionZip: Longer is Better but Not Necessary in VLMs. CVPR 2025. [3] Shang et al. LLaVA-PruMerge. ICCV 2025. [4] Lee et al. Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction. arXiv:2606.18681, 2026. [5] Zhang et al. Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs. NeurIPS 2025. [6] Du et al. Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention. CVPR 2026.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.