acceptodds
Withdrawn from ICLR 2027

Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video Large Language Models (Video LLMs) have achieved remarkable progress in video understanding, but their inference efficiency is constrained by the large number of visual tokens produced by long videos. Recent video token compression methods increasingly introduce sophisticated strategies for token selection, pruning, and merging. This raises a fundamental question: how much of compression performance can be obtained by simply preserving the structure encoded in the visual representations? We investigate this question with SimpleCluster, a simple and training-free baseline that performs position-aware cross-frame clustering in the visual feature space and represents each cluster using the mean of its original visual features. Extensive experiments across four video understanding benchmarks and three representative Video LLMs show that SimpleCluster achieves competitive or superior performance over recent compression methods across a wide range of token retention ratios, with particularly strong robustness under extremely low retention rates (e.g., 1%). To understand this behavior, we analyze the feature space preserved by different compression methods in terms of local approximation fidelity and global coverage. The results show that stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially its global coverage. These findings highlight feature-space preservation as an important consideration for video token compression under highly constrained token budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Compression, Video Large Language Models, Efficient Large Multimodal Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VwXHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits visual token compression for Video LLMs and proposes SimpleCluster, a simple training-free approach based on position-aware cross-frame clustering. The method groups visual tokens using spatiotemporal information and represents each cluster by averaging its original visual features. Experiments on four video understanding benchmarks and three Video LLMs demonstrate competitive performance, particularly under aggressive token compression. The paper further analyzes feature-space preservation through local approximation fidelity and global coverage to explain the effectiveness of the proposed approach.

Critical strengths
  1. I think this paper is generally well written.
  2. The organization is logical.
Critical weaknesses
  1. One of my main concerns is that the proposed semantic feature aggregation strategy seems conceptually similar to AOT, as both methods aim to reduce visual redundancy by aggregating semantically related tokens. Although their implementations differ, the authors should better clarify the key differences and the technical novelty of SimpleCluster compared with AOT.
  2. While the paper reports improvements in inference latency, cross-frame clustering itself may introduce considerable computational cost as the number of visual tokens increases. The authors should provide a clearer breakdown of clustering time, memory consumption, and end-to-end latency, particularly for longer videos and different token budgets.
Questions for authors

Please see the Critical Weaknesses section.

Reviewer_4XNGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates visual token compression for Video LLMs by proposing SimpleCluster, a training-free baseline that applies cross-frame spherical K-Means clustering on post-projector visual tokens (augmented with 3D RoPE for cluster assignment) and represents each cluster by its mean visual feature. Experiments across three Video LLMs and four benchmarks show strong accuracy retention, especially at 1%–5% token budgets, supported by an analysis of feature-space quantization error and cosine coverage.

Critical strengths
  1. Valuable simple baseline perspective: Questioning whether complex token pruning/merging pipelines truly outperform classical spherical K-Means under tight budgets is a useful contribution to Video LLM efficiency research.
  2. Strong performance under extreme compression: SimpleCluster consistently preserves accuracy across three backbones and four benchmarks at aggressive 1%–5% retention ratios.
Critical weaknesses
  1. Omission of essential training-free video token compression baselines (SF-LLaVA, TS-LLaVA, and FLoC): Although positioned as a "simple yet strong training-free baseline study" (Sections 1 & 7), the paper omits established training-free baselines with the exact same positioning, specifically SF-LLaVA (SlowFast-LLaVA, Xu et al., 2024) and TS-LLaVA (Qu et al., 2024), as well as feature-clustering methods cited in the text (FLoC, Cho et al., 2026; HoliTom, Shao et al., 2025; Chat-UniVi, Jin et al., 2024). In Section 4.3 (Table 4), SimpleCluster is only compared against naive single-scale pooling and top-left grid sampling (Line 268), whereas SF-LLaVA and TS-LLaVA explicitly combine local spatial fidelity (high-resolution Slow/Thumbnail tokens) and global temporal coverage (dense Fast/Sampled tokens)—the exact two principles emphasized in Section 5—at zero clustering overhead (unlike SimpleCluster, whose iterative K-Means overhead limits actual TTFT speedup at 10% retention to just 20.1% on InternVL3-2B and 37.1% on LLaVA-Video-7B in Tables 10–11).
  2. Modest end-to-end speedups on LLaVA-Video-7B and InternVL3-2B due to clustering overhead (Appendix B.2): While compressing visual tokens to 10% cuts theoretical prefilling FLOPs by 66.5%–73.2%, the actual end-to-end Time-To-First-Token (TTFT) reduction is only 37.1% on LLaVA-Video-7B (1722.3 ms to 1082.6 ms, Table 10) and only 20.1% on InternVL3-2B (1079.2 ms to 862.6 ms, Table 11). This large gap between FLOPs reduction and actual wall-clock speedup indicates that combining vision encoding with O(T × N × B × d) iterative cross-frame clustering introduces substantial runtime overhead. Moreover, the paper only reports TTFT at 10% retention (B = 627 or 1,082), omitting latency evaluations at 15% and 25% retention where B is 1.5×–2.5× larger (which further increases K-Means clustering cost while reducing LLM prefill savings), as well as omitting the number of K-Means iterations and prototype initialization details.

References

  • Mingze Xu et al., SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models (SF-LLaVA), arXiv:2407.15841, 2024.
  • Tingyu Qu et al., TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models, arXiv:2411.11373, 2024.
  • Janghoon Cho et al., FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding, ICLR 2026.
  • Kele Shao et al., HoliTom: Holistic Token Merging for Fast Video Large Language Models, NeurIPS 2025.
  • Peng Jin et al., Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding, CVPR 2024.
Questions for authors
  1. How does SimpleCluster compare in both accuracy and end-to-end latency against SF-LLaVA, TS-LLaVA, and FLoC across the evaluated retention ratios?
  2. What is the exact wall-clock runtime breakdown (in ms) of the SimpleCluster clustering step itself vs. Vision Encoder and LLM Prefill across retention ratios of 1%, 5%, 10%, 15%, and 25% on all three backbones (and how many K-Means iterations are executed)?
Reviewer_uqn8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimpleCluster compresses video tokens by clustering projected visual features across frames after applying a 3D RoPE transform, then averaging the original features within each cluster. The paper evaluates it on three models and four benchmarks, with the clearest gains at very low token budgets.

Critical strengths
  1. SimpleCluster is training-free and performs well at aggressive compression. At 1% retention, it retains 84.8%–88.3% of the uncompressed average score across the three tested models.
  2. The evaluation covers multiple models, benchmarks, and token budgets, making the approach relevant to practical video-token compression.
Critical weaknesses
  1. The role of SimpleCluster’s added 3D RoPE is unclear for models that already use RoPE for multimodal positions, such as Qwen3-VL with interleaved M-RoPE. The paper does not test Qwen3-VL or explain how positions are assigned to clusters spanning multiple frames.
  2. The experiments do not fully isolate the source of the gains. Ablations are reported only at 10%; there is no controlled comparison with DPC-KNN-based methods such as FastVID and HoliTom. The link between feature metrics and downstream performance is also not quantified, and the efficiency comparison is limited.
Questions for authors
  1. Qwen3-VL already uses interleaved M-RoPE for spatial-temporal modeling. What additional signal does SimpleCluster’s 3D RoPE provide, and how are native position indices assigned to clusters spanning multiple frames? Please test Qwen3-VL with and without the extra RoPE at matched budgets. (Qwen3-VL technical report)
  2. Can you report key ablations at 1%, 5%, and 25% retention, and compare against DPC-KNN-based methods under the same model, budget, and aggregation setup?
  3. Can you pair downstream accuracy with L2-NQE and Cosine Coverage at matched budgets, ideally across backbones and benchmarks, and report whether the metrics consistently track performance?
  4. Please compare compression time, end-to-end latency, throughput, and peak memory against the main baselines using a consistent setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.