acceptodds
Withdrawn from ICLR 2027

Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video large language models (Video-LLMs) represent videos as dense sequences of visual tokens, whose length grows with the temporal and spatial extent of the input. These tokens often contain substantial redundancy arising from repeated visual patterns, leading to unnecessary computation in the subsequent language-model processing. Existing token compression methods, including pruning and merging, perform compression online during inference, repeatedly incurring additional computation for each input video and often relying on model-specific designs that limit their generality, we instead rethink this paradigm by shifting the costly compression process offline. We propose ONCE, a plug-in video token compression framework that introduces an offline-to-online paradigm: a frequency-aware global codebook is learned once in the visual feature space and reused for lightweight online compression through codebook lookup and aggregation, reducing repeated per-video computation and the need for model-specific compression designs. Extensive experiments across multiple video understanding benchmarks and against diverse compression baselines demonstrate that our approach achieves a strong accuracy-efficiency trade-off, maintaining competitive performance while achieving the lowest inference latency among compared methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Visual Token Compression, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zwBeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ONCE, a video token compression framework based on a reusable global codebook. Instead of performing token clustering independently for each video, ONCE learns a codebook offline using temporal stratified sketching and frequency-aware optimization. During inference, visual tokens are assigned to selected codewords and aggregated through mean pooling to reduce the number of tokens processed by the LLM.

Critical strengths
  1. Offline-to-Online Compression Paradigm. Unlike existing methods that perform token grouping separately for each video, ONCE learns a reusable global codebook offline. This design reduces the need for repeated per-video clustering and offers potential practical benefits.

  2. Effective Temporal Stratified Sketching. The proposed temporal stratification helps preserve feature coverage across different video segments during codebook learning. The ablation results demonstrate its importance, as removing temporal partitioning leads to a significant accuracy drop.

Critical weaknesses
  1. Limited Novelty. The proposed method mainly combines existing techniques, including clustering, nearest-neighbor assignment, and mean pooling. The algorithmic novelty beyond these standard operations is relatively limited.

  2. Weak Accuracy-Efficiency Trade-off. At a token budget of 512, ONCE achieves only around 3.1% lower latency than FastVID, while showing worse accuracy (54.50% vs. 56.45% on MVBench). This limits the practical benefits of the proposed method.

  3. Insufficient Analysis of Online Compression Overhead. Although ONCE moves codebook learning offline, inference still requires similarity computation with 8,192 codewords and token reassignment. A detailed breakdown of the online compression latency is missing, making it difficult to assess the actual efficiency gains.

Questions for authors

No

Reviewer_VhLYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

ONCE learns a global codebook for each frozen visual feature space and reuses it for video token compression. At inference, it selects active codewords, reassigns source tokens to the retained support, and outputs means of current-video features. The Video-LLM remains frozen. Experiments cover LLaVA-OneVision and Qwen3.5, several token budgets, inference-policy and fitting controls, feature-coverage diagnostics, and inference efficiency.

Critical strengths

The separation between reusable codebook geometry and input-specific pooled content is useful. Output-representation controls show that replacing source features with codewords changes performance, helping clarify what the learned artifact contributes.

Critical weaknesses
  1. The selected LLaVA codebook does not use the proposed Sinkhorn updates. Table 7 selects the refined initializer at epoch 0 for LLaVA, while Qwen selects epoch 5. Table 9's separate sweep also favors uniform weights or Hard-Lloyd over weighted, long-tail-corrected Sinkhorn. The contribution of the elaborate fitting stage is therefore unclear. Compare initializer, Hard-Lloyd, and Sinkhorn codebooks using identical data and checkpoint-selection rules on each backbone.

  2. Whole-video pooling can erase event order unless time information is preserved separately. Eqs. (8)–(9) group tokens by similarity, average them, and order the outputs by codeword frequency. With frame features that contain no temporal index, the same frames in reverse order produce the same compressed representation. Specify how the LLaVA path retains timestamps, and test reversed frame sequences to establish whether event order survives compression.

  3. The baseline comparison does not establish competitiveness across backbones. LLaVA-OneVision-7B is compared only with ForestPrune and FastVID; Qwen3.5-9B has no competing compressor results. Relevant additions include VisionZip [1], VidCom2 [2], MMG-Vid [3], HoliTom [4], and V-CAST [5]. Reproduce compatible baselines on Qwen3.5, documenting adaptation changes and pruning stages. Compare accuracy, end-to-end latency, and peak memory under matched inputs and comparable computation budgets. Qwen3.5-27B and Qwen3.5-35B-A3B, each with its own fitted codebook, would further test larger dense and MoE models. Extend fitting-cost amortization to compatible compression baselines.

  4. QA-based Action Localization does not establish timestamp accuracy. Evaluate temporal grounding and video captioning on suitable benchmarks to test whether global grouping preserves event boundaries and temporal relations.

[1] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. VisionZip: Longer Is Better but Not Necessary in Vision Language Models. CVPR, 2025.

[2] Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models, EMNLP, 2025.

[3] Junpeng Ma, Qizhe Zhang, Ming Lu, Zhibin Wang, Qiang Zhou, Jun Song, and Shanghang Zhang. MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs, AAAI, 2026.

[4] Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. HoliTom: Holistic Token Merging for Fast Video Large Language Models, NeurIPS, 2025.

[5] Xinying Lin, Xuyang Liu, Yiyu Wang, Teng Ma, and Wenqi Ren. V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models, NeurIPS, 2026.

Questions for authors
  1. Under the same checkpoint-selection criterion and inference settings, which parts of balanced fitting improve over the refined initializer and a matched Hard-Lloyd codebook? Please distinguish initialization, later optimization, and checkpoint selection.

  2. How is temporal position represented after frequency-ordered grouping on the LLaVA path? Does reversing the same frame features change the compressed representation or predictions on order-sensitive questions?

  3. The submitted code provides only interface stubs, with both fitting and inference unimplemented, and therefore cannot reproduce the reported results.

Reviewer_gpNpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ONCE learns a global codebook for each frozen backbone and uses it to aggregate video tokens before language-model inference. The method shows benefits under severe compression

Critical strengths
  • The method is effective at very low token budgets. ONCE outperforms both evaluated compression baselines across all four LLaVA-OneVision benchmarks at budgets of 32, 64, and 128. This provides consistent evidence of its usefulness when visual-token capacity is severely constrained.

  • The codebook can be reused without modifying the Video-LLM. Once fitted for a backbone, it supports multiple datasets and compression budgets without additional backbone training. Aggregating current-video features also avoids directly replacing visual evidence with fixed prototypes.

Critical weaknesses
  • The full computational cost is insufficiently quantified. The paper should report offline feature-extraction and codebook-fitting costs, alongside stage-wise online latency under matched optimized attention backends, particularly given the modest 3.1% latency reduction over FastVID at budget 512 and lower MVBench accuracy.

  • Codebook sensitivity and the necessity of the fitting procedure remain unclear. Experiments varying codebook size and fitting data, together with a simple spherical k-means baseline, are needed to assess robustness and justify the added complexity.

  • Frequency-based selection and mean pooling may erase task-critical details. Complete reassignment does not guarantee information preservation, so targeted evaluations of brief events, small objects, and temporal order are needed.

  • The Qwen3.5 evaluation lacks competing compression baselines. Comparisons with FastVID, ForestPrune, and a representative merging method under matched settings, reporting both accuracy and latency, are needed to substantiate cross-backbone effectiveness.

Questions for authors

See Weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.