acceptodds
Withdrawn from ICLR 2027

Do We Still Need Token Compression in Video Large Language Models?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Visual-token pruning is widely used to improve the efficiency of video large language models (VideoLLMs). However, backbones with native dynamic-resolution support introduce an alternative: controlling the visual-token budget through input resolution before vision encoding. We revisit this design choice through a controlled study of Qwen3-VL-4B, GLM-4.6V-Flash, VideoLLaMA3, and Oryx, covering 64- and 128-frame inputs and four video question-answering benchmarks. Under matched actual token budgets and controlled video interfaces, established pruning methods do not consistently outperform random dropping, and method rankings vary substantially across backbones. Simple resolution reduction can also match or exceed the accuracy of post-encoding pruning. To understand these results, we compare random dropping with VCAST and examine the roles of spatiotemporal coverage, evidence aggregation, and backbone characteristics. Our analysis suggests that selecting informative tokens alone is insufficient: compression must also preserve the distribution and structure of visual evidence. Random dropping can retain broad spatiotemporal coverage, while downsampling aggregates local evidence before the vision encoder constructs a dense representation. End-to-end profiling further shows that resolution reduction saves vision-encoding computation that post-encoding pruning has already incurred. Building on these findings, we investigate dynamic resolution beyond uniform downsampling, separating budget fidelity from content-aware allocation through a budget-controlled oracle-to-heuristic study. Our results motivate dynamic resolution as a promising paradigm for efficient VideoLLMs: it directly reduces vision-encoding cost and offers a route to allocating visual computation adaptively across frames and task demands. More broadly, effective visual compression requires joint consideration of the backbone, evidence preservation, and where computation is reduced.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient VLM, Video Understanding, Video LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_w4CrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper revisits the necessity of visual token compression in modern VideoLLMs with native-resolution processing and built-in token reduction. It compares V-CAST against random token dropping, input downsampling, and a simple dynamic-resolution baseline across four VideoLLMs and four video benchmarks under 12.5% and 5% token-retention budgets. The paper finds that V-CAST does not consistently outperform simpler alternatives and argues that adaptive input-resolution allocation may provide a better accuracy--efficiency trade-off. It further analyzes backbone-dependent token geometry and selection behavior to explain why post-encoding compression can be less effective in these architectures.

Critical strengths
  1. The paper studies an important and timely question regarding the efficiency of modern VideoLLMs.

  2. The paper is well written, clearly organized, and easy to follow, which makes the empirical findings accessible.

Critical weaknesses

1. The paper substantially overgeneralizes from a narrowly defined experimental regime.

The main experiments consider only 12.5% and 5% retention, with the retention ratio defined after each model's native visual processing. These correspond to extremely aggressive additional-compression regimes, while moderate settings such as 80%, 60%, 40%, and 20% are not evaluated at all.

The architecture coverage is also too narrow to justify a general conclusion about "modern VideoLLMs." All four evaluated backbones are native-resolution architectures with built-in token reduction mechanisms, leaving other architectural regimes untested. Therefore, the evidence supports a much narrower conclusion: additional post-encoding compression may exhibit diminishing returns on already-compressive native-resolution architectures under extremely low token budgets. It does not support the broader implication that visual token compression itself is no longer necessary.

2. The empirical evidence is insufficient to support a general conclusion about sophisticated token compression methods.

The main cross-architecture evaluation effectively relies on a single compression method, V-CAST. KiToke, FlashVID, MMTok, EchoPrune, and TTF are evaluated only in a preliminary experiment under a single model/benchmark/ratio setting (Qwen3-VL-4B, VideoMME, 12.5% retention), after which V-CAST is selected as a "strong representative" and used as the sole sophisticated compression method in the main four-model evaluation. At the same time, several relevant methods discussed in the paper's own related work, including DyCoke, PruneVid, FastVID, and HoliTom, are not included in the main evaluation, while other recent methods such as FastV, VisionZip, UniComp, and SURGE are not considered.

Consequently, the experiments establish at most that V-CAST is often uncompetitive with simple baselines under the evaluated settings. They do not provide sufficient evidence to conclude that sophisticated visual token compression methods are broadly ineffective or unnecessary.

Questions for authors
  1. Can the authors provide results over a broader range of retention ratios, including moderate settings such as 80%, 60%, 40%, and 20%?

  2. Why is V-CAST used as the sole representative of sophisticated token compression in the main cross-architecture evaluation?

  3. Does the main conclusion remain valid when evaluated with a broader set of recent compression methods beyond V-CAST?

Reviewer_G7UrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper revisits the effectiveness of visual token compression in modern Video LLMs, mainly with native-resolution and built-in visual aggregation. The author compare V-CAST against random token dropping, uniform spatiotemporal downsampling, and dynamic-resolution across four architectures and four benchmarks. The conclusion is that sophisticated compression does not consistently outperform these simpler alternatives under the evaluated token budgets. Based on these findings, the paper advocates adaptive input-resolution allocation as a promising direction.

Critical strengths
  1. The paper challenges whether the benefits of specialized token compression transfer reliably to newer architectures, which is admirable.

  2. The experiments cover four architectures, four complementary benchmarks, two frame configurations, and two token budget ratios. The results reveal differences across backbones.

Critical weaknesses
  1. My primary concern is the limited novelty of this paper. The paper mainly revisits comparisons between existing token pruning methods and simple token reduction methods on modern VideoLLMs, without introducing a fundamentally new method, or providing sufficiently deep analysis. In my view, the resulting insights do not constitute a strong enough contribution to meet the standard for ICLR. Furthermore, to my knowledge, the observation that simple pruning strategies can outperform more sophisticated methods has already been discussed in several prior works.

  2. Another major concern is whether the evidence sufficiently supports the broader premise suggested by the title, “Do We Still Need Token Compression in Video Large Language Models?”, which is a complex problem. Some previous pruning methods were not designed for the architectures of modern Video LLMs. Their limited effectiveness on these models does not, by itself, imply that token compression is no longer needed. Moreover, the main tables evaluate only one compression method, V-CAST, making it difficult to generalize the findings to token compression as a whole.

Questions for authors
  1. How broadly do the conclusions generalize beyond V-CAST?

  2. What drives the gains of dynamic resolution? Could the author establish whether the gains arise from adaptive resolution allocation?

Reviewer_abWNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the necessity of recent token compression strategies of VideoLLMs against random drop and uniform spatio-temporal downsampling. The main finding from this evaluation is that recent token compression strategies do not necessarily outperform random/uniform downsampling. The authors further suggest adjusting input resolution before visual encoding by validating a simple method using CLIP latent-based relevance weighting (DynRes).

Critical strengths
  • The paper investigates an important topic in the field: practicality of visual token compression methods in VideoLLMs.
Critical weaknesses
  • Baseline token compression method includes only V-CAST, others only compared with the VideoMME benchmark. Experiments need to be more extensive for an evaluation and benchmark focused work.
  • Fine-grained motion understanding is where random or uniform downsampling might underperform. Including video benchmarks for fine-grained motion understanding can be helpful.
  • In summary, the question of practicality of visual token compression methods is important, but not extensively answered in this evaluation work which limits its contribution.
Questions for authors
  • Please consider performing a more extensive evaluation sweep across 1) token compression methods (methods included in Figure 2), 2) VideoLLM model parameter sizes, and 3) benchmarks (MotionBench at least, for fine-grained motion understanding).
  • Is DynRes short for dynamic resolution? Is it the name of a specific method (perhaps the Dynamic Resolution in Sec. 3.2.)? Please define and explain the abbreviation at its first appearance.
  • Some minor suggestions for improving readability and clarity: 1) provide x,y axis labels and T/H/W abbreviation definition for Figure 2. 2) Please refer to main tables or figures for each interpretation of Sec. 3.4. Main Results. 3) Pale grey lines in Figure 3 are distracting. Please either make it bolder or remove it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.