acceptodds
Withdrawn from ICLR 2027

GleanVID: Complementary Token Selection for Efficient Video Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video Large Language Models (VideoLLMs) have achieved strong video understanding capabilities but incur substantial inference overhead due to the large number of visual tokens. Existing VideoLLM token compression methods largely rely on selection-independent scoring, overlooking cross-frame complementarity and consequently retaining redundant evidence across frames. Instead, we view video token selection as a progressive evidence accumulation process. It aims to retain visual evidence that is individually informative and collectively complementary under a limited token budget. Building on this insight, we introduce GleanVID, a training-free inference acceleration framework for VideoLLMs. Specifically, GleanVID first allocates the global token budget across frames according to temporal novelty and then selects tokens by jointly considering local representativeness and subspace complementarity, thereby preserving richer and less redundant visual evidence. Extensive experiments across diverse VideoLLMs and benchmarks demonstrate that GleanVID consistently achieves state-of-the-art performance. Notably, with only 25% of visual tokens, GleanVID preserves 98.6% of Qwen3-VL's original performance while reducing its prefill latency by 44.7%. On LLaVA-OV-7B, GleanVID at a 25% retention ratio even slightly surpasses the original model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient Large Multimodal models, Video Large Language Models, Visual Token Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jWXwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-free visual token compression for efficient VLLMs. It addresses the limitations of existing token compression methods that overlook cross-frame complementarity and consequently retain redundant visual evidence. It proposes GleanVID, which combines Temporal Novelty-Guided Budgeting with Complementary Token Selection based on local representativeness and subspace-level cross-frame complementarity. Experiments on Qwen3-VL-8B-Instruct show that GleanVID preserves 98.6% of the original performance with 25% of visual tokens while reducing prefill latency by 44.7%.

Critical strengths

GleanVID is training-free and does not require access to attention weights, making it compatible with different VideoLLM architectures.

The method achieves competitive performance across three VideoLLMs and four video understanding benchmarks, particularly under tighter token budgets.

The paper provides component ablations and additional analyses to investigate the proposed design and its performance-efficiency tradeoff.

Critical weaknesses

The related work discussion overlooks several relevant VideoLLM token compression methods. STTM[1] exploits spatial and temporal redundancy through token merging, while MeToM[2] and DyToK[3] investigate adaptive frame-level token allocation. AOT[4] further explores local and global context aggregation. This paper should discuss how GleanVID differs from these methods and consider comparisons with representative approaches under matched settings.

The distinction and advantages of subspace-level complementarity over existing set-aware and diversity-driven token selection methods remain insufficiently validated. Although MMG-Vid [5] is cited in the related work, it is not included in the experimental comparisons. Other relevant methods, such as FLoC [6] and CDPruner [7], are also omitted. FLoC selects representative and diverse tokens using a facility location objective, while CDPruner employs DPP-based selection to maximize conditional diversity. However, Table 6(b) only compares SCS with a pairwise-distance criterion. This paper should further clarify the advantages of subspace-level complementarity over these alternative formulations and provide direct comparisons under matched evaluation settings.

The motivation does not sufficiently establish that subspace-level complementarity corresponds to task-relevant evidence. Repeated spatial regions may contain distinct temporal information, while a small projection residual does not necessarily imply that a token is irrelevant to downstream reasoning. Although Appendix C.4 analyzes Effective Rank, Cross-Frame Residual Energy, and Coverage Error, it does not directly establish how these geometric properties relate to task-specific prediction accuracy. This paper should investigate this relationship on tasks involving repeated actions, event counting, and subtle state changes.

The relationship between TNB and CTS requires further investigation. TNB estimates temporal novelty using historical neighborhood prototypes from preceding frames, whereas CTS measures complementarity against previously retained tokens (Eqs. 4 and 9). Consequently, a frame may receive a smaller token budget because similar content appeared earlier, even if that content was not retained. This paper should investigate whether such cases lead to insufficient token allocation and degraded downstream performance.

The temporal novelty estimation in TNB relies on historical neighborhood prototypes at corresponding spatial locations (Eq. 4), which may be sensitive to object and camera motion. Although the proposed 2 × 2 neighborhood pooling can mitigate small spatial shifts, it does not explicitly account for larger displacements across frames. This may cause TNB to allocate fewer tokens to frames containing visual evidence that appeared earlier but was not retained by CTS. This paper should further investigate the robustness of TNB under substantial object motion and camera movement, and examine their effects on frame-budget allocation and downstream performance.

The evaluation of the performance-efficiency tradeoff is incomplete, particularly for longer video inputs. In Table 4, GleanVID improves VideoMME over V-CAST by only 0.1 point while increasing prefill latency from 121.2s to 132.6s, with nearly identical total latency and throughput. Moreover, efficiency is only evaluated with 32 frames at a 25% retention ratio. According to Appendix B.2 and B.3, the default use of all preceding frames introduces a history-dependent computational cost that grows with the number of input frames. Tables 1 and 3 report accuracy with up to 64 input frames, while Table 7 further evaluates 80 and 160 frames under a fixed LLM-side visual-token budget. However, the corresponding runtime efficiency is not reported. Additional efficiency evaluations across different frame counts and retention ratios are needed to establish the scalability and practical advantages of GleanVID.

The qualitative analysis lacks direct visualizations of retained visual tokens. Figure 3 shows frame-level budget allocation, while Figure 5 presents video question answering examples. Neither figure illustrates which spatial tokens are selected across consecutive frames. Visual comparisons with VidCom² and V-CAST would help demonstrate how GleanVID reduces cross-frame redundancy and preserves complementary visual evidence.

[1] Hyun, Jeongseok, et al. "Multi-granular spatio-temporal token merging for training-free acceleration of video llms." ICCV 2025.

[2] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[3] Li, Yulin, et al. "Less is more, but where? dynamic token compression via llm-guided keyframe prior." NeurIPS 2025.

[4] Li, Jinlong, et al. "Token reduction via local and global contexts optimization for efficient video large language models." CVPR 2026.

[5] Ma, Junpeng, et al. "Mmg-vid: Maximizing marginal gains at segment-level and token-level for efficient video llms." AAAI 2026.

[6] Cho, Janghoon, et al. "Floc: Facility location-based efficient visual token compression for long video understanding." ICLR 2026.

[7] Zhang, Qizhe, et al. "Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms." NeurIPS 2025.

Questions for authors

Plz refer to the weaknesses.

Reviewer_CcTUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GleanVID, a training-free token compressor for VideoLLMs. It allocates frame budgets using temporal novelty and selects tokens by combining local representativeness with their residual relative to the subspace of previously retained evidence. Experiments cover three backbones and four video-understanding benchmarks, with additional component, sensitivity, and fixed-output-budget evaluations. The reported results indicate an accuracy–efficiency trade-off, including 98.6% relative performance on Qwen3-VL at 25% retention.

Critical strengths

The paper gives a coherent formulation of cross-frame complementarity and separates frame allocation from token selection. The component replacements and LRS/SCS ablations help identify the contribution of both stages.

Critical weaknesses
  1. The subspace update is not generally correct as written. Appendix B.1, Eq. (13), and Algorithm 2 filter basis columns by the diagonal of an unpivoted QR factorization. For residual columns e1,e1,e2e_1,e_1,e_2, a valid QR has diagonal (1,0,0)(1,0,0), causing e2e_2 to be discarded and later assigned novelty 1 instead of 0. The authors should clarify the actual rank-revealing update and whether correcting the stated rule changes the results.

  2. The methodological novelty appears incremental. Temporal budget allocation, local representativeness, and projection residuals are established ideas. The main distinction is their organization around the span of previously selected tokens. The authors should clarify the contribution beyond existing selection-dependent compression methods and demonstrate its benefit against the closest alternatives.

  3. Long-context and model-scale generalization remain insufficiently evaluated. Main results use 32/64 frames, additional fixed-budget accuracy tests cover 80/160 frames. The authors should extend evaluation to 768 frames with intermediate settings and include Qwen3.5-27B (dense) and Qwen3.5-35B-A3B (MoE), reporting accuracy, compression overhead, end-to-end latency, peak GPU memory, and OOM outcomes under clearly specified budgets. Since full-history processing depends on history length and accumulated subspace rank (Eq. (14)), the current 7–8B dense-model results leave its practical scalability unclear. Furthermore, VideoQA does not establish preservation of event boundaries or descriptive details. Evaluate temporal grounding and video captioning on suitable benchmarks to test temporal precision and event coverage beyond QA.

Questions for authors
  1. Does the implementation use a rank-revealing update absent from Algorithm 2? Please address the counterexample and, if the stated rule was used, report whether its correction changes token selection or the main results.

  2. What advance does frame-sequential subspace selection provide over established residual-based selection? Please isolate the roles of history-update granularity and local representativeness beyond the existing pairwise-distance and LRS/SCS controls.

  3. How do quality and compression-inclusive cost scale to 768 frames with intermediate settings, and to Qwen3.5-27B/35B-A3B? Please report realized budgets, peak memory, and OOM behavior as well as latency. Does compression preserve temporal localization and event-description coverage on suitable non-QA benchmarks? Please provide evidence or state this limit of the current QA evaluation.

Reviewer_yLPYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose to utilize complementarity and temporal novelty metrics to employ visual token reduction for Video LLM. To this end, they first propose a temporal novelty metric to select the visual tokens within each frame, and then accumulating the frames' novelty gradually which helps to find out the redunctant and informative tokens that will be considered to be removed or kept. This paper then uses a dynamic budget allocation method to shift the visual tokens among neighboring frames, which helps to keep more tokens for informative frames while reducing tokens for less informative ones. Moreover, the authors compute a subspace score for estimating the removal or kept token candidates, which is then combined into the cross-frame token operation.

Critical strengths
  1. The visual token reduction strikes competitive performance maintain and efficiency, compared with existing works.

  2. Extensive benchmark evaluations and models are conducted and compared with previous works.

Critical weaknesses
  1. There are lots of proposed components in this paper that confuse me and make it hard to capture the main motivation or demonstration in the main paper. I can give you some examples. In Fig. 1, how can I combine the so-called complementary evidence and subspace? What is the relationship between them? This is the main concern of this paper, how can I capture the main idea of this paper and how to understand various proposed components in the main paper which presents fragmented contributions. What is the main difference from previous methods, like the metrics to select out the visual tokens?

  2. How to demonstrate the Temporal Novelty among neighboring frames, which I suggest to make the visualization demonstration of this part.

  3. Since the uniform sampling strategy will be employed for Video LLM under 32, 64, or more video frames, this exposes sparsity among the sampled neighboring frames, while LRS applies a pooling operation to compute the local representative score that may bring lots of noise. Can the authors explain this in detail?

  4. Regarding Fig. 2, there are too many words or symbols in the figure, complicating understanding. I deduce this was probably created by AI tools but without further adjustment. I suggest simplifying this figure to help others capture and understand the essential idea l directly.

  5. In the introduction part, the academic writing like Fig. 1 show correctly while typos like Table <> and Figure <> happens frequently which should be Table. <> and Figure. <> to be consistent, do the authors revise carefully?

  6. I am concerned about the demonstration in Fig. 3: the video samples usually have complex scene dynamics, while the chosen one showcases a much simpler and object-centric case, which raises concern about the demonstration.

  7. In Tab. 5 and Tab. 6, the MVBenchLVB shows typos or poor presentation.

  8. I suggest visualizing the removed and kept visual tokens across the frames and comparing them with the existing method in the paper.

Questions for authors

There are quite a lot confusions and clarifications in the main paper that needs to explain and demonstrate. I suggest the authors refer to my questions and confusions above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.