- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces an efficient, training-free framework tailored for long video understanding by integrating key token selection with an adaptive early exit mechanism. The proposed approach is empirically validated on relevant long video understanding benchmarks.
Critical strengths
- Competitive Performance: The paper achieves competitive performance on the evaluated long video understanding benchmarks.
- Practical Design: The integration of token compression and early exit mechanisms presents a practical and effective approach, particularly under the circumstance where computational overhead must be strictly managed.
- Empirical Insights: The finding that utilizing information entropy as the metric for key token selection provides a heuristic and valuable insights for future research in token pruning.
Critical weaknesses
- Incremental Novelty: The contribution appears largely incremental, primarily acting as an integration of multiple existing optimization techniques. Since each individual component has already been extensively explored and discussed in prior literature, the fundamental architectural novelty of the proposed framework is somewhat limited.
- Limited Architectural Scope: The framework is currently evaluated predominantly on MLLMs employing the conventional "Vision Encoder + LLM" paradigm. Its generalizability warrants further validation on more unified, end-to-end multimodal architectures, such as those based on the Mixture-of-Transformers (MoT) paradigm.
Questions for authors
- Broader Multimodal Applications: The current experimental evaluations are largely confined to standard vision-language understanding tasks (e.g., Visual Question Answering). To demonstrate the robustness and versatility of the proposed training-free framework, it would strengthen the paper to explore its efficacy across a broader spectrum of complex modality combinations, such as Vision-Language-Action or Vision-Language-Audio tasks.
4 comments
[2/3] Applicability to Mixture-of-Transformers architectures (W2)
Our experiments indeed focus on VLMs following the “vision encoder + LLM” paradigm. We address architectural generalization through both transferability within this paradigm and extensibility beyond it.
-
Transferability within the “vision encoder + LLM” paradigm. We evaluate AdaptToken on InternVL2.5-8B, Qwen2.5-VL-7B/72B, and Qwen3-VL-8B, covering three model families and four backbone configurations. The gains across four long-video benchmarks demonstrate effective transfer across these backbones and scales within this paradigm, without additional training or changes to their architectures or weights. This paradigm is still adopted by many recent VLMs, including Qwen3-VL and InternVL3.5.
-
Extensibility beyond this paradigm to MoT. However, the core AdaptToken framework is not inherently limited to the “vision encoder + LLM” paradigm and has a natural extension path to compatible MoT backbones. For MoT models supporting video-conditioned autoregressive text generation, response entropy can still guide cross-group allocation. MoT’s modality-specific Q/K/V projections are followed by global self-attention, allowing text-to-visual attention to rank visual tokens wherever the attention mask permits, as reflected in the MoT implementation. The selected visual input representations can then be reassembled in temporal order for joint inference, preserving positional and modality information and adapting masks and input-layout metadata. These correspondences support structural feasibility; the resulting accuracy and efficiency require validation.
Given the limited rebuttal time, we have not completed the implementation and systematic evaluation on MoT. We will discuss this extension and its accuracy–efficiency validation as future work in the revision.
[3/3] Broader multimodal applications (Q1)
Following the reviewer’s suggestion, we have extended AdaptToken to the vision–language–audio setting and conducted an additional experiment.
-
Benchmark and adaptation. We evaluate Qwen2.5-Omni-7B on LVOmniBench, which contains 1,014 questions on 10–90-minute videos, averaging 34 minutes. We apply the same reference-layer cross-modal attention scoring criterion to both visual and audio tokens, scoring 4,096 frames and the full audio track group-wise. Final inference retains approximately 7.1K visual and 7.5K audio tokens, totaling about 14.7K multimodal tokens, without additional training.
-
Results. AdaptToken achieves 36.8% overall accuracy, exceeding the 32.0% reported for Qwen2.5-Omni-7B in LVOmniBench by 4.8 percentage points, with higher accuracy across all four categories. Accuracy scores (%) are shown below.
| Model | Understanding | Perception | Inference | Logical | Overall |
|---|---|---|---|---|---|
| Qwen2.5-Omni-7B, reported in LVOmniBench | 29.1 | 34.9 | 31.8 | 28.2 | 32.0 |
| Qwen2.5-Omni-7B + AdaptToken, training-free | 38.7 | 38.6 | 34.9 | 31.2 | 36.8 |
On the subset of videos longer than 60 minutes, AdaptToken achieves 44.1% accuracy over 34 questions.
- Implications and scope. These results provide direct empirical evidence that AdaptToken extends to joint visual and audio inputs using the same training-free token-selection framework. We will include this experiment in the revision. Given the limited rebuttal time, we have not conducted experiments on vision–language–action tasks and leave this extension for future work.
I acknowledge the authors' response to the reviews. After thoroughly evaluating the rebuttal and the perspectives shared by other reviewers, I will maintain my original recommendation.
[1/3] Methodological contribution (W1)
Our core contribution is a training-free method for globally selecting visual evidence under MLLM memory and context constraints, with three methodological advances.
Global token selection with attention and response entropy. Global selection is challenging because a long video's candidate tokens can exceed both GPU memory and the backbone's supported context length. To our knowledge, AdaptToken is the first framework to jointly use model attention and response entropy for global visual-token selection across separately processed video groups. Attention ranks tokens within groups, while entropy guides budget allocation across groups, enabling global selection without placing the entire candidate video in one context. On Qwen2.5-VL-7B, AdaptToken outperforms FlexSelect by 2.3–4.3 percentage points across four benchmarks (Table 1).
From LLM answer voting to MLLM evidence selection and aggregation. Related entropy and self-certainty methods have mainly targeted text-only LLM reasoning, commonly using voting to combine answers (Kang et al., 2025; Fu et al., 2025b). We experimentally establish the association between response entropy and video evidence coverage (Fig. 3, left), and use this finding to guide MLLM token selection. We then aggregate selected visual tokens for joint final reasoning, allowing complementary evidence across groups to inform one answer. This produces more reliable answers in our evaluation: Table 5 shows gains of 3.0 / 2.5 / 2.1 percentage points over the strongest voting baseline on VideoMME / MLVU / LongVideoBench.
Further using entropy for early stopping. The same entropy signal also supports AdaptToken-Lite's early stopping, skipping additional group evaluations once the low-entropy criterion is met. On Qwen2.5-VL-7B, Lite reduces end-to-end latency by approximately 40–53% relative to full AdaptToken, with an average accuracy decrease of only 0.7 percentage points across four benchmarks (Table 2).