acceptodds
Under review as a conference paper at ICLR 2027

AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

Abstract

Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM’s self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B–72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-modal Large language model, long video understanding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_5ESmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an efficient, training-free framework tailored for long video understanding by integrating key token selection with an adaptive early exit mechanism. The proposed approach is empirically validated on relevant long video understanding benchmarks.

Critical strengths
  • Competitive Performance: The paper achieves competitive performance on the evaluated long video understanding benchmarks.
  • Practical Design: The integration of token compression and early exit mechanisms presents a practical and effective approach, particularly under the circumstance where computational overhead must be strictly managed.
  • Empirical Insights: The finding that utilizing information entropy as the metric for key token selection provides a heuristic and valuable insights for future research in token pruning.
Critical weaknesses
  • Incremental Novelty: The contribution appears largely incremental, primarily acting as an integration of multiple existing optimization techniques. Since each individual component has already been extensively explored and discussed in prior literature, the fundamental architectural novelty of the proposed framework is somewhat limited.
  • Limited Architectural Scope: The framework is currently evaluated predominantly on MLLMs employing the conventional "Vision Encoder + LLM" paradigm. Its generalizability warrants further validation on more unified, end-to-end multimodal architectures, such as those based on the Mixture-of-Transformers (MoT) paradigm.
Questions for authors
  • Broader Multimodal Applications: The current experimental evaluations are largely confined to standard vision-language understanding tasks (e.g., Visual Question Answering). To demonstrate the robustness and versatility of the proposed training-free framework, it would strengthen the paper to explore its efficacy across a broader spectrum of complex modality combinations, such as Vision-Language-Action or Vision-Language-Audio tasks.
4 comments
AuthorsOpenReview ↗

[1/3] Methodological contribution (W1)

Our core contribution is a training-free method for globally selecting visual evidence under MLLM memory and context constraints, with three methodological advances.

  1. Global token selection with attention and response entropy. Global selection is challenging because a long video's candidate tokens can exceed both GPU memory and the backbone's supported context length. To our knowledge, AdaptToken is the first framework to jointly use model attention and response entropy for global visual-token selection across separately processed video groups. Attention ranks tokens within groups, while entropy guides budget allocation across groups, enabling global selection without placing the entire candidate video in one context. On Qwen2.5-VL-7B, AdaptToken outperforms FlexSelect by 2.3–4.3 percentage points across four benchmarks (Table 1).

  2. From LLM answer voting to MLLM evidence selection and aggregation. Related entropy and self-certainty methods have mainly targeted text-only LLM reasoning, commonly using voting to combine answers (Kang et al., 2025; Fu et al., 2025b). We experimentally establish the association between response entropy and video evidence coverage (Fig. 3, left), and use this finding to guide MLLM token selection. We then aggregate selected visual tokens for joint final reasoning, allowing complementary evidence across groups to inform one answer. This produces more reliable answers in our evaluation: Table 5 shows gains of 3.0 / 2.5 / 2.1 percentage points over the strongest voting baseline on VideoMME / MLVU / LongVideoBench.

  3. Further using entropy for early stopping. The same entropy signal also supports AdaptToken-Lite's early stopping, skipping additional group evaluations once the low-entropy criterion is met. On Qwen2.5-VL-7B, Lite reduces end-to-end latency by approximately 40–53% relative to full AdaptToken, with an average accuracy decrease of only 0.7 percentage points across four benchmarks (Table 2).

AuthorsOpenReview ↗

[2/3] Applicability to Mixture-of-Transformers architectures (W2)

Our experiments indeed focus on VLMs following the “vision encoder + LLM” paradigm. We address architectural generalization through both transferability within this paradigm and extensibility beyond it.

  1. Transferability within the “vision encoder + LLM” paradigm. We evaluate AdaptToken on InternVL2.5-8B, Qwen2.5-VL-7B/72B, and Qwen3-VL-8B, covering three model families and four backbone configurations. The gains across four long-video benchmarks demonstrate effective transfer across these backbones and scales within this paradigm, without additional training or changes to their architectures or weights. This paradigm is still adopted by many recent VLMs, including Qwen3-VL and InternVL3.5.

  2. Extensibility beyond this paradigm to MoT. However, the core AdaptToken framework is not inherently limited to the “vision encoder + LLM” paradigm and has a natural extension path to compatible MoT backbones. For MoT models supporting video-conditioned autoregressive text generation, response entropy can still guide cross-group allocation. MoT’s modality-specific Q/K/V projections are followed by global self-attention, allowing text-to-visual attention to rank visual tokens wherever the attention mask permits, as reflected in the MoT implementation. The selected visual input representations can then be reassembled in temporal order for joint inference, preserving positional and modality information and adapting masks and input-layout metadata. These correspondences support structural feasibility; the resulting accuracy and efficiency require validation.

Given the limited rebuttal time, we have not completed the implementation and systematic evaluation on MoT. We will discuss this extension and its accuracy–efficiency validation as future work in the revision.

AuthorsOpenReview ↗

[3/3] Broader multimodal applications (Q1)

Following the reviewer’s suggestion, we have extended AdaptToken to the vision–language–audio setting and conducted an additional experiment.

  1. Benchmark and adaptation. We evaluate Qwen2.5-Omni-7B on LVOmniBench, which contains 1,014 questions on 10–90-minute videos, averaging 34 minutes. We apply the same reference-layer cross-modal attention scoring criterion to both visual and audio tokens, scoring 4,096 frames and the full audio track group-wise. Final inference retains approximately 7.1K visual and 7.5K audio tokens, totaling about 14.7K multimodal tokens, without additional training.

  2. Results. AdaptToken achieves 36.8% overall accuracy, exceeding the 32.0% reported for Qwen2.5-Omni-7B in LVOmniBench by 4.8 percentage points, with higher accuracy across all four categories. Accuracy scores (%) are shown below.

ModelUnderstandingPerceptionInferenceLogicalOverall
Qwen2.5-Omni-7B, reported in LVOmniBench29.134.931.828.232.0
Qwen2.5-Omni-7B + AdaptToken, training-free38.738.634.931.236.8

On the subset of videos longer than 60 minutes, AdaptToken achieves 44.1% accuracy over 34 questions.

  1. Implications and scope. These results provide direct empirical evidence that AdaptToken extends to joint visual and audio inputs using the same training-free token-selection framework. We will include this experiment in the revision. Given the limited rebuttal time, we have not conducted experiments on vision–language–action tasks and leave this extension for future work.

I acknowledge the authors' response to the reviews. After thoroughly evaluating the rebuttal and the perspectives shared by other reviewers, I will maintain my original recommendation.

Reviewer_CC5POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces AdaptToken, a training-free token selection framework designed for long-video understanding with Multimodal Large Language Models (MLLMs). To overcome context window limits and memory bottlenecks, the proposed pipeline divides a long video into interleaved frame groups. Within each group, visual tokens are scored using cross-modal attention extracted from an intermediate MLLM layer. Across groups, the model's response entropy (computed over the least certain output tokens) is used as a proxy for prompt relevance and confidence, dynamically allocating the visual token budget across groups via a temperature-scaled softmax. A location- and feature-similarity pruning step is subsequently applied to enforce diversity. Additionally, an early-stopping variant (AdaptToken-Lite) is introduced to terminate group evaluation once a threshold number of low-entropy groups is encountered.

The authors evaluate AdaptToken across four video benchmarks (VideoMME, MLVU, LongVideoBench, and LVBench) using several open-source MLLM backbones (InternVL2.5-8B, Qwen2.5-VL 7B/72B, and Qwen3-VL 8B). The paper reports accuracy improvements over base models and prior selection baselines, claiming scalability up to 10,000 frames.

Critical strengths
  1. Extensive empirical evaluations across diverse backbones: The authors conduct comprehensive experiments spanning four established long-video benchmarks (VideoMME, MLVU, LongVideoBench, LVBench) and validate the approach on multiple MLLM architectures and scales (InternVL2.5 8B, Qwen2.5-VL 7B/72B, and Qwen3-VL 8B), demonstrating consistent downstream score gains over uncompressed base models.

  2. Interesting empirical validation of response entropy as evidence sufficiency: The analysis in Section 3.1, along with the clue-coverage experiments on NEXT-GQA and the Needle-in-a-Haystack experiments in Appendix C, effectively demonstrates a visible correlation between lower response entropy and the presence of ground-truth visual clues in multiple-choice settings.

Critical weaknesses
  1. Questionable computational efficiency and self-defeating motivation: The primary motivation for token selection in long video is to alleviate the heavy computation and memory footprint of feeding long sequences into an MLLM. However, AdaptToken requires running the visual encoder AND the MLLM decoder on every single frame group just to compute the response entropy and cross-modal attention weights. As revealed in Appendix D.3, group-wise MLLM inference is the dominant bottleneck (1.05s per group). For a 10K-frame input with 64-frame groups, this necessitates processing ~156 separate MLLM passes before the "actual" final inference pass even begins. While AdaptToken-Lite introduces early stopping, it still incurs significant multi-pass decoding overhead. Compared to lightweight pre-filtering methods (e.g., using CLIP/SigLIP embeddings or lightweight visual encoders), invoking full MLLM decoders repeatedly across groups undermines the claim of practical inference efficiency.

  2. Unfair baseline comparisons and artificial "OOM" advantages: In Table 4, the authors emphasize that AdaptToken scales to 10,000 frames while baselines like AdaReTAKE and FlexSelect suffer from Out-Of-Memory (OOM) errors. However, this comparison appears fundamentally unfair: AdaReTAKE and FlexSelect ran out of memory because they were forced to ingest candidate frames into a single unified context window, whereas AdaptToken avoids OOM purely by virtue of its chunked, group-by-group processing harness. Any existing token-selection baseline wrapped in an equivalent group-wise chunking wrapper would circumvent context memory overflow. Attributing this memory scalability to the proposed entropy-guided selection algorithm itself is misleading.

  3. Methodological incrementalism and temporal degradation from interleaved grouping: The pipeline is an ad-hoc combination of existing heuristics: (i) MLLM cross-attention for intra-group ranking (following Retake/FlexSelect), (ii) output entropy for uncertainty (following Fadeeva et al. / Kang et al.), and (iii) pairwise cosine plus exponential temporal distance decay for token pruning. Furthermore, using interleaved sampling (Eq. 3) to construct groups means each group samples frames with a large stride G. This stride disrupts local temporal continuity and motion cues within each individual group, which can severely hinder the model from correctly understanding fine-grained temporal dynamics during the intra-group scoring phase.

Questions for authors
  1. Wall-clock time and FLOPs breakdown vs. lightweight retrieval: Could the authors provide a rigorous end-to-end FLOPs and wall-clock latency comparison against a competitive two-stage pipeline that uses a lightweight, fast retriever (e.g., SigLIP2 or a small visual encoder) for frame/token filtering followed by a single MLLM call? Given that group-wise MLLM inference takes 1.05s per group, what is the exact cumulative time/FLOPs consumed by AdaptToken when scaling to 10K frames across all 156 groups on the entire benchmark?

  2. Fair OOM comparison: If AdaReTAKE or FlexSelect were deployed with the exact same group-wise chunking strategy (processing 64 frames per chunk sequentially and pooling scores), would they still encounter OOM at 4,096 or 10,000 frames? If not, please provide these benchmark results under a strictly matched memory harness.

  3. Redundancy of the final MLLM pass in AdaptToken-Lite: In AdaptToken-Lite, early stopping is triggered precisely because the MLLM has already generated low-entropy (i.e., highly confident) answers for m examined groups. If the model is already confident in those groups, what is the performance if you simply return the answer generated during the early-stopping group check (e.g., via confidence-weighted selection) without performing the final token aggregation and second-pass MLLM inference?

  4. Impact on temporal reasoning tasks: Interleaved sampling across distant strides breaks continuous motion. Did the authors evaluate performance specifically on temporal-ordering or action-speed benchmark subsets (e.g., in VideoMME or LongVideoBench) to verify whether interleaved grouping degrades temporal reasoning compared to contiguous chunking?

5 comments
AuthorsOpenReview ↗

[1/5] Computational cost and comparison with lightweight retrieval (W1, Q1)

(a) Two-stage comparison. We thank the reviewer for this suggestion. We reuse the SigLIP2-so400m (NaFlex) token-retrieval implementation from Table 4, which uses the same answering MLLM and final token budget. For the comparison below, we report FLOPs and H100 latency for the same 10,000-frame input (720 × 540; mean over three MLVU videos).

StageAdaptTokenSigLIP2-so400m NaFlex + one MLLM call
Video decoding + preprocessing (CPU)202.6 s215.1 s
ViT on all frames (GPU)5.35 PF / 29.2 s7.09 PF / 38.3 s
MLLM selection (GPU)79 group-wise LLM passes: 17.26 PF / 76.1 s—
Final MLLM call (+ de-duplication) (GPU)0.11 PF / 0.4 s0.10 PF / 0.2 s
GPU total22.7 PF / 105.7 s7.2 PF / 38.5 s
End-to-end (CPU + GPU)308.3 s253.7 s

(b) End-to-end cost. The timings in Table 4 cover GPU work only. When video processing is included, the wall-clock gap narrows further. On H100, end-to-end latency is 308.3 s for AdaptToken and 253.7 s for the retriever, including both CPU and GPU work—a ratio of 1.22×. A smaller scorer accelerates selection, but the evaluated SigLIP2 baseline still decodes and preprocesses all candidate frames. These stages limit the end-to-end speedup from cheaper scoring.

(c) Accuracy at comparable GPU latency. SigLIP2 remains less accurate than AdaptToken at similar GPU latency, even with a higher frame cap. In Table 4, AdaptToken achieves 74.9% accuracy in 10.8 s at the 1,024-frame cap. SigLIP2 achieves 66.7% in 10.7 s at the 4,096-frame cap and 65.9% in 12.6 s at the 10,000-frame cap. At the 10,000-frame cap, AdaptToken achieves 77.1% versus 65.9% for SigLIP2.

(d) Benefit of early stopping. AdaptToken-Lite preserves this accuracy advantage through early stopping. At the 4,096-frame cap, it achieves 76.3% in 10.1 s versus 66.7% in 10.7 s for the SigLIP2 variant; at the 10,000-frame cap, it achieves 76.7% in 11.4 s versus 65.9% in 12.6 s. Lite therefore improves accuracy by 9.6 and 10.8 percentage points, respectively, at slightly lower GPU latency. Compared with Full AdaptToken, Lite reduces GPU time by 33–53% across the frame caps while accuracy decreases by only 0.3–0.5 percentage points. These comparisons support the efficiency of early stopping and show that entropy-guided global token selection remains effective when fewer groups are processed.

AuthorsOpenReview ↗

[2/5] Baseline protocols, OOM, and additional H200 results (W2, Q2)

(a) Preserving the original baseline methods. Thank you for raising this concern. However, introducing an additional group-wise chunking strategy into AdaReTAKE or FlexSelect could alter the original methods and their performance. To remain consistent with the methods presented in the original papers and ensure a fair comparison, we used the officially released implementations with their recommended configurations, without modifying the original algorithms.

(b) The accuracy–latency advantage is already visible in non-OOM comparisons. On MLVU with Qwen2.5-VL-7B, at the 2,048-frame cap, AdaReTAKE achieves 75.0% accuracy in 29.7 s, whereas AdaptToken achieves 76.0% in 18.7 s and AdaptToken-Lite achieves 75.6% in 9.6 s. At the 4,096-frame cap, the corresponding results are 75.0% / 35.0 s, 76.8% / 21.5 s, and 76.3% / 10.1 s. Thus, both AdaptToken variants are more accurate and substantially faster than AdaReTAKE at both caps. At the 2,048-frame cap, FlexSelect has nearly identical latency to AdaptToken (18.6 vs. 18.7 s), but is 3.3 percentage points less accurate (72.7% vs. 76.0%). AdaptToken-Lite also outperforms both FlexSelect and FlexSelect-Lite in accuracy and latency: 75.6% / 9.6 s versus 72.7% / 18.6 s and 70.1% / 11.1 s. These comparisons establish an accuracy–efficiency advantage without relying on any OOM entry.

(c) Additional results on H200 with the original baseline implementations. To reduce the impact of the original hardware's memory limit while preserving the baseline algorithms, we additionally evaluated the previously OOM configurations on H200. At the 4,096-frame cap, FlexSelect-Lite achieves 70.3% accuracy in 13.6 s, and FlexSelect achieves 73.0% in 20.8 s. At the 10,000-frame cap, AdaReTAKE achieves 75.1% in 40.6 s, while both FlexSelect variants still encounter OOM. These additional results are consistent with our conclusions: at 4,096 frames, both FlexSelect variants remain below the accuracies reported for AdaptToken / AdaptToken-Lite (76.8% / 76.3% in Table 4); at 10,000 frames, AdaReTAKE remains below 77.1% / 76.7%. Thus, the accuracy gap persists even when a previously OOM baseline configuration becomes feasible.

AuthorsOpenReview ↗

[3/5] Methodological contribution of response entropy (W3)

Thank you for raising the novelty concern. Our main contribution addresses item (ii): turning response entropy into a control signal for global visual-token selection and adaptive computation. It has three aspects.

(a) Global selection under memory and context constraints. Computing attention jointly over an entire long video is constrained by both GPU memory and the backbone's effective context capacity. We combine within-group attention with response entropy to allocate a fixed visual-token budget across groups. To our knowledge, this is the first use of response entropy to guide cross-group global visual-token budgeting for long-video MLLMs. At the same 4,096-frame cap, adding global allocation improves VideoMME / MLVU / LongVideoBench / LVBench by 1.8 / 3.1 / 2.6 / 1.7 percentage points (Table 10, d→e). The full framework exceeds FlexSelect by 2.3–4.3 points across these benchmarks with Qwen2.5-VL-7B (Table 1).

(b) Connecting uncertainty to video evidence and token aggregation. Related self-certainty methods primarily target LLM reasoning, using output uncertainty to select reasoning traces or weight answer votes (Kang et al.; Fu et al.). We establish an empirical link to video evidence: holding the question, source video, and 64-frame input length fixed, increasing annotated clue coverage shifts response entropy toward lower values (Fig. 3, left). We use this signal to select and aggregate visual tokens across groups for joint final reasoning, allowing evidence from different groups to be combined before producing the final answer. AdaptToken outperforms the strongest voting baseline by 3.0 / 2.5 / 2.1 percentage points on VideoMME / MLVU / LongVideoBench (Table 5).

(c) Reusing entropy for early stopping. The same signal determines when to stop examining further groups, without an additional stopping model. AdaptToken-Lite reduces end-to-end latency by approximately 40–53%, with a mean accuracy decrease of only 0.7 percentage points across the four benchmarks on Qwen2.5-VL-7B (Table 2).

Regarding items (i) and (iii), attention-based ranking and similarity-based pruning are supporting components, and we do not present them as our main contributions. We acknowledge prior attention-based token selection/compression, including ReTaKe and FlexSelect. We build on these components to handle longer candidate inputs: each group's attention and entropy are computed in a fully independent MLLM call, without carrying decoder context or KV caches between groups. With a fixed group size and final token budget, the context size and decoder working memory required by each call remain bounded as the candidate video grows.

Reviewer_9PSeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdaptToken, a training-free method for long-video understanding that uses response entropy to allocate visual tokens across frame groups and supports early stopping to reduce computation. Experiments on four benchmarks with multiple multimodal models show consistent accuracy improvements, while the lightweight version achieves comparable performance with substantially lower inference time.

Critical strengths
  • AdaptToken improves all four benchmark scores across the evaluated backbones, including Qwen2.5-VL-72B and Qwen3-VL-8B, supporting its usefulness beyond one model configuration.
  • Lite provides a useful accuracy–latency tradeoff on Qwen2.5-VL-7B, reducing latency by approximately 40–53% across benchmarks with an average accuracy decrease of 0.7 points.
Critical weaknesses
  • As written, Algorithm 1 compares negative certainty against a positive entropy cutoff and checks stopping only after visiting all groups, so it does not match the stated stopping procedure
  • The numerical sampling FPS and selected attention-layer indices are not reported, making the accuracy and runtime results difficult to reproduce precisely.
  • The 10K experiments report maximum-frame caps without actual frame-count distributions, leaving unclear how many examples were evaluated near 10,000 frames.
Questions for authors
  • Could you provide corrected stopping pseudocode and the complete evaluation settings, including sampling FPS, group sizes, and attention-layer choices for each backbone?
  • How does entropy-based stopping compare with a fixed group budget using the same candidate frames and traversal order at matched average latency?
  • Could you clarify the distinction between AdaptToken’s entropy-guided token allocation and the confidence-guided temporal search and stopping used in TimeSearch/ZoomV?
6 comments
AuthorsOpenReview ↗

[1/5] Corrected stopping pseudocode (W1, Q1)

We thank the reviewer for pointing out the inconsistencies in Algorithm 1. We will correct the threshold sign and move the stopping check inside the group traversal loop.

Group response certainty is defined as C_g = −H_g, where H_g is response entropy. Therefore, the certainty threshold should be C* = −0.75, so that C_g > C* is equivalent to H_g < 0.75. AdaptToken-Lite checks this condition after scoring each group and immediately exits the traversal once three examined groups qualify. These groups need not be consecutive. If fewer than three qualify, all groups are examined.

The corrected pseudocode is:

C_star, m = -0.75, 3
seen, hits = [], 0

for g in max_margin_order(groups):
    H_g, R_g, visual_g = score_group(g, prompt)
    C_g = -H_g
    seen.append((C_g, R_g, visual_g))

    if lite and C_g > C_star:
        hits += 1
        if hits >= m:
            break

weights = softmax([C_g / 2 for C_g, _, _ in seen])
candidates = allocate_and_select(seen, weights, 1.1 * B)
tokens = remove_redundancy(candidates, budget=B, sigma=0.3)
return answer(temporal_sort(tokens), prompt)

Final token selection uses all examined groups, including those that do not satisfy the stopping threshold. Full AdaptToken processes all groups.

AuthorsOpenReview ↗

[2/5] Sampling and attention-layer settings (W2, Q1)

We sample frames at 2 FPS for InternVL2.5-8B and at 4 FPS for Qwen2.5-VL-7B, Qwen2.5-VL-72B, and Qwen3-VL-8B. All four backbones use a group size of 64 frames.

For token scoring, we select the decoder attention layer using the layer-wise text-to-visual retrieval evaluation described in Sec. 3.2. We use layer 15 for InternVL2.5-8B, layer 20 for Qwen2.5-VL-7B and Qwen3-VL-8B, and layer 60 for Qwen2.5-VL-72B. We will add these settings to the implementation details.

AuthorsOpenReview ↗

[3/5] Frame-count distribution under the 10K cap (W3)

To clarify how many evaluation examples contain long frame sequences, we computed the pre-cap candidate-frame counts from video durations at the 4 FPS sampling rate used by Qwen2.5-VL-7B in Tables 4 and 8. Counts below are reported per question instance.

Evaluation splitTotal questionsPre-cap frames > 4,096Pre-cap frames ≥ 10,000
Video-MME test2,700912420
MLVU-Dev, multiple-choice2,17414344
LongVideoBench validation1,33740854
LVBench1,5491,5491,144

Across the four evaluation splits, 3,012 of 7,760 question instances (38.8%) exceed 4,096 frames, and 1,662 (21.4%) reach or exceed 10,000 frames before capping. In particular, all 1,549 LVBench questions exceed 4,096 frames, and 1,144 (73.9%) reach or exceed 10,000 frames. These statistics demonstrate substantial long-input coverage in the 10K-cap evaluation, especially on LVBench.

Reviewer_KAhiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaptToken, a training-free adaptive token selection framework for long-video understanding with MLLMs. It uses response entropy to estimate the relevance of different video groups and combines it with cross-modal attention for global token budget allocation and intra-group token selection. The authors further introduce an entropy-based early-stopping mechanism to avoid processing unnecessary video groups. Experiments on four long-video benchmarks and multiple MLLM backbones demonstrate consistent accuracy improvements and reduced inference latency.

Critical strengths

S1. The paper addresses a practical problem in long-video understanding: globally allocating visual tokens across temporally distant video segments rather than performing only local token compression.

S2. Using response entropy as a group-level relevance signal is an interesting idea. Its integration into both global token allocation and early stopping provides a relatively unified framework.

S3. The experimental evaluation is relatively comprehensive, covering four long-video benchmarks, multiple MLLM backbones, and extensive ablations. The 10K-frame experiments and efficiency improvements of AdaptToken-Lite are particularly encouraging.

Critical weaknesses

W1. Reliability of entropy as a relevance measure remains questionable. The method assumes that lower response entropy indicates that a video group contains more relevant evidence. However, MLLMs can also produce highly confident but incorrect answers when sufficient evidence is absent. Thus, entropy may reflect the model's confidence rather than the actual relevance of the video content. The current experiments do not sufficiently distinguish these two cases.

W2. Early stopping may miss important video evidence. For questions requiring reasoning across multiple temporal segments, the model may generate low-entropy responses based on the already examined groups, while decisive evidence may appear in later, unprocessed segments. Relying solely on local response confidence to determine when to stop may therefore compromise the reliability of temporal reasoning.

W3. The source of computational efficiency gains needs further clarification. AdaptToken requires separate MLLM inference over multiple frame groups, which introduces additional computational overhead. Although AdaptToken-Lite reduces latency through early stopping, the full AdaptToken pipeline is not necessarily more efficient than existing methods in all settings. A more detailed analysis of computational costs under different video lengths and token budgets would strengthen the efficiency claims.

Questions for authors
  1. How does AdaptToken prevent confidently incorrect responses from assigning excessive token budgets to groups that do not actually contain relevant evidence?

  2. For questions requiring reasoning across multiple temporal segments, how does early stopping avoid missing decisive evidence in unprocessed groups? Can the authors provide dedicated experiments on temporal reasoning tasks?

3 comments
AuthorsOpenReview ↗

[1/3] Reliability of response entropy and protection against confidently incorrect answers (W1, Q1)

Evidence for the entropy signal. In the no-evidence conditions in Figs. 3 (left) and 5, highly confident, low-entropy responses are uncommon, and even correct guesses in the no-needle condition typically have high entropy. Moreover, Fig. 3 (left) shows that increasing annotated clue coverage for the same question and source video shifts the response distribution toward lower entropy. This dependence on evidence coverage supports using response entropy to guide token allocation across groups.

Soft allocation and joint reasoning. Even when a group produces a confident answer from insufficient evidence, our softmax allocation (τ = 2, Eq. 5) shares the budget across groups. Other groups containing relevant evidence receive equal or larger budget shares if their certainty matches or exceeds that of the overconfident group. We then select visual tokens within each group using cross-modal attention and pool them for a fresh prediction. This allows the final pass to combine relevant cues from multiple groups even if some retained tokens are irrelevant. Consistent with this rationale, Table 5 shows that full AdaptToken with InternVL2.5-8B outperforms the strongest voting baseline, certainty-weighted voting over group answers, by 3.0, 2.5, and 2.1 percentage points on VideoMME, MLVU, and LongVideoBench, respectively.

AuthorsOpenReview ↗

[2/3] Early stopping and temporal evidence in unprocessed groups (W2, Q2)

Temporal coverage. Interleaved grouping preserves broad temporal coverage even when processing stops early: each group spans the full video, so unprocessed groups add sampling density across the same timeline rather than introduce wholly unseen segments. With the coarse-to-fine processing order, the groups examined before AdaptToken-Lite stops (median: 3) already contain 1.5× (minimum: 1.3×) as many evenly spread frames as the base model's entire input.

Stopping behavior. This coverage is complemented by the stopping criterion, which requires three groups with response entropy below 0.75. Questions requiring evidence from multiple parts of a video are less likely to meet this criterion: early stopping triggers on 37% of MLVU's action-order and action-count questions, compared with 62% of single-detail questions. The policy therefore tends to continue processing on the temporal questions where distributed evidence is most important.

Dedicated temporal experiments. We evaluate action order and action count on MLVU dev and report accuracy (%) below. To test the role of temporal coverage, we apply the same stopping rule to contiguous groups visited in temporal order, keeping all other settings identical. In this setting, action-count accuracy falls from 50.0% with AdaptToken to 42.2% with AdaptToken-Lite, consistent with missing occurrences in unprocessed later segments. By comparison, interleaved Lite stays close to full AdaptToken on both action order and action count.

Task#QBaseAdaptToken (interleaved)AdaptToken (contiguous)AdaptToken-Lite (interleaved)AdaptToken-Lite (contiguous)
Action order25958.779.572.678.872.6
Action count20640.858.350.056.342.2
Order + count46550.870.162.668.859.1

Contiguous: Every group is one consecutive segment of the same length, and AdaptToken-Lite examines the segments in temporal order.

Interleaved AdaptToken-Lite thus retains most of AdaptToken's temporal-reasoning gains while substantially outperforming the Base model and both contiguous variants. On the combined subset, it achieves 68.8%, close to full AdaptToken's 70.1% and well above Base (50.8%) and contiguous Full / Lite (62.6% / 59.1%). This strong temporal-task performance is achieved with about 40% fewer group forward passes on MLVU.

AuthorsOpenReview ↗

[3/3] Efficiency gains and costs across video lengths and token budgets (W3)

The main efficiency gain of AdaptToken-Lite comes from early stopping. FlexSelect-Lite and our implemented SigLIP2 variant reduce scoring cost by using a smaller model. This approach has two limitations.

(a) Selection quality. At the 1,024-frame cap in Table 4, FlexSelect-Lite achieves 70.3% accuracy versus 72.5% for FlexSelect. AdaptToken-Lite retains the same MLLM scorer as Full, with a much smaller accuracy change (74.9% → 74.6% at this cap).

(b) End-to-end cost. Both evaluated token selectors still process all candidate frames, so video decoding, preprocessing, and backbone visual encoding remain necessary. Accelerating scoring alone therefore gives a bounded end-to-end speedup.

Savings from early stopping. With the stopping rule described above, every unvisited group avoids its visual encoding and group-wise MLLM inference; with preprocessing performed on demand, it also avoids that group’s preprocessing. We verified bit-identical preprocessing inputs on 24 MLVU videos (43 questions). Video decoding remains an upfront pass. Thus, early stopping saves work across several stages while preserving MLLM-based selection quality. In Table 4, Lite reduces model-side time by 33–53% across the frame caps, with accuracy decreases of only 0.3–0.5 percentage points. Full AdaptToken evaluates all groups and is not uniformly cheaper than lightweight retrievers.

Costs by video length. We use Table 4 as the reference: on MLVU dev, Full and Lite take 21.5 and 10.1 s/question, respectively. All times below follow Table 4’s model-side timing convention and exclude CPU video decoding and preprocessing.

Video length (# questions; bin estimates)AdaptToken (s)AdaptToken-Lite (s)
≤ 8.5 min (1,514)17.58.4
> 8.5 min (660)30.814.1
All (Table 4)21.510.1

Longer videos require more group evaluations until the frame cap is reached, making group-wise inference the main scaling cost reduced by Lite.

Costs by final token budget. We also vary the final token budget while fixing the candidate-frame cap at 4,096. The times below follow the same model-side timing convention, in seconds per question.

Method1,024 tokens (est.)7,010 tokens (Table 4)14,080 tokens (est.)
AdaptToken21.121.522.9
AdaptToken-Lite9.810.111.6

The group passes and stopping decisions do not depend on the final token budget. Increasing this budget mainly changes final token processing and answer inference, so the total cost varies much less with token budget than with input length.

Comments on the paper

1 comment

We thank all four reviewers for their time, careful reading, and constructive feedback. We appreciate their recognition of the following strengths of AdaptToken:

1. Practical relevance and unified framework design.

Reviewer KAhi highlights the practical need to allocate visual tokens globally across temporally distant video segments and the use of a shared response-entropy signal for allocation and early stopping. Reviewer 5ESm also recognizes the practical value of combining token compression with early exit under computational constraints.

2. Response-entropy insights and empirical grounding.

Reviewer KAhi finds response entropy an interesting group-level relevance signal, and Reviewer 5ESm notes its value as an insight for future token-pruning research. Reviewer CC5P specifically recognizes that the clue-coverage and Needle-in-a-Haystack experiments demonstrate a correlation between lower response entropy and the presence of ground-truth visual clues in multiple-choice settings.

3. Consistent performance and comprehensive evaluation.

Reviewers 9PSe and CC5P recognize consistent gains across four long-video benchmarks and multiple evaluated backbones and scales. Reviewer KAhi highlights the breadth of the evaluation and ablations, including the encouraging 10K-frame experiments, while Reviewer 5ESm notes the competitive benchmark performance.

4. Accuracy–latency trade-off of AdaptToken-Lite.

Reviewers KAhi and 9PSe recognize Lite’s efficiency improvements, with 9PSe highlighting approximately 40–53% lower latency relative to full AdaptToken for a mean accuracy decrease of 0.7 percentage points on Qwen2.5-VL-7B. Reviewer 5ESm likewise values the integration of token compression and early exit for managing computational cost.

We have addressed the reviewers’ questions and concerns individually in the point-by-point responses below, providing clarifications, additional analyses, and new experimental results. We thank the reviewers again for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.