acceptodds
Under review as a conference paper at ICLR 2027

EviQ: Hierarchical Query-Conditioned Evidence Selection for Efficient VideoLLMs

Abstract

Video large language models (VideoLLMs) have achieved strong performance in video understanding, yet their large visual token budgets introduce substantial inference latency and memory consumption. Most existing training-free token compression methods rely primarily on visual cues, potentially overlooking query-critical evidence scattered across sparsely sampled frames, while some query-aware approaches incur additional computation through within-LLM attention or auxiliary models. Instead, we reveal that cosine similarity between projected visual tokens and embedded query tokens provides a lightweight relevance signal both across and within sparsely sampled frames. Building on this observation, we propose **EviQ**, a training-free pre-LLM token pruning framework that constructs a compact query-conditioned evidence set in two stages. **EviQ** first routes the global token budget across frames according to frame-level query relevance, and then selects tokens within each frame by jointly preserving informative visual context and promoting complementary coverage of query semantics. Extensive experiments on diverse VideoLLMs and multiple benchmarks demonstrate that **EviQ** achieves state-of-the-art accuracy-efficiency trade-offs among training-free token compression methods. Notably, **EviQ** preserves **99.9%** of vanilla LLaVA-OV-7B accuracy using only 25% of visual tokens, while achieving a 1.59 speedup and reducing GPU memory consumption by 8.7%. Code is released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VideoLLMs, Token Pruning, Query Awareness, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_XT4xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviQ, a training-free, query-conditioned visual token pruning framework for efficient VideoLLM inference. The method aims to reduce the computational and memory overhead of processing video tokens while preserving information relevant to the input query. EviQ adopts a two-stage hierarchical approach. First, it dynamically allocates token budgets across video frames based on cosine similarity between visual and query embeddings. Second, it selects tokens within each frame by combining visual informativeness, measured through token L2 norms, with query-semantic coverage formulated as a submodular optimization problem. A greedy selection algorithm is employed with a theoretical approximation guarantee.

Critical strengths
  1. Paper is easy to understand.
  2. EviQ is training-free and performs token pruning before LLM inference, making it easy to integrate with existing VideoLLMs. It achieves a favorable accuracy-efficiency trade-off, retaining nearly the original accuracy with only 25% of visual tokens.
Critical weaknesses
  1. Limited novelty. The proposed framework primarily combines existing techniques, including cosine similarity, token L2 norms, softmax-based budget allocation, and greedy submodular optimization. The method lacks a fundamentally new mechanism for video understanding, particularly for modeling temporal dependencies and cross-frame interactions.

  2. Marginal improvements and insufficient justification of key components. The improvements are relatively small.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Point-by-Point Responses

Clarification on Methodological Novelty and Our Paradigm for Sparse Video Understanding (for Weakness 1)

  • Core Motivation and Efficiency Advantage of EviQ: This work aims to preserve dense VideoLLM inference performance while keeping the pruning pipeline minimally complex. Because sparse frame sampling in VideoLLMs weakens inter-frame correlation and confines query-relevant evidence to only a few frames, diversity-based methods relying on inter-frame redundancy risk dropping critical cues. Conversely, existing query-aware approaches depend on auxiliary models or intra-LLM attention, which bottlenecks actual inference acceleration. EviQ resolves this dilemma by exploiting the synergy between visual context and query coverage, achieving state-of-the-art accuracy-efficiency trade-offs without heavy computational overhead.

  • We disagree that utilizing cosine similarity and token L2L_2 norms lacks novelty for video understanding. Taking cosine similarity as an example, while CDPruner[1] initially adopted it via pre-aligned CLIP contrastive embeddings, that work merely noted that projector outputs could serve as visual embeddings for dedicated visual encoders without providing empirical proof or video-level validation, and methods like CRISP[2] similarly lack deep investigation. In contrast, we systematically compare representations in the video domain and visually demonstrate that cosine similarity captures both inter-frame and intra-frame query relevance under sparse sampling. Furthermore, integrating token L2L_2 norms, softmax-based budget allocation, and greedy submodular optimization is the result of rigorous design validated by extensive ablations. Adapting foundational metrics from other domains to address unique challenges in sparse video pruning represents a targeted empirical insight rather than a lack of novelty.

  • A New Two-Stage Paradigm for Sparse Video Understanding: Far from lacking a fundamentally new mechanism for video understanding, EviQ introduces a paradigm shift tailored to the sparse nature of VideoLLM inputs. Because VideoLLMs process downsampled videos with low inter-frame correlation, explicitly modeling temporal dependencies and cross-frame interactions becomes inherently less reliable. We therefore reformulate sparse video pruning as a global optimization problem solved in lightweight two decoupled stages, combining inter-frame budget allocation to capture global query relevance with intra-frame submodular optimization to jointly preserve visual context and query coverage. Thus, the claim that our method lacks a new mechanism does not hold.

[1]Zhang, Qizhe, et al. "Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs" NIPS 2025

[2]Li, Xu, et al. "CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference." ICME 2026.

Clarification on Accuracy-Efficiency Trade-offs and Component Ablations (for Weakness 2)

  • Evaluating Holistic Efficiency-Accuracy Trade-offs Beyond Marginal Accuracy Gains: We disagree with the characterization of our improvements as marginal, as evaluating a token pruning framework solely on accuracy overlooks its primary goal of inference acceleration. While EviQ achieves competitive or superior accuracy against state-of-the-art baselines, its key advantage lies in substantially reducing computational overhead. On LLaVA-OneVision-7B, although our accuracy gain over FlashVid[3] is 0.1, EviQ slashes the pruning computation time by 38% and reduces overall end-to-end latency by 9%. Assessing performance purely through accuracy metrics without accounting for these significant speedups presents an incomplete evaluation of our method.

  • Comprehensive Ablations and Robustness Verification of Key Components: Regarding the concern over insufficient justification of key components, we have already provided rigorous empirical validation throughout the manuscript. Specifically, we systematically ablated the individual contributions of QCBR, PCE, and QCE, and compared them against various alternative formulations in Table 6 and Table 14 (origin Table 13). Moreover, we demonstrated the stability of our method through extensive hyperparameter robustness analyses in Figure 4 and Table 15 (origin Table 14). These results collectively confirm both the necessity of our core components and the robustness of EviQ, and we welcome further clarification from the reviewer on any specific aspect that may require additional analysis.

[3]Ziyang Fan, et al. "FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging" ICLR 2026

AuthorsOpenReview ↗

Concluding Remarks

To better highlight our novelty in realizing a lightweight, decoupled pre-LLM two-stage pruning paradigm under sparse video sampling and to emphasize our state-of-the-art accuracy-efficiency trade-offs, we have refined the formulations and experimental analyses in the Introduction (including Contributions), Experiments (Sections 4.2 & 4.3), and Conclusion, with all revisions highlighted in blue in the updated manuscript.

Overall, we hope these responses resolve your concerns and clarify the technical contributions of our work, and we warmly welcome any further questions during the rebuttal discussion phase.

Reviewer_6PSoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviQ, a training-free visual token pruning method that uses cosine similarity between projected visual tokens and LLM text embeddings to construct compressed inputs through query-conditioned frame-level budget allocation and intra-frame token selection.

Critical strengths
  • The method is evaluated across multiple benchmarks and model families. Experiments on Qwen and LLaVA models provide evidence of its applicability across different VideoLLM architectures.
  • EviQ is straightforward to integrate into existing models. It operates before LLM inference without additional training, auxiliary models, or access to internal LLM attention.
Critical weaknesses
  • The baseline comparison could include additional recent methods, such as EarlyTom. Although several 2026 baselines are included, comparing against EarlyTom would strengthen the evaluation of end-to-end inference efficiency.
  • The accuracy improvements over strong baselines are small, and their statistical significance remains unclear. On LLaVA-OV-7B, the reported average improvement over FlashVID is approximately 0.1 percentage points, comparable to the variation across seeds; several other average gains are also below 0.5 points. Paired significance tests or confidence intervals would help establish whether these improvements are reliable.
  • Additional comparisons under matched token and compute budgets would strengthen the evaluation. Retaining 25% of tokens from 128 frames could be compared with an uncompressed 32-frame input, assuming the same tokens per frame, while additional 64-frame experiments would help characterize scaling. Because encoding more frames incurs extra cost, equal retained-token counts should not be treated as equal end-to-end compute budgets.
  • The average improvements are not consistent across individual benchmarks. On Qwen3-VL-8B with 25% retention, EviQ scores 60.9 on VideoMME, below FlashVID’s 62.1 and VidCom²’s 62.5; at 35% retention, it scores 61.8 versus CDPruner’s 63.6. Further analysis is needed to explain when EviQ helps and why it underperforms on this benchmark.
Questions for authors

See weakness above.

4 comments
AuthorsOpenReview ↗

Point-by-Point Responses (Part 1/2)

Empirical Comparison with EarlyTom on End-to-End Efficiency and Accuracy (for Weakness 1)

Following the suggestion to include recent baselines for a more comprehensive evaluation of end-to-end inference efficiency, we integrated the official implementation of EarlyTom[1] into our evaluation framework. In terms of accuracy, EarlyTom merges tokens based on visual similarity between adjacent frames, a query-agnostic signal that struggles to pinpoint frame-level evidence in sparsely sampled long videos. By dynamically allocating frame budgets conditioned on the query and preserving complementary intra-frame cues, EviQ consistently surpasses EarlyTom on long-form benchmarks including LongVideoBench and MLVU across both backbones. In terms of efficiency, sparse sampling enlarges the temporal gap between consecutive frames, resulting in fewer frame pairs meeting the merging threshold in EarlyTom and thus limiting its computational savings in subsequent visual layers. EviQ circumvents this bottleneck by reusing a single visual-query similarity matrix for both inter-frame budget allocation and intra-frame selection, achieving superior average accuracy while delivering lower total end-to-end latency on both LLaVA-OV-7B and LLaVA-Video-7B.

MethodPruning RatioMVBenchLongVideoBenchMLVUVideoMMEAverage PerformanceOverhead Latency(ms)Total Latency(ms)
LLaVA-OV-7B/58.356.463.058.459.00462.4
EarlyTom[CVPR26]25%57.855.762.758.158.628.7303.9
EviQ(Ours)25%57.757.463.057.859.015.7291.1
LLaVA-Video-7B/60.358.867.364.362.70545.8
EarlyTom[CVPR26]25%58.056.161.362.759.532.5360.7
EviQ(Ours)25%57.957.565.162.360.715.6328.6

[1]Hesong Wang, et al. "EarlyTom: Early Token Compression Completes Fast Video Understanding" CVPR 2026

Deterministic Evaluation and Consistent Superiority Across Random Seeds (for Weakness 2)

Regarding the concern that our accuracy gains over strong baselines such as FlashVID[1] on LLaVA-OV-7B are small and comparable to cross-seed fluctuations, we have thoroughly analyzed multi-seed stability in Section F.1. Specifically, under any fixed random seed, the inference and token selection pipeline is fully deterministic, meaning repeated runs of the same method yield identical accuracy with zero variance. More importantly, when varying the random seeds to control for sampling fluctuations, although absolute accuracies shift slightly across seeds, EviQ consistently surpasses competing baselines under every individual seed configuration. This strict paired dominance across multiple random seeds confirms that our performance gains are structurally reliable rather than artifacts of random noise, and it is further accompanied by substantial reductions in pruning overhead and end-to-end latency.

AuthorsOpenReview ↗

Point-by-Point Responses (Part 2/2)

Iso-Token Frame Scaling Analysis (for Weakness 3)

Following the suggestion to compare different frame counts under an identical token budget and analyze end-to-end computational costs, we evaluated LLaVA-OV-7B with 32 uncompressed frames, 64 frames at 50% retention, and 128 frames at 25% retention. In terms of accuracy, under the exact same LLM token budget, sampling more frames with EviQ steadily improves overall average accuracy, with the most pronounced gains observed on long-video understanding benchmarks including LongVideoBench and MLVU. This confirms that pairing denser temporal sampling with EviQ locates and retains more critical evidence than keeping all spatial tokens from fewer frames. In terms of efficiency, the results validate the observation of the reviewer that encoding more frames introduces an almost linear increase in visual encoder latency alongside a modest compression overhead from our method, while keeping the LLM prefill latency constant across all settings. Overall, this demonstrates the strong scalability of EviQ across varying frame counts, enabling models to process denser video inputs for superior temporal reasoning.

Method#FramePruning RatioMVBenchLongVideoBenchMLVUVideoMMEAverage PerformanceEncode Latency(ms)Prefill Latency(ms)Overhead Latency(ms)Total Latency(ms)GPU Average Memory(MB)
LLaVA-OV-7B32/58.356.463.058.459.0175.2255.70462.422086
EviQ(Ours)6450%58.857.464.358.859.8365.0255.99.2652.327473
EviQ(Ours)12825%58.957.766.058.960.4717.3256.615.51005.933285

Superiority in Long-Form Video Understanding and Analysis across VideoMME Subsets (for Weakness 4)

EviQ excels particularly in long-form video understanding, consistently achieving either the best or second-best accuracy across all backbone models on canonical long-video benchmarks including LongVideoBench and MLVU, while also performing strongly on VideoMME-Long. Regarding the observation that EviQ trails certain baselines on VideoMME, this mainly stems from the VideoMME-Short and VideoMME-Medium subsets. Under a fixed frame sampling budget, shorter videos exhibit much denser temporal intervals where consecutive frames often contain duplicate query-relevant cues. Inter-frame merging methods like FlashVID[2] can collapse these cross-frame redundancies to free up budget for broader visual tokens, whereas EviQ evaluates query coverage independently within each frame and may spend budget on relevant evidence repeated across adjacent frames. Even so, EviQ remains highly competitive on shorter subsets and ultimately secures the highest overall average accuracy across all evaluated benchmarks with substantially lower end-to-end latency, confirming its superior holistic accuracy-efficiency trade-off.

[2]Ziyang Fan, et al. "FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging" ICLR 2026

AuthorsOpenReview ↗

Concluding Remarks

We have incorporated comparisons with EarlyTom in Tables 1, 2, 4, and 11 (originally Table 10), and included the iso-token frame scaling and end-to-end latency analysis on LLaVA-OV-7B in Appendix F.2 and Table 10. Moreover, to better highlight our novelty in realizing a lightweight, decoupled pre-LLM two-stage pruning paradigm under sparse video sampling and to emphasize our state-of-the-art accuracy-efficiency trade-offs, we have refined the formulations and experimental analyses in the Introduction (including Contributions), Experiments (Sections 4.2 & 4.3), and Conclusion, with all revisions highlighted in blue in the updated manuscript.

Overall, we hope these responses resolve your concerns and clarify the technical contributions of our work, and we warmly welcome any further questions during the rebuttal discussion phase.

Thank you for the response. I still have a few concerns.

  1. Stage 1 visualization. Figure 6 does not clearly show the relevance scores and token budgets across all frames. It is hard to tell whether frames containing key evidence actually receive more tokens, especially for motion and temporal reasoning questions. In the video game examples, the token budgets appear quite similar across frames. Could the authors provide a clearer visualization of Stage 1?

  2. CPE effectiveness. CPE assumes that tokens with larger L2 norms contain more information. Is there any direct evidence or visualization supporting this assumption? Also, which types of questions suffer most from token pruning? In some visualizations, the retained tokens do not seem focused on query-relevant regions. Are these tokens providing useful context, or are they simply artifacts of the importance metric?

  3. Efficiency breakdown. Could the authors report the computational overhead of CPE, and QCE separately? This would help clarify where the additional inference time comes from.

  4. Stage 1 design choice. The paper argues that cosine similarity alone may repeatedly favor the same query concepts while missing complementary evidence. This motivates CPE and QCE in Stage 2. However, Stage 1 still relies only on mean cosine similarity for budget allocation. Why not incorporate visual informativeness and query coverage into frame-level scoring as well? Would this provide a more reliable budget allocation, especially for complex temporal questions?

Reviewer_gNvGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-free visual token pruning for efficient VideoLLMs. Existing methods primarily rely on visual cues, potentially overlooking query-critical evidence in sparsely sampled frames. Meanwhile, some query-aware methods incur additional computation through within-LLM attention or auxiliary models. To address these limitations, this paper proposes EviQ, a training-free pre-LLM framework that combines query-conditioned frame-level budget routing with intra-frame evidence selection. On LLaVA-OneVision-7B, EviQ preserves 99.9% of the original accuracy with 25% visual tokens, achieving a 1.59× inference speedup and an 8.7% GPU memory reduction.

Critical strengths
  1. The paper addresses a practical problem in VideoLLM inference. Its training-free design requires neither additional training nor auxiliary models, making it practical for deployment across different VideoLLMs.
  2. The proposed hierarchical framework is clearly formulated, combining frame-level budget allocation with intra-frame evidence selection.
  3. The experiments demonstrate that EviQ preserves nearly all the original accuracy while reducing inference latency and GPU memory consumption.
Critical weaknesses
  1. The novelty of the proposed visual-query relevance signal is overstated. Q-Frame [1] already explores cross-modal similarity for query-aware frame selection, while CRISP [2] uses a similar signal for pre-LLM pruning. Moreover, EviQ computes relevance using the entire formatted prompt, including answer choices and instruction tokens. This may introduce irrelevant information and interference from incorrect options.

  2. Adaptive token budget allocation has been widely explored in prior work. InfoMerge [3] and MeToM [4] allocate token budgets based on video content, while KVTP [5] and QuoTA [6] further incorporate query relevance. Vista-LLM [7] also combines query-guided budget allocation with fine-grained token selection. Together with the limited novelty of the relevance signal, EviQ appears to be an incremental combination of existing techniques rather than a substantial methodological advance.

  3. The role of query-agnostic context in CPE needs further justification. Under a limited token budget, CPE may also compete with query-critical evidence. Table 6 shows that CPE alone nearly matches full EviQ on MVBench (57.5 vs. 57.7) and MLVU (62.6 vs. 63.0), while QCE alone scores only 51.0 and 55.2, respectively. This raises questions about the actual contribution of query-conditioned selection. Furthermore, how does EviQ perform in multi-turn VideoQA?

  4. The visualizations provide limited support for the effectiveness of the relevance signal. In Figures 2, 5, and 6, the similarity scores appear rather diffuse, with some background regions receiving high relevance scores. This raises concerns about the reliability of embedding similarity in estimating query relevance.

  5. EviQ does not consistently outperform existing methods on long-video understanding, despite this being a central motivation. On VideoMME-Long, EviQ underperforms FastVID on LLaVA-Video-7B (51.0 vs. 52.9), VidCom² on Qwen3-VL-8B (53.1 vs. 56.2), and FlashVID on Qwen3-VL-32B (58.4 vs. 61.3). These results call into question the claimed advantages of query-conditioned pruning under aggressive temporal downsampling.

[1] Zhang, Shaojie, et al. "Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms." ICCV 2025.

[2] Li, Xu, et al. "CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference." arXiv 2026.

[3] Liu, Xinxin, et al. "InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models." arXiv 2026.

[4] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[5] Liu, Yudong, et al. "Keyframe-oriented vision token pruning: Enhancing efficiency of large vision language models on long-form video processing." ICCV 2025.

[6] Luo, Yongdong, et al. "Quota: Query-oriented token assignment via cot query decouple for long video comprehension." AAAI 2026.

[7] Li, Zhenyu, et al. "Vista-LLM: Decoupled Query-Guided Visual Token Pruning for Efficient Long-Video Large Language Models." ACL 2026.

Questions for authors

Plz refer to the weaknesses.

10 comments
AuthorsOpenReview ↗

Overview of Our Core Motivation and Design Rationale

We sincerely thank the reviewer for the constructive feedback. Before providing our point-by-point responses, we would like to briefly recap the core motivation and design rationale of EviQ to clarify potential misunderstandings and establish a clear context for the discussions below.

This work aims to preserve dense VideoLLM inference performance while keeping the pruning pipeline minimally complex. Because sparse frame sampling in VideoLLMs weakens inter-frame correlation and confines query-relevant evidence to only a few frames, diversity-based methods relying on inter-frame redundancy risk dropping critical cues. Conversely, existing query-aware approaches depend on auxiliary models or intra-LLM attention, which bottlenecks actual inference acceleration. EviQ resolves this dilemma before LLM inference by exploiting the synergy between visual context and query coverage through a lightweight two-stage optimization process, achieving state-of-the-art accuracy-efficiency trade-offs without heavy computational overhead. With this overarching goal in mind, we address each of your specific concerns below.

AuthorsOpenReview ↗

Point-by-Point Responses (Part 1/4)

Clarification on Cross-Modal Relevance Signals and Full Prompt Formulation (for Weakness 1)

  • Fundamental Distinctions from Q-Frame[1] and CRISP[2] in Sparse Video Pruning: We would clarify the distinction between prior uses of cross-modal similarity and our specific findings in sparsely sampled videos. First, Q-Frame relies on an external image-text matching model to perform coarse-grained frame selection and resolution adaptation from a candidate pool, which introduces auxiliary model overhead without addressing fine-grained intra-frame token redundancy. Second, CRISP focuses exclusively on static single-image pruning before LLM inference and leaves video temporal dynamics unexplored. As detailed in Section 3.2, rather than introducing external matching networks, EviQ directly leverages the native multimodal projector outputs and demonstrates that cosine similarity between projected visual tokens and query embeddings simultaneously captures both inter-frame and intra-frame query relevance under sparse video sampling. Therefore, we do not overstate the novelty of the cosine similarity operator itself, as our contribution lies in uncovering and validating this dual-level relevance property in sparsely sampled videos rather than claiming the mathematical metric as a new invention.

  • Empirical Justification for Using the Full Formatted Prompt over Filtered Queries: Regarding the concern that using the full formatted prompt with instructions and answer options may introduce distractions, we have systematically investigated query construction strategies in Appendix F.4 and paste the results below for convenience. Adopting the noun extraction strategy from CRISP to filter out instructions and non-noun tokens actually degrades the average accuracy across the four benchmarks from 59.0 to 58.6. A plausible reason is that aggressively filtering the query removes crucial verbs, attributes, and temporal connectives that are indispensable for localizing action-oriented and temporal evidence in videos. Furthermore, retaining candidate answer options in the prompt guides the pruner to preserve visual regions associated with all plausible hypotheses, allowing the downstream LLM to effectively verify true options and falsify distractors during reasoning.

MethodPruning RatioMVBenchLongVideoBenchMLVUVideoMMEAverage Performance
LLaVA-OV-7B/58.356.463.058.459.0
prompt noun-only25%57.156.662.658.258.6
EviQ(Ours)25%57.757.463.057.859.0

[1] Zhang, Shaojie, et al. "Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms." ICCV 2025.

[2] Li, Xu, et al. "CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference." arXiv 2026.

AuthorsOpenReview ↗

Point-by-Point Responses (Part 2/4)

Clarification on Holistic Novelty (for Weakness 2)

  • Comparison with Existing Pruning Methods on Accuracy and Efficiency: As summarized in the comparison table below, existing adaptive token pruning methods face distinct limitations in balancing accuracy and inference efficiency under sparse video sampling. Because VideoLLMs aggressively downsample long videos and weaken visual continuity between sampled frames, query-agnostic methods like InfoMerge[3] and MeToM[4] that allocate budgets based on video content risk spending the limited token budget on visually distinct or high-motion regions that are irrelevant to the question, while MeToM also relies on intra-LLM attention and bitstream residuals that restrict inference speed. Conversely, while query-aware approaches including KVTP[5], QuoTA[6], and Vista-LLM[7] incorporate query relevance for budget allocation or fine-grained selection, they all depend on auxiliary relevance scoring models, introducing extra computational overhead before the main LLM that limits actual wall-clock speedups. In contrast, EviQ resolves these accuracy and efficiency bottlenecks by operating strictly before the LLM without any auxiliary models, leveraging native projector features and zero-overhead priors to jointly preserve visual context and query-relevant evidence with minimal latency.

  • Unified Two-Stage Paradigm Beyond Incremental Combination: We disagree with characterizing EviQ as an incremental combination, as our work constitutes a unified and holistic framework rather than a piecemeal assembly of metrics. By narrowly focusing on the prior existence of adaptive budget routing across frames and query-based selection concepts, the reviewer overlooks our overarching contribution in realizing a two-stage paradigm in a lightweight, pre-LLM manner. Specifically, by reusing a single native projector cosine similarity matrix alongside a zero-overhead token L2L_2-norm prior, EviQ seamlessly bridges inter-frame budget routing and intra-frame submodular selection, achieving state-of-the-art accuracy-efficiency trade-offs.

MethodQuery-Based (Accuracy)Pre-LLM Compression Only (Efficiency)No Auxiliary Relevance Model (Efficiency)
InfoMerge×✓✓
MeToM [CVPR26]××✓
KVTP [ICCV25]✓✓×
QuoTA [AAAI26]✓✓×
Vista-LLM [ACL26]✓✓×
EviQ (Ours)✓✓✓

[3] Liu, Xinxin, et al. "InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models." arXiv 2026.

[4] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[5] Liu, Yudong, et al. "Keyframe-oriented vision token pruning: Enhancing efficiency of large vision language models on long-form video processing." ICCV 2025.

[6] Luo, Yongdong, et al. "Quota: Query-oriented token assignment via cot query decouple for long video comprehension." AAAI 2026.

[7] Li, Zhenyu, et al. "Vista-LLM: Decoupled Query-Guided Visual Token Pruning for Efficient Long-Video Large Language Models." ACL 2026.

Reviewer_RhoAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EviQ, a training-free, pre-LLM visual token pruning framework for VideoLLMs. Utilizing cosine similarity between projected visual tokens and embedded query tokens as a lightweight relevance signal, EviQ operates in two stages: (1) Frame-Level Query-Conditioned Budget Routing (QCBR), which distributes the global token budget across frames via standardized softmax routing based on mean visual-query token similarity; and (2) Intra-Frame Query-Conditioned Evidence Selection, which combines a visual token ℓ2\ell_2-norm prior (Context-Preserving Evidence, CPE) with max-pooled query semantic coverage (Query-Complete Evidence, QCE) into a monotone submodular objective solved via standard greedy optimization. The method is evaluated on LLaVA-OneVision-7B, LLaVA-Video-7B, and Qwen3-VL (8B/32B) across MVBench, LongVideoBench, MLVU, and VideoMME.

Critical strengths
  1. Practical Pre-LLM Deployability: Because token pruning is executed prior to LLM forwarding without relying on mid-LLM attention scores or external auxiliary models, EviQ is fully compatible with FlashAttention and introduces only 15.7ms overhead on LLaVA-OV-7B, achieving a 1.59×1.59\times speedup.
  2. Comprehensive Empirical Coverage: The method is evaluated across multiple VideoLLM backbones (7B to 32B), retention ratios (15%–35%), random seeds, and keyframe sampling strategies (Uniform, AKS, MDP3).
Critical weaknesses
  1. Limited Technical Novelty (Incremental Combination): The proposed framework largely assembles existing techniques: pre-LLM visual-text cosine similarity directly follows recent image-level pruning methods (e.g., CDPruner, CRISP); the CPE term directly adopts token ℓ2\ell_2-norm as an informativeness proxy from prior LLM pruning literature (Oyama et al., 2023; Sun et al., 2024).
  2. Marginal Empirical Contribution of Query-Conditioned Modules: Ablation studies significantly weaken the paper's central claim regarding hierarchical query-conditioned selection. In Table 13 (LLaVA-OV-7B, 25% retention), the query-agnostic CPE baseline (selecting tokens solely by ℓ2\ell_2-norm) already achieves an average score of 58.3 (98.9% of vanilla accuracy), and Uniform + CPE + QCE reaches 58.7 (99.5%) versus 59.0 (99.9%) for full EviQ—showing only a 0.3-point gain from QCBR. More strikingly, in Table 14 (Qwen3-VL-8B, 25% retention), removing frame-level routing (τ=0.0\tau=0.0) or removing intra-frame query coverage (α=0.0\alpha=0.0, CPE-only) both yield 62.4 (97.3%), trailing full EviQ (62.6, 97.7%) by merely 0.2 points. This indicates that nearly all retained accuracy comes from the simple ℓ2\ell_2-norm prior rather than the proposed query-conditioned routing or submodular coverage
  3. Missing Comparison with Relevant Recent Baseline (MeToM): While the paper includes several 2026 baselines such as ForestPrune (CVPR 2026), FlashVID (ICLR 2026), and MMG-Vid (AAAI 2026), it omits MeToM: Metadata-Guided Token Merging for Efficient Video LLMs, which also investigates training-free dynamic budget allocation and token compression for VideoLLMs. The authors should discuss MeToM and provide empirical accuracy/latency comparisons under the same evaluation protocol.
  4. Lack of Explicit Spatiotemporal Visual Deduplication: Because CPE is strictly modular (∑∥xiv,f∥2\sum \Vert{}x_i^{v,f}\Vert{}_2) and greedy selection runs independently within each frame, EviQ cannot explicitly penalize redundant visual tokens with large ℓ2\ell_2-norms across adjacent frames in static scenes.
Questions for authors

Please see Critical Weaknesses above.

5 comments
AuthorsOpenReview ↗

Overview of Our Core Motivation and Design Rationale

We sincerely thank the reviewer for the constructive feedback. Before providing our point-by-point responses, we would like to briefly recap the core motivation and design rationale of EviQ to clarify potential misunderstandings and establish a clear context for the discussions below.

This work aims to preserve dense VideoLLM inference performance while keeping the pruning pipeline minimally complex. Because sparse frame sampling in VideoLLMs weakens inter-frame correlation and confines query-relevant evidence to only a few frames, diversity-based methods relying on inter-frame redundancy risk dropping critical cues. Conversely, existing query-aware approaches depend on auxiliary models or intra-LLM attention, which bottlenecks actual inference acceleration. EviQ resolves this dilemma before LLM inference by exploiting the synergy between visual context and query coverage through a lightweight two-stage optimization process, achieving state-of-the-art accuracy-efficiency trade-offs without heavy computational overhead. With this overarching goal in mind, we address each of your specific concerns below.

AuthorsOpenReview ↗

Point-by-Point Responses (Part 1/3)

Clarification on Technical Novelty and the Incremental Combination Concern (for Weakness 1)

  • Unified Two-Stage Paradigm Beyond Incremental Combination: We disagree with characterizing EviQ as an incremental combination, as our work constitutes a unified and holistic framework rather than a piecemeal assembly of metrics. By narrowly focusing on the prior existence of cosine similarity and the token L2L_2 norm, the reviewer overlooks our overarching contribution in realizing a two-stage paradigm in a lightweight, pre-LLM manner. Specifically, by reusing a single native projector cosine similarity matrix alongside a zero-overhead token L2L_2-norm prior, EviQ seamlessly bridges inter-frame budget routing and intra-frame submodular selection, achieving state-of-the-art accuracy-efficiency trade-offs.

  • Domain-Specific Investigation and Verification of Proxy Metrics: Furthermore, the reviewer overlooks our rigorous domain-specific investigation into the suitability of these two proxies, as we did not blindly adopt them without verifying their effectiveness in the sparse video regime. For cosine similarity, while CDPruner[1] initially adopted it via pre-aligned CLIP contrastive embeddings and merely noted that projector outputs could serve as visual embeddings without empirical proof or video-level validation, and methods like CRISP[2] similarly lack deep investigation, we systematically compare representations in Figure 2&5 to visually demonstrate that projector-level cosine similarity simultaneously captures both inter-frame and intra-frame query relevance in sparsely sampled videos far more accurately than CLS and shallow-layer LLM attention. For the token L2L_2 norm in PCE, rather than taking its effectiveness for granted, we conducted extensive ablation studies in Table 14 (origin Table 13) and confirmed that it consistently outperforms alternative priors including channel variance and activation entropy. Adapting foundational metrics through rigorous verification to serve a unified mathematical framework represents a targeted empirical insight rather than a lack of novelty, enabling state-of-the-art accuracy-efficiency trade-offs.

[1]Zhang, Qizhe, et al. "Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs" NIPS 2025

[2]Li, Xu, et al. "CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference." ICME 2026.

AuthorsOpenReview ↗

Point-by-Point Responses (Part 2/3)

Clarification on Marginal Accuracy Gains (for Weakness 2)

  • Expected Ablation Behavior and Synergistic Design of QCBR, PCE, and QCE: The ablation results fully align with our design expectations and confirm the distinct role of each component. (i) QCBR is primarily introduced to reduce the algorithmic complexity of the selection pipeline. As shown in the table below, compared with global greedy selection in PCE+QCE, hierarchical decoupling via QCBR cuts the greedy selection time by 61.6% and simultaneously boosts the average accuracy across the four benchmarks from 58.5 to 59.0. (ii) PCE and QCE operate in a complementary manner. In isolation, PCE achieves high accuracy because retaining globally informative visual context suffices to answer many general scene-level questions, whereas QCE alone performs worse because focusing exclusively on query alignment discards the foundational scene structure required for coherent reasoning. However, QCE plays an indispensable role in preserving critical evidence located in subtle local regions or brief temporal intervals that query-agnostic PCE inevitably misses. (iii) Ultimately, QCBR, PCE, and QCE function as a unified whole that aims to preserve dense VideoLLM inference performance while keeping the pruning pipeline minimally complex.
MethodComplexityOverhead Latency(ms)Total Latency(ms)Accuracy
LLaVA-OV-7B//462.459.0
QCBR+PCE+QCEO(KMN)O(KMN)15.7291.159.0
Uniform+PCE+QCEO(KMN)O(KMN)15.5290.958.7
PCE+QCEO(KFMN)O(KFMN)25.5300.958.5

Remark: FF is the number of sampled frames, NN is the number of candidate visual tokens per frame, MM is the number of query tokens used for coverage scoring, and KK is the total visual token budget.

  • Evaluating Holistic Inference Acceleration Beyond Marginal Accuracy Gains: Furthermore, evaluating a token pruning framework solely on accuracy overlooks its primary goal of inference acceleration. While EviQ achieves competitive or superior accuracy against state-of-the-art baselines, its key advantage lies in substantially reducing computational overhead. On LLaVA-OneVision-7B, although our accuracy gain over FlashVid[3] is 0.1, EviQ slashes the pruning computation time by 38% and reduces overall end-to-end latency by 9%. Assessing performance purely through accuracy metrics without accounting for these significant wall-clock speedups presents an incomplete evaluation of our method.

[3]Ziyang Fan, et al. "FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging" ICLR 2026

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the Area Chair and all reviewers for their constructive feedback and dedicated time.

The core objective of EviQ is to preserve dense VideoLLM inference accuracy while keeping the computational complexity of the pruning pipeline minimal, thereby achieving an optimal accuracy-efficiency trade-off in sparsely sampled video understanding. To the best of our knowledge, we are the first to investigate how lightweight pruning can preserve as much of the original model performance as possible, providing meaningful guidance for future work on understanding sparsely sampled videos. Below we provide a focused and in-depth clarification on the three common core concerns raised by the reviewers regarding technical novelty, the accuracy-efficiency trade-off, and component ablations, followed by concise summaries of our responses to the specific questions of the reviewers.

AuthorsOpenReview ↗

Summary of Rebuttal (Part 1/2)

1. Technical Novelty [Reviewer RhoA W1, Reviewer gNvG W1 & W2, Reviewer XT4x W1]

Addressing the Accuracy and Efficiency Dilemma of Existing Methods. Existing visual token pruning methods for VideoLLMs either rely on inter-frame correlation and visual diversity where the retained visual tokens do not necessarily match the user query under sparse frame sampling, or utilize intra-LLM attention and external scoring models that introduce excessive pruning overhead. EviQ resolves this dilemma entirely before LLM prefilling by exploiting the synergy between visual context prior via PCE and query coverage estimation via QCE alongside query-conditioned frame budget routing via QCBR, avoiding the issues above from both accuracy and efficiency perspectives as summarized in the comparison table below.

MethodQuery-Based (Accuracy)Pre-LLM Compression Only (Efficiency)No Auxiliary Model (Efficiency)
InfoMerge×\times✓\checkmark✓\checkmark
KVTP[ICCV25]✓\checkmark✓\checkmark×\times
CDPruner[NIPS25]✓\checkmark✓\checkmark×\times
QuoTA[AAAI26]✓\checkmark✓\checkmark×\times
EarlyTom[CVPR26]×\times✓\checkmark✓\checkmark
MeToM[CVPR26]×\times×\times✓\checkmark
CRISP[ICME26]✓\checkmark✓\checkmark×\times
Vista-LLM[ACL26]✓\checkmark✓\checkmark×\times
EviQ(Ours)✓\checkmark✓\checkmark✓\checkmark

Clarification on Lightweight Proxies and Unified Framework Design. The concerns from the reviewers regarding novelty mainly stem from the similarity of individual modules to certain prior works. However, we not only articulate the strict efficiency rationale for adopting these lightweight pre-LLM designs in VideoLLMs, but also verify their effectiveness under sparse video sampling through visualization analyses in Figures 2 and 5 and ablation experiments in Table 14. Specifically, we demonstrate that native projector cosine similarity simultaneously captures both inter-frame and intra-frame query relevance in sparsely sampled videos far more accurately than static CLS attention and costly shallow-layer LLM attention, while the zero-overhead token L2L_2 norm consistently outperforms alternative priors including channel variance and activation entropy. Focusing narrowly on the similarity of individual modules to prior methods and viewing our approach as an incremental combination overlooks how these modules function as an organic whole to achieve a superior accuracy-efficiency trade-off.

2. Accuracy-Efficiency Trade-off [Reviewer RhoA W2, Reviewer 6PSo W2, Reviewer XT4x W2]

The reviewers focused primarily on marginal accuracy gains and viewed our performance improvements as limited, overlooking that our method is designed to achieve a superior accuracy-efficiency trade-off as illustrated in Figure 1c. Compared with recent high-performing baselines such as FlashVID, EviQ not only achieves consistently higher accuracy across multi-seed evaluations in Appendix F.1, but also reduces pruning overhead latency by over 61.4% and improves end-to-end inference speed by over 8%.

AuthorsOpenReview ↗

Summary of Rebuttal (Part 2/2)

3. Component Ablations[Reviewer RhoA W2, Reviewer gNvG W3, Reviewer XT4x W2]

As noted above, our method is not an incremental combination, so observing varying accuracy levels when evaluating individual modules in isolation is completely normal and aligns with our expectations.

In isolation, (i) PCE achieves strong performance because retaining globally informative visual tokens preserves the foundational scene structure sufficient to answer many general scene-level questions. Conversely, QCE alone performs worse because focusing exclusively on query alignment discards the underlying scene context required for coherent reasoning. (ii) However, QCE serves to capture critical query-specific evidence in subtle local regions or brief temporal intervals that query-agnostic PCE inevitably misses, which explains why combining PCE and QCE yields superior performance. (iii) Furthermore, QCBR is specifically proposed to reduce the algorithmic complexity of global greedy submodular optimization by decoupling selection into independent intra-frame steps. Experiments confirm that this module not only reduces pruning overhead latency by 39.2%, but also achieves slightly better accuracy than global greedy selection.

4. Other Concerns

Beyond addressing the common concerns above, we briefly summarize our responses and experimental findings for the remaining reviewer-specific questions below.

  • Comparisons with MeToM and EarlyTom.[Reviewer RhoA W3, Reviewer 6PSo W1] Under unified evaluation settings across multiple backbones, EviQ consistently outperforms both CVPR 2026 baselines in average accuracy while substantially reducing pruning overhead and total end-to-end latency.
  • Cross-Frame High-Norm Token Deduplication.[Reviewer RhoA W4] Explicitly deduplicating high L2L_2 norm tokens across adjacent frames degrades average accuracy across all benchmarks, confirming that recurring high-norm tokens act as essential temporal anchors under sparse sampling.
  • Full Prompt Formulation and Multi-Turn VideoQA.[Reviewer gNvG W1 and W3] Full prompts outperform noun-only filtering by preserving verbs, temporal cues, and candidate options, while reusing first-turn pruned visual KV caches in multi-turn VideoQA incurs only a 0.5% relative accuracy drop, highlighting the strong generalization of our method in multi-turn VideoQA.
  • Spatially Diffuse Similarity Visualizations.[Reviewer gNvG W4] Vision Transformer self-attention aggregates foreground semantics into surrounding patches without pixel-level supervision, making projector cosine similarity appear spatially diffuse while still highlighting query-relevant regions far more reliably than static CLS attention.
  • Long-Form Video Performance and VideoMME Subsets.[Reviewer gNvG W5, Reviewer 6PSo W4] EviQ ranks first or second across all backbones on LongVideoBench and MLVU, whereas methods based on inter-frame correlation or diversity hold more advantages on the VideoMME Short and Medium subsets, leading to slightly higher overall performance for those methods on VideoMME.
  • Iso-Token Frame Scaling Analysis.[Reviewer 6PSo W3] Scaling from fewer uncompressed frames to denser frame sampling with EviQ under an identical LLM token budget steadily improves reasoning accuracy, validating the effectiveness of our method.

5.Concluding Remarks

To better highlight our novelty in realizing a lightweight, decoupled pre-LLM two-stage pruning paradigm under sparse video sampling and to emphasize our state-of-the-art accuracy-efficiency trade-offs, we have refined the formulations and experimental analyses in the Introduction (including Contributions), Experiments (Sections 4.2 & 4.3), and Conclusion. Additionally, we have incorporated comparisons with MeToM and EarlyTom in Tables 1, 2, 4, and 11 ((origin Table 10), with all revisions highlighted in blue in the updated manuscript.

Through these comprehensive experiments and targeted clarifications, we think we have thoroughly addressed every concern raised by the reviewers, and we sincerely hope the Area Chair and reviewers will recognize the unified methodological contribution, rigorous empirical validation, and superior accuracy-efficiency trade-offs of our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.