- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EviQ, a training-free, query-conditioned visual token pruning framework for efficient VideoLLM inference. The method aims to reduce the computational and memory overhead of processing video tokens while preserving information relevant to the input query. EviQ adopts a two-stage hierarchical approach. First, it dynamically allocates token budgets across video frames based on cosine similarity between visual and query embeddings. Second, it selects tokens within each frame by combining visual informativeness, measured through token L2 norms, with query-semantic coverage formulated as a submodular optimization problem. A greedy selection algorithm is employed with a theoretical approximation guarantee.
Critical strengths
- Paper is easy to understand.
- EviQ is training-free and performs token pruning before LLM inference, making it easy to integrate with existing VideoLLMs. It achieves a favorable accuracy-efficiency trade-off, retaining nearly the original accuracy with only 25% of visual tokens.
Critical weaknesses
-
Limited novelty. The proposed framework primarily combines existing techniques, including cosine similarity, token L2 norms, softmax-based budget allocation, and greedy submodular optimization. The method lacks a fundamentally new mechanism for video understanding, particularly for modeling temporal dependencies and cross-frame interactions.
-
Marginal improvements and insufficient justification of key components. The improvements are relatively small.
Questions for authors
N/A
2 comments
Concluding Remarks
To better highlight our novelty in realizing a lightweight, decoupled pre-LLM two-stage pruning paradigm under sparse video sampling and to emphasize our state-of-the-art accuracy-efficiency trade-offs, we have refined the formulations and experimental analyses in the Introduction (including Contributions), Experiments (Sections 4.2 & 4.3), and Conclusion, with all revisions highlighted in blue in the updated manuscript.
Overall, we hope these responses resolve your concerns and clarify the technical contributions of our work, and we warmly welcome any further questions during the rebuttal discussion phase.
Point-by-Point Responses
Clarification on Methodological Novelty and Our Paradigm for Sparse Video Understanding (for Weakness 1)
Core Motivation and Efficiency Advantage of EviQ: This work aims to preserve dense VideoLLM inference performance while keeping the pruning pipeline minimally complex. Because sparse frame sampling in VideoLLMs weakens inter-frame correlation and confines query-relevant evidence to only a few frames, diversity-based methods relying on inter-frame redundancy risk dropping critical cues. Conversely, existing query-aware approaches depend on auxiliary models or intra-LLM attention, which bottlenecks actual inference acceleration. EviQ resolves this dilemma by exploiting the synergy between visual context and query coverage, achieving state-of-the-art accuracy-efficiency trade-offs without heavy computational overhead.
We disagree that utilizing cosine similarity and token L2 norms lacks novelty for video understanding. Taking cosine similarity as an example, while CDPruner[1] initially adopted it via pre-aligned CLIP contrastive embeddings, that work merely noted that projector outputs could serve as visual embeddings for dedicated visual encoders without providing empirical proof or video-level validation, and methods like CRISP[2] similarly lack deep investigation. In contrast, we systematically compare representations in the video domain and visually demonstrate that cosine similarity captures both inter-frame and intra-frame query relevance under sparse sampling. Furthermore, integrating token L2 norms, softmax-based budget allocation, and greedy submodular optimization is the result of rigorous design validated by extensive ablations. Adapting foundational metrics from other domains to address unique challenges in sparse video pruning represents a targeted empirical insight rather than a lack of novelty.
A New Two-Stage Paradigm for Sparse Video Understanding: Far from lacking a fundamentally new mechanism for video understanding, EviQ introduces a paradigm shift tailored to the sparse nature of VideoLLM inputs. Because VideoLLMs process downsampled videos with low inter-frame correlation, explicitly modeling temporal dependencies and cross-frame interactions becomes inherently less reliable. We therefore reformulate sparse video pruning as a global optimization problem solved in lightweight two decoupled stages, combining inter-frame budget allocation to capture global query relevance with intra-frame submodular optimization to jointly preserve visual context and query coverage. Thus, the claim that our method lacks a new mechanism does not hold.
[1]Zhang, Qizhe, et al. "Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs" NIPS 2025
[2]Li, Xu, et al. "CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference." ICME 2026.
Clarification on Accuracy-Efficiency Trade-offs and Component Ablations (for Weakness 2)
Evaluating Holistic Efficiency-Accuracy Trade-offs Beyond Marginal Accuracy Gains: We disagree with the characterization of our improvements as marginal, as evaluating a token pruning framework solely on accuracy overlooks its primary goal of inference acceleration. While EviQ achieves competitive or superior accuracy against state-of-the-art baselines, its key advantage lies in substantially reducing computational overhead. On LLaVA-OneVision-7B, although our accuracy gain over FlashVid[3] is 0.1, EviQ slashes the pruning computation time by 38% and reduces overall end-to-end latency by 9%. Assessing performance purely through accuracy metrics without accounting for these significant speedups presents an incomplete evaluation of our method.
Comprehensive Ablations and Robustness Verification of Key Components: Regarding the concern over insufficient justification of key components, we have already provided rigorous empirical validation throughout the manuscript. Specifically, we systematically ablated the individual contributions of QCBR, PCE, and QCE, and compared them against various alternative formulations in Table 6 and Table 14 (origin Table 13). Moreover, we demonstrated the stability of our method through extensive hyperparameter robustness analyses in Figure 4 and Table 15 (origin Table 14). These results collectively confirm both the necessity of our core components and the robustness of EviQ, and we welcome further clarification from the reviewer on any specific aspect that may require additional analysis.
[3]Ziyang Fan, et al. "FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging" ICLR 2026