acceptodds
Withdrawn from ICLR 2027

QD-Prune: Query-Driven Vision Token Pruning for Efficient Video-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video-language models encode videos into hundreds of vision tokens, which dominate the input sequence and drive up inference cost. Training-free pruning methods such as FastV remove tokens with low attention scores, but we find that their pruning signal is query-agnostic: it is taken from the last text token and stays nearly identical across different questions about the same video (cosine similarity 0.94), so query-critical tokens are removed and causal and temporal reasoning degrade. We propose QD-Prune, a training-free method that scores vision tokens by query-to-vision attention (M1) together with temporal diversity (M2), and selects the pruning layer by attention entropy minimization (M3). It requires no trainable parameters and reuses attention weights already computed during the standard prefill pass. At a 20% keep ratio on NExT-QA with Qwen2.5-VL-7B, QD-Prune reaches 64.3% overall accuracy, outperforming FastV by +5.3% (McNemar p=0.030), with the largest margin on causal reasoning (+7.0%). On MVBench it reaches 43.8%, +7.7% over FastV (p<0.001), matching the full-token baseline (44.1%, p=0.817).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video-language models,Token pruning,Query relevance,Attention entropy

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nVYQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a training-free pruning method for video input. This work is motivated by previous work FastV’s limitation, where the last text token such as <|im_start|> that attends to vision tokens based on overall visual saliency rather than query relevance. This work overcomes this issue by aggregating the attention weights for all the query tokens. In addition, this work also measures the deviation of each visual token within one frame, where bigger deviation means more distinctive information. Combining these two scores, this work evaluates their method on two benchmarks.

Critical strengths
  1. The paper identifies a meaningful limitation of last-token attention-based pruning. Figure 1 demonstrates that FastV produces highly similar attention distributions across different queries, whereas aggregating attention from query tokens generates more query-dependent importance scores. The observation that random selection can outperform FastV on certain tasks also provides interesting evidence about the limitations of existing pruning signals.
  2. QD-Prune does not require additional learnable parameters or auxiliary models. The combination of query-driven attention, hidden-state diversity, and attention entropy is straightforward and easy to interpret.
Critical weaknesses
  1. The improvement is not consistent, especially on the NExT-QA benchmark, where the performance on Temporal and Descriptive questions is identical or worse then random pruning. Even though the authors value the Causal question the most, but the performance gap raises concerns that the hypothesis of this work may not be solid enough.
  2. In Table4, with only the query-relevance score as pruning guidance, the performance is better than the full version, in other words, the temporal score is having a negative impact on performance.
  3. This work only evaluates on two benchmarks, and one shows inconsistent improvement, which lacks empirical evidence to prove the method’s effectiveness.
Questions for authors

See weakness.

Reviewer_BGw5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes QD-Prune, a training-free visual token pruning method for video-language models. It combines query-to-vision attention with within-frame feature distinctiveness and selects the scoring layer based on attention entropy. The retained tokens are then used to reconstruct a shorter input sequence for generation. Experiments on NExT-QA and MVBench using Qwen2.5-VL report improvements over FastV at a 20% token retention ratio.

Critical strengths

The motivation is reasonable: visual token selection should consider the user's query rather than relying entirely on generic attention scores. The method is simple and requires no additional training. I also appreciate the inclusion of random and uniform sampling baselines, which provide useful references for evaluating attention-based pruning.

Critical weaknesses

1. Serious inconsistencies in the experimental results.

I find it difficult to trust the main conclusions given the discrepancies across the reported results. For example, Table 2 and Figure 3 report different accuracies for SparseVLM (66.0% vs. 64.7%) and the full model (61.0% vs. 60.0%), despite specifying the same setting. QD-Prune's Causal accuracy also differs between Tables 2 and 4 (69.0% vs. 68.0%). Table 5 further reports 70.0% at r=0.2, compared with 64.3% in Table 2, without clearly explaining the evaluation setting or metric. The Overall accuracy in Table 2 also appears to be a macro-average rather than the standard sample-level accuracy. These inconsistencies directly affect the reported improvements and statistical significance. I consider this a major concern for the reliability of the experimental evidence.

2. Limited novelty and insufficient evidence of advantages over existing methods.

Query-aware pruning is already a well-established direction. SparseVLM (ICML 2025) uses text-to-vision attention for training-free token selection, while PruneVid (Findings of ACL 2025) already incorporates query relevance into video token pruning. More recent works, including Vista-LLM (ACL 2026), TransPrune (CVPR 2026), and HAWK (CVPR 2026), further explore query/instruction-guided token selection. The paper does not sufficiently distinguish its contribution from these methods, and its experimental baselines largely overlook recent developments.

Even with the reported results, QD-Prune does not demonstrate a clear advantage. On NExT-QA, it underperforms SparseVLM (64.3% vs. 66.0%) and only slightly exceeds uniform sampling (63.7%). The ablation also shows that removing M2 improves overall accuracy to 64.7%, while the independent benefit of M3 is not established.

The evaluation is limited to one model family and two older video QA benchmarks, without broader evaluations such as Video-MME, MLVU, or LongVideoBench. Moreover, four MVBench tasks were not evaluated due to missing videos. Finally, despite the reported FLOPs reduction, the actual latency (0.371 s vs. 0.364 s) and memory usage (16.51 GB vs. 15.73 GB) are both higher than the unpruned model. Thus, the practical efficiency benefit remains unclear.

Questions for authors
  1. Could the authors reconcile the conflicting results across Tables 2, 4, and 5 and Figure 3? Please clarify the evaluation subsets, accuracy aggregation, and sample counts underlying the reported significance tests.

  2. What is the specific technical contribution beyond existing query-aware methods such as TransPrune, HAWK, and Vista-LLM? Why are more recent baselines not included, and what evidence establishes the necessity of M2 and M3?

  3. Which temporal diversity formulation was actually implemented, given the discrepancy between Figure 2 and Eq. (5)? Does the FLOPs calculation include the initial scoring pass and subsequent generation? A complete end-to-end efficiency breakdown would be helpful.

  4. Could the authors provide results on additional models and more comprehensive benchmarks?

Reviewer_zSmGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QD-Prune, a training-free visual token pruning method for video-language models. It combines attention aggregated from query tokens (M1) with within-frame feature distinctiveness (M2), and selects a scoring layer using last-token attention entropy (M3). Experiments on NExT-QA and MVBench demonstrate improvements over FastV, particularly on causal questions, accompanied by module ablations and analyses of attention patterns and temporal coverage.

Critical strengths
  1. Useful diagnosis and clear methodological presentation. The paper illustrates insufficient query sensitivity in last-token-based pruning by comparing attention patterns across questions about the same video. This motivates query-conditioned selection, particularly for causal reasoning, where different questions may require evidence from different moments. The method is training-free and straightforward, with clearly explained components and visualizations that make its intended behavior easy to understand.
  2. Informative evaluation beyond aggregate accuracy. Random and uniform baselines, module ablations, layer-wise analyses, and paired significance tests provide useful controls. Reporting causal and temporal accuracy separately exposes meaningful trade-offs and helps readers assess whether improvements reflect better evidence selection rather than simply outperforming one pruning heuristic. The observation that simple sampling strategies can compete with attention-based pruning is also informative for evaluating token-selection methods.
Critical weaknesses
  1. The methodological advance and alignment between the proposed signals and their objectives are insufficiently established. M1 averages query-to-vision attention and ranks tokens independently, without explicitly accounting for redundant selections or complementary evidence across frames. Its advantage over existing query-aware pruning remains unclear; SparseVLM achieves higher overall accuracy on NExT-QA. M2 measures within-frame feature distinctiveness rather than cross-frame diversity, so it does not directly prevent temporal collapse. Removing M2 actually improves overall and temporal accuracy in Table 4. M3 likewise minimizes last-token attention entropy without establishing why attention concentration identifies the best layer for preserving query-relevant evidence. Together, these components form an intuitive heuristic, but their specific technical value requires stronger justification.
  2. The execution strategy introduces unresolved scoring–execution mismatch, while the experiments do not establish practical efficiency. Importance is estimated from a full-sequence pass, but generation restarts from pruned layer-0 embeddings. Token importance measured with complete context may change when discarded tokens are absent, and this mismatch is insufficiently examined. The scoring pass also imposes overhead: despite the reported FLOPs reduction, Table 6 shows increased latency (0.364 to 0.371 seconds) and memory (15.73 to 16.51 GB). Complete computational accounting is therefore necessary. Additionally, inconsistent NExT-QA results across Tables 2, 4, and 5 require reconciliation. Assigning zero accuracy to unavailable MVBench tasks also produces a nonstandard aggregate, even if applied consistently across methods.
Questions for authors
  1. What exactly constitutes the query-token set in M1? How sensitive are the results to including instructions and answer options rather than using only the question?
  2. How does within-frame distinctiveness in M2 preserve temporal evidence beyond identifying feature outliers? How do the authors explain the improved overall and temporal accuracy when M2 is removed?
  3. Is M3’s layer selected once per model or adaptively per input? What evidence shows that lower last-token attention entropy predicts better pruning performance on held-out data?
  4. Does the efficiency accounting include the complete scoring pass and subsequent pruned generation? Please clarify the attention backend, TTFT, and the implications of scoring with full context before restarting from pruned embeddings.
  5. Can the authors reconcile Tables 2, 4, and 5 and report MVBench results on the common available subset, explaining how unavailable examples were handled in significance tests?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.