- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed a training-free pruning method for video input. This work is motivated by previous work FastV’s limitation, where the last text token such as <|im_start|> that attends to vision tokens based on overall visual saliency rather than query relevance. This work overcomes this issue by aggregating the attention weights for all the query tokens. In addition, this work also measures the deviation of each visual token within one frame, where bigger deviation means more distinctive information. Combining these two scores, this work evaluates their method on two benchmarks.
Critical strengths
- The paper identifies a meaningful limitation of last-token attention-based pruning. Figure 1 demonstrates that FastV produces highly similar attention distributions across different queries, whereas aggregating attention from query tokens generates more query-dependent importance scores. The observation that random selection can outperform FastV on certain tasks also provides interesting evidence about the limitations of existing pruning signals.
- QD-Prune does not require additional learnable parameters or auxiliary models. The combination of query-driven attention, hidden-state diversity, and attention entropy is straightforward and easy to interpret.
Critical weaknesses
- The improvement is not consistent, especially on the NExT-QA benchmark, where the performance on Temporal and Descriptive questions is identical or worse then random pruning. Even though the authors value the Causal question the most, but the performance gap raises concerns that the hypothesis of this work may not be solid enough.
- In Table4, with only the query-relevance score as pruning guidance, the performance is better than the full version, in other words, the temporal score is having a negative impact on performance.
- This work only evaluates on two benchmarks, and one shows inconsistent improvement, which lacks empirical evidence to prove the method’s effectiveness.
Questions for authors
See weakness.