acceptodds
Under review as a conference paper at ICLR 2027

Query-Answer-Aligned Frame Selection for Long Video Understanding

Abstract

Multimodal large language models (MLLMs) process multimodal inputs by converting text, images, and videos into token sequences that are then processed by a backbone language model. However, the limited visual token budget of these models makes long-form video understanding challenging. For instance, a 5-minute video at 24 frames per second (fps) contains 7,200 frames. However, practical MLLM inference typically uses only a subset of these frames, usually ranging from 8 to 64 frames. These frames are usually sampled uniformly, regardless of their relevance to the question. Recently, training-free, model-agnostic frame-selection methods have been proposed to improve video question answering. In this work, we propose two training-free, model-agnostic frame-selection methods: QV and QAV. QV scores frames using question-frame cosine similarity. QAV additionally incorporates multiple-choice answer options and scores frames by their maximum cosine similarity across the resulting question-option pairs. Both methods first construct a candidate pool by sampling video frames at a fixed rate. They then select the highest-scoring frames, up to a predefined frame budget, and restore their chronological order before passing them to the downstream MLLM. We evaluate QV and QAV on MLVU, Video-MME, and LongVideoBench using three MLLMs: Qwen2-VL-7B, LLaVA-Video-7B, and InternVL3.5-8B. Our results show that QAV improves over uniform sampling across the evaluated model-benchmark settings and achieves competitive performance against the evaluated training-free frame-selection baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-video understanding, video question answering, multimodal large language models, training-free frame selection, answer-aware retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5GbrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript presents a work on long-video understanding under a fixed MLLM frame budget. It introduces two training-free, model-agnostic, plug-and-play frame-selection methods. The claimed contributions include a cosine-similarity-based query–video (QV) frame-scoring method that operates on token vectors from the pre-trained encoders, and a query–answer–video (QAV) aligned frame-selection method that incorporates the multiple-choice answer options as inference-time cues through max-over-options scoring. Experiments conducted on MLVU, Video-MME, and LongVideoBench validate the proposed methods.

Critical strengths

The paper is well presented and easy to follow. Experiments conducted on MLVU, Video-MME, and LongVideoBench validate the proposed methods. Different duration groups have been tested.

Critical weaknesses

The novelty is limited, and the proposed methods are incremental over existing query-aware methods. QV largely follows a conventional retrieval-based frame selection approach using question-frame similarity. Meanwhile, QAV relies on predefined answer options, limiting its applicability to open-ended video question answering. The practical generality of the proposed methods remain unclear. On the experiments side, only a 16-frame setting is considered.

Questions for authors

The proposed methods are incremental over existing query-aware methods. Please better state the novelty of the proposed method compared to the existing works.

The improvements compared to existing methods on the three datasets are very limited, with only a fixed 16-frame setting. Please better explain the reasons of the improvements? It would be interesting to know if the improvements could be consistent to different settings.

Reviewer_1tGEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes two training-free, model-agnostic video frame selection strategies by ranking based on similarity of the video frame to the question text embedding (QV) and additionally with each multiple-choice answer option (QAV). After subsampling the video to 1 FPS, the top-k highest scoring frames are greedily picked, maintaining chronological order. QAV wins most settings, but not uniformly.

Critical strengths

All the results on the three benchmarks (MLVU, Video-MME, LongVideoBench), across three MLLMs (Qwen2-VL-7B, LLaVA-Video-7B, InternVL3.5-8B), against uniform sampling and four training-free baselines are openly shown and discussed in the main paper, even though several baselines beat it in specific duration/category slices.

Critical weaknesses
  1. The gains of QV and QAV in some duration/category slices across different benchmarks are non-uniform. Although the results are shown openly, the reason behind the failure cases for particular benchmarks and categories are not well-discussed, which could be helpful for the readers to understand the gain of this method in different new settings.
  2. Q-Frame has Gumbel-noise stochastic sampling while the proposed method has deterministic top-k sampling. An ablation between inner product ranking vs cosine ranking, keeping everything else same, is needed.

Minor weakness: Figure quality could be improved.

Questions for authors
  1. Noting the inconsistent gains of QV and QAV in some duration/category slices across different benchmarks, can the authors design an ablation experiment to prove whether swapping dot-product-ranking (Q-Frame) for cosine-ranking (QV/QAV) is itself the source of improvement, with everything else held fixed?

  2. Suggestion: Could an additional evaluation metric besides accuracy (grounding, evidence sufficiency, information-extraction quality) be added to see if QV/QAV consistently outperforms the other baselines in any of these metrics?

Reviewer_Lu6tOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes two training-free frame selectors. QV ranks frames by LongCLIP cosine similarity with the question. QAV concatenates the question with each answer option and scores each frame by its maximum similarity across these pairs. Evaluated on MLVU, Video-MME, and LongVideoBench, QAV achieves the best overall accuracy in most settings.

Critical strengths
  1. The method is very simple, training-free and plug-and-play, so it is easy to reproduce.
  2. The gains are consistent: across 15 overall settings (3 MLLMs × 3 benchmarks), QAV outperforms uniform sampling in all of them and achieves the best result in 13. This suggests the approach is robust across models and datasets.
Critical weaknesses
  1. Limited novelty and missing comparison with BOLT. QV (CLIP question-frame cosine similarity followed by top-K) is explicitly described as a "commonly used" baseline in BOLT (CVPR 2025), which is not compared. QAV's gain comes from appending options to the query, yet Table 7 shows this trick helps AKS (+2.62), FOCUS (+2.25) and Q-Frame (+1.52) just as much, and the authors note it is "not exclusive to QAV."
  2. Top-K selects redundant frames and is weakest on long videos. In Fig. 5, 13 of the 16 selected frames fall within indices 134–152, i.e., near-duplicates, a redundancy BOLT has already identified in top-K selection. In long video settings, the method is weakest: on Video-MME Long without subtitles, InternVL3.5 + QAV gets 48.11, below uniform (49.11) and FOCUS (52.00).
  3. Narrow applicability and unsupported efficiency claims. QAV requires multiple-choice options and cannot handle open-ended QA, which is the more common real-world setting. Also, although the Intro is motivated by computational cost, no selection runtime is reported.
Questions for authors

See weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.