- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SAVE, a training-free framework for selecting a limited number of frames for long-video question answering by allocating the visual budget according to question-conditioned evidence demand. SAVE first introduces Q-SelfYes, which reuses the same frozen VLM employed for final question answering to estimate the relevance of candidate frames from grouped Yes/No logits, avoiding the need for a separate relevance model. The resulting frame-level relevance scores are normalized into a temporal semantic-demand distribution, and frame selection is formulated as a demand-weighted temporal quantization problem that jointly accounts for semantic relevance and temporal coverage without requiring an explicit trade-off coefficient. By exploiting the one-dimensional temporal ordering of video frames, the proposed allocation objective can be solved exactly using dynamic programming and is further interpreted through a free-mass optimal transport perspective. Experiments on Video-MME, LongVideoBench, and MLVU using four different VLM backbones show that SAVE consistently improves benchmark-level average performance over the evaluated training-free frame-selection baselines under the same final frame budget. Additional ablations suggest that both answerer-aligned relevance estimation and the proposed OT-based temporal allocation contribute to the improvements, with the allocation mechanism providing a larger gain over simpler relevance-based selection strategies.
Critical strengths
-
The paper is well written and clearly organized, making the motivation, methodology, and experimental results easy to follow.
-
The proposed method is simple, intuitive, and effective, providing a way to balance question-conditioned semantic relevance with temporal coverage when selecting frames under a limited visual budget.
-
The paper provides extensive ablation studies and analyses covering the relevance scorer, allocation strategy, frame budget, model scale, candidate-frame density, and computational cost.
Critical weaknesses
-
The semantic-demand formulation relies on specific design choices, including Softmax normalization and a squared-distance allocation objective, but these choices appear somewhat heuristic. A broader analysis of alternative demand transformations and allocation objectives, e.g., ablations on the Softmax temperature , different normalization schemes, or alternatives to the squared-distance objective, would strengthen the justification for the proposed formulation.
-
Although the OT interpretation is mathematically reasonable, it is not entirely clear whether framing the allocation problem through OT is necessary for obtaining the reported benefits. Simpler formulations, such as weighted temporal allocation or clustering-based approaches, may potentially exhibit similar behavior. While Table 3 compares against several existing selection baselines, comparisons with such simpler allocation alternatives would help clarify the practical benefit of the proposed OT formulation beyond its conceptual interpretation.
-
According to the authors' analysis, Q-SelfYes provides only a modest improvement over strong external relevance scorers, making its empirical advantage relatively limited. At the same time, repeatedly invoking the full answer VLM to score candidate frames can introduce substantial computational overhead. This concern is further highlighted by prior work [1], which proposes a related strategy that distills a VLM's Yes/No relevance judgments into a smaller external scorer, potentially retaining VLM-aligned relevance estimation while reducing inference cost. A more direct comparison or discussion of this trade-off would better establish the practical advantage of Q-SelfYes.
[1] M-LLM Based Video Frame Selection for Efficient Video Understanding, CVPR 2025.
Questions for authors
N/A
2 comments
Follow-up on your review
Thank you again for your suggestions on the demand formulation, simpler allocation methods, and scoring efficiency. As the discussion period draws to a close, we’d appreciate hearing whether our response and additional experiments addressed your concerns. If any questions remain, we’d be happy to clarify.
Response to Reviewer eiPX
W1. Demand transformations and allocation objectives. We evaluated alternative Softmax temperatures, demand normalization schemes, and an absolute-distance objective. The sigmoid-based normalization ablation is already reported in Table 3 of the original paper.
Under the assumption of an absolutely continuous evidence-demand distribution, quadratic cost admits an optimal transport map given by the gradient of a convex potential, as established by Brenier's theorem. Our ablations further support the squared-distance choice empirically.
W2. Comparison with simpler allocation alternatives. We compared SAVE with weighted temporal quantiles (WTQ) and weighted temporal K-means (WTKM).
SAVE jointly accounts for demand weights and temporal distances, and finds a globally optimal allocation under its objective rather than relying on quantile placement or iterative clustering. It also achieves higher accuracy than both alternatives.
W3. Q-SelfYes and scoring efficiency. One of our motivations is to let the answering VLM itself provide frame-selection signals rather than rely on an external scorer. To improve efficiency, we developed SAVE-Flash, which retains Q-SelfYes and OT allocation while adding adaptive candidate selection.
SAVE-Flash outperforms both baselines at lower computational cost. Unlike the trained selector in [1], it reuses the frozen answering VLM without an auxiliary model or additional training. We will include SAVE-Flash and these comparisons in the main text.