acceptodds
Under review as a conference paper at ICLR 2027

SAVE: Let VLMs Allocate Their Own Viewing Budgets for Long-Video Frame Selection

Abstract

Long-video understanding under a limited visual budget requires selecting a small set of frames that captures question-relevant evidence. Existing query-aware selectors often rely on external models to score frame relevance. Balancing relevance with temporal coverage remains challenging: prioritizing high-scoring frames can yield redundant views of a narrow interval, whereas broad coverage can allocate scarce frames to irrelevant content. We propose SAVE (Self-guided Allocation of Visual budgets via Evidential demand), a VLM-native framework for visual budget allocation. We use the same frozen vision-language model (VLM) for evidence scoring and final question answering, converting frame-level relevance scores into a semantic demand distribution over time. Given a fixed frame budget, we formulate selection as an optimal transport problem that minimizes the demand-weighted squared temporal distance from each candidate frame to its nearest selected frame. This objective couples relevance and temporal coverage, adaptively allocating the visual budget without a separate balancing coefficient. Exploiting the one-dimensional temporal structure, we solve this objective exactly and efficiently via dynamic programming, guaranteeing a globally optimal allocation under the proposed objective. Experiments across multiple long-video benchmarks and diverse VLM architectures show consistent improvements over the evaluated baselines in benchmark-level average performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Video Understanding, Vision-Language Models, Frame Selection, Visual Budget Allocation, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eiPXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAVE, a training-free framework for selecting a limited number of frames for long-video question answering by allocating the visual budget according to question-conditioned evidence demand. SAVE first introduces Q-SelfYes, which reuses the same frozen VLM employed for final question answering to estimate the relevance of candidate frames from grouped Yes/No logits, avoiding the need for a separate relevance model. The resulting frame-level relevance scores are normalized into a temporal semantic-demand distribution, and frame selection is formulated as a demand-weighted temporal quantization problem that jointly accounts for semantic relevance and temporal coverage without requiring an explicit trade-off coefficient. By exploiting the one-dimensional temporal ordering of video frames, the proposed allocation objective can be solved exactly using dynamic programming and is further interpreted through a free-mass optimal transport perspective. Experiments on Video-MME, LongVideoBench, and MLVU using four different VLM backbones show that SAVE consistently improves benchmark-level average performance over the evaluated training-free frame-selection baselines under the same final frame budget. Additional ablations suggest that both answerer-aligned relevance estimation and the proposed OT-based temporal allocation contribute to the improvements, with the allocation mechanism providing a larger gain over simpler relevance-based selection strategies.

Critical strengths
  1. The paper is well written and clearly organized, making the motivation, methodology, and experimental results easy to follow.

  2. The proposed method is simple, intuitive, and effective, providing a way to balance question-conditioned semantic relevance with temporal coverage when selecting frames under a limited visual budget.

  3. The paper provides extensive ablation studies and analyses covering the relevance scorer, allocation strategy, frame budget, model scale, candidate-frame density, and computational cost.

Critical weaknesses
  1. The semantic-demand formulation relies on specific design choices, including Softmax normalization and a squared-distance allocation objective, but these choices appear somewhat heuristic. A broader analysis of alternative demand transformations and allocation objectives, e.g., ablations on the Softmax temperature τ\tau, different normalization schemes, or alternatives to the squared-distance objective, would strengthen the justification for the proposed formulation.

  2. Although the OT interpretation is mathematically reasonable, it is not entirely clear whether framing the allocation problem through OT is necessary for obtaining the reported benefits. Simpler formulations, such as weighted temporal allocation or clustering-based approaches, may potentially exhibit similar behavior. While Table 3 compares against several existing selection baselines, comparisons with such simpler allocation alternatives would help clarify the practical benefit of the proposed OT formulation beyond its conceptual interpretation.

  3. According to the authors' analysis, Q-SelfYes provides only a modest improvement over strong external relevance scorers, making its empirical advantage relatively limited. At the same time, repeatedly invoking the full answer VLM to score candidate frames can introduce substantial computational overhead. This concern is further highlighted by prior work [1], which proposes a related strategy that distills a VLM's Yes/No relevance judgments into a smaller external scorer, potentially retaining VLM-aligned relevance estimation while reducing inference cost. A more direct comparison or discussion of this trade-off would better establish the practical advantage of Q-SelfYes.

[1] M-LLM Based Video Frame Selection for Efficient Video Understanding, CVPR 2025.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer eiPX

W1. Demand transformations and allocation objectives. We evaluated alternative Softmax temperatures, demand normalization schemes, and an absolute-distance objective. The sigmoid-based normalization ablation is already reported in Table 3 of the original paper.

ComponentConfiguration3-Avg ↑
DefaultSoftmax T=1T=1 + squared distance69.74
TemperatureT=0.25T=0.2569.06
TemperatureT=0.5T=0.569.23
TemperatureT=2T=268.35
TemperatureT=4T=466.74
NormalizationMin-max66.05
Demand weightsUniform65.03
Distance objectiveSoftmax T=1T=1 + absolute distance69.28

Under the assumption of an absolutely continuous evidence-demand distribution, quadratic cost admits an optimal transport map given by the gradient of a convex potential, as established by Brenier's theorem. Our ablations further support the squared-distance choice empirically.

W2. Comparison with simpler allocation alternatives. We compared SAVE with weighted temporal quantiles (WTQ) and weighted temporal K-means (WTKM).

Allocation method3-Avg ↑
Weighted temporal quantiles (WTQ)68.99
Weighted temporal K-means (WTKM)68.68
SAVE (OT)69.74

SAVE jointly accounts for demand weights and temporal distances, and finds a globally optimal allocation under its objective rather than relying on quantile placement or iterative clustering. It also achieves higher accuracy than both alternatives.

W3. Q-SelfYes and scoring efficiency. One of our motivations is to let the answering VLM itself provide frame-selection signals rather than rely on an external scorer. To improve efficiency, we developed SAVE-Flash, which retains Q-SelfYes and OT allocation while adding adaptive candidate selection.

Method3-Avg ↑Scoring GPU-h ↓Scoring + answering GPU-h ↓Scoring TFLOPs/ question ↓
AKS67.7622.16023.434346.67
WFS-SB68.369.28610.560454.68
SAVE69.7428.27329.5471,313.44
SAVE-Flash69.07‾\underline{69.07}4.2425.516197.31

SAVE-Flash outperforms both baselines at lower computational cost. Unlike the trained selector in [1], it reuses the frozen answering VLM without an auxiliary model or additional training. We will include SAVE-Flash and these comparisons in the main text.

AuthorsOpenReview ↗

Follow-up on your review

Thank you again for your suggestions on the demand formulation, simpler allocation methods, and scoring efficiency. As the discussion period draws to a close, we’d appreciate hearing whether our response and additional experiments addressed your concerns. If any questions remain, we’d be happy to clarify.

Reviewer_44vZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper solves the problem of frame selection in the VLM by utilizing the same frozen VLM to score each candidate frame, then use the selection method to select the final frames to consider both the scores and temporal coverage. Though the paper have shown some gains on some benchmarck than other method, but the the paper’s gain seems relatively limited, especially it use whole VLM model as the scorer. We can see from the table 10, based on the config B128-M32, the scorer use 4.97 GPU hour, and the answer process only use about 2 GPU hour.

Critical strengths
  1. The paper is clearly motivated and presents a coherent framework that connects VLM-based relevance estimation, semantic-demand construction, and exact temporal allocation. The 1-D formulation admits an exact dynamic-programming solution, which is a clean and appealing property.

2.The method is training-free and requires no auxiliary scoring model (reusing the frozen answering VLM). Under a matched 32-frame answering budget, it achieves consistent benchmark-level average gains over the reproduced baselines across three long-video benchmarks and four VLM backbones.

Critical weaknesses
  1. Cost–accuracy trade-off. At B=128→M=32, SAVE achieves 68.52 3-Avg at approximately 6.94 GPU-hours, while Uniform-128 achieves 69.95 at 6.43 GPU-hours. Please report the full-1fps scoring cost and clarify under what compute, latency, or memory constraints SAVE is preferable to using more uniformly sampled frames.

  2. Allocation comparison. The proposed OT-based allocation is not directly compared with existing state-of-the-art allocation strategies, such as AKS. Please compare them using the same relevance scores, candidate frames, and frame budget, and report both accuracy and computational cost.

Questions for authors
  1. Cost–accuracy trade-off. At B=128→M=32, SAVE achieves 68.52 3-Avg at approximately 6.94 GPU-hours, while Uniform-128 achieves 69.95 at 6.43 GPU-hours. Please report the full-1fps scoring cost and clarify under what compute, latency, or memory constraints SAVE is preferable to using more uniformly sampled frames.

  2. Allocation comparison. The proposed OT-based allocation is not directly compared with existing state-of-the-art allocation strategies, such as AKS. Please compare them using the same relevance scores, candidate frames, and frame budget, and report both accuracy and computational cost.

  3. Value of Q-SelfYes. Q-SelfYes exceeds SigLIP by only 0.45 3-Avg, performs worse on LongVideoBench, and requires substantially more computation. Please provide significance and equal-compute comparisons and clarify whether its main advantage is accuracy or avoiding an auxiliary model.

  4. Longest-video performance. With Qwen3.5, SAVE scores 59.78 on the LongVideoBench Long subset, below all four query-aware baselines. Why does its advantage disappear on the longest videos, and is this behavior statistically consistent across backbones?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 44vZ

W1, Q1. Cost–accuracy trade-off. The full-1fps Q-SelfYes scoring cost is 28.273 GPU-hours. To reduce this cost, we developed SAVE-Flash, which retains Q-SelfYes and OT allocation while adaptively selecting at most 128 frames for scoring.

Configuration3-AvgScoring GPU-hScoring + answering GPU-hScoring TFLOPs/ question
SAVE69.7428.27329.5471,313.44
SAVE-Flash69.07‾\underline{69.07}4.2425.516197.31

SAVE-Flash provides competitive accuracy with one quarter of Uniform-128's answering-frame budget. For memory-constrained deployment, our reported 32-frame answering peak is 9.58 GiB versus 12.21 GiB for Uniform-128. We will include SAVE-Flash and these results in the main text.

W2, Q2. Allocation comparison. We compared SAVE and AKS using the same relevance scores (SigLIP or Q-SelfYes), candidate frames, and frame budget.

ScorerAKS 3-AvgSAVE 3-Avg
SigLIP67.3969.29
Q-SelfYes68.0969.74

SAVE outperforms AKS with both scorers.

Q3. Value of Q-SelfYes. One of our motivations is to use the answering VLM itself to provide frame-selection signals, without relying on an external scorer. Q-SelfYes realizes this by reusing the frozen answering VLM for relevance scoring, requiring neither an auxiliary model nor additional training.

Q4. Longest-video performance. The gap with Qwen3.5 is concentrated in single-moment perception questions (43.48% of the Long subset), which require identifying local visual details at a specific moment. These questions favor concentrating the frame budget on a short, relevant segment, with limited benefit from broader temporal coverage. SAVE balances local relevance with coverage across the video, which may be less advantageous for such localized-detail questions under a 32-frame budget.

Nevertheless, SAVE ranks second on this subset when averaged across the other three backbones.

thanks for the auther reponse and add the flash verison of SAVE. But I still donot see any cost–accuracy advantage of SAVE method itself based on its time cost. For the Save flash version, seems we donot know how the details of the method, and on the average preformance drop about 0.7. Also we donot see the performance of the Save and save-flash under other congiguration. So I will maintain my score.

AuthorsOpenReview ↗

SAVE-Flash: Higher Accuracy and Efficiency than Compared Methods; SAVE: Best Overall Accuracy

1. Cost–accuracy comparison. SAVE-Flash achieves higher average accuracy than both AKS and WFS-SB at lower computational cost and model runtime. Even compared with WFS-SB, which is competitive in runtime among existing selectors, SAVE-Flash improves 3-Avg from 68.36 to 69.07 while reducing scoring-plus-answering model time from 10.560 to 5.516 GPU-hours. Original SAVE trades additional computation for higher average accuracy, reaching 69.74.

A comparison of accuracy and computational cost with competing methods is shown below:

Method3-Avg ↑Scoring + answering GPU-h ↓Scoring TFLOPs/question ↓
AKS67.7623.434346.67
WFS-SB68.3610.560454.68
SAVE69.7429.5471,313.44
SAVE-Flash69.07‾\underline{69.07}5.516197.31

Timings cover scoring and answering model computation across the same 6,211 questions.

2. Details of SAVE-Flash. SAVE-Flash retains the original Q-SelfYes scorer and OT allocation, adding adaptive candidate selection over the 1-fps candidate pool:

  • Initial sampling: We score 32 uniformly spaced frames, including the first and last candidates.
  • Adaptive refinement: We add 48 candidates per round for two rounds. The first 32 are selected by repeatedly splitting the interval with the highest priority, defined as L(0.1+0.7R+0.2D)L(0.1+0.7R+0.2D). Here, LL is the current interval’s index span, RR is the higher normalized relevance rank of the two observed frames bracketing its midpoint, and DD is their absolute score difference normalized by the observed-score standard deviation, clipped at 3 and divided by 3. We insert the interval’s midpoint and update the intervals after each selection. The remaining 16 candidates are selected by repeatedly inserting a midpoint into the largest remaining gap, regardless of relevance scores.
  • Batched scoring: All 48 positions are planned using scores available at the start of the round and then scored together. Newly planned positions split intervals but do not supply scores until the next round. This yields 32 → 80 → 128 scored candidates, with each candidate scored only once.
  • Final allocation: We apply the original semantic-demand construction and OT allocation to select the answering frames. Videos with at most 128 candidates are scored in full.

In practice, users can choose SAVE for higher accuracy or SAVE-Flash for greater efficiency, with both variants achieving higher average accuracy than the compared baselines and SAVE-Flash also offering lower computational cost and model runtime.

Reviewer_3Vf7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SAVE, a training-free framework for long-video frame selection. It uses a frozen VLM to estimate frame relevance and formulates budget allocation as a demand-weighted temporal optimization problem, solved exactly using dynamic programming. Experiments on three benchmarks and four VLM backbones demonstrate consistent improvements in average QA accuracy.

Critical strengths
  1. The optimization formulation provides a principled approach to balancing relevance and temporal coverage, supported by an exact solution.

  2. Extensive cross-backbone experiments and controlled ablations support the effectiveness of the proposed allocation strategy.

Critical weaknesses
  1. Limited practical efficiency. VLM-based relevance scoring accounts for approximately 70% of inference computation in the reported deployment setting. The modest gains over strong baselines may not justify this additional cost.

  2. Limited connection between the objective and actual evidence quality. Temporal proximity is only a proxy for evidence substitutability. Exact optimization of this objective does not guarantee that the selected frames preserve the evidence required for answering.

Questions for authors
  1. How does SAVE compare with existing selectors under matched total inference FLOPs or latency rather than only matched final frame budgets?

  2. Can the authors directly evaluate evidence coverage and analyze failures involving abrupt scene changes or temporally distant events?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3Vf7

W1, Q1. Computational efficiency. Q-SelfYes was designed to demonstrate that frame selection can use the answering VLM's own signals without an external scorer. To address your efficiency concern, we developed SAVE-Flash by adding adaptive candidate selection to the original Q-SelfYes pipeline. It first scores 32 uniformly sampled frames, then refines sampling in two rounds using relevance scores and temporal-gap exploration, while retaining the original scorer and OT allocation. SAVE-Flash achieves higher accuracy than AKS and WFS-SB at lower computational cost, as shown below.

Method3-Avg ↑Scoring GPU-h ↓Scoring + answering GPU-h ↓Scoring TFLOPs/ question ↓
AKS67.7622.16023.434346.67
WFS-SB68.369.28610.560454.68
SAVE69.7428.27329.5471,313.44
SAVE-Flash69.07‾\underline{69.07}4.2425.516197.31

Timings cover scoring and answering model computation across all 6,211 questions. We thank the reviewer for raising this question and will include SAVE-Flash in the revised manuscript.

W2, Q2. Evidence preservation. We recruited three volunteers to independently assess whether the selected frames provided sufficient evidence to support the correct answer on 100 cases. Excluding five cases with incomplete assessments, we obtained complete ratings for 95 cases. We classified evidence as sufficient when all three agreed, borderline when two judged it sufficient, and insufficient when at least two judged it insufficient.

Evidence supportUniformSAVE
Sufficient53.68%64.21%
Borderline36.84%30.53%
Insufficient9.47%5.26%

We further examined cases involving abrupt scene changes and temporally distant events:

ScenarioEvidence supportUniformSAVE
Abrupt scene changes (14 cases)Sufficient42.86%64.29%
Borderline21.43%28.57%
Insufficient35.71%7.14%
Temporally distant events (11 cases)Sufficient36.36%63.64%
Borderline36.36%36.36%
Insufficient27.27%0.00%

SAVE increases the proportion of cases with unanimously judged sufficient evidence by 10.53 percentage points over Uniform, with improvements in both challenging scenarios. These results directly support SAVE's ability to preserve answer-relevant evidence beyond optimizing the temporal-distance objective.

The author addressed my concerns, I kept my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses to Reviewer Concerns

Dear Area Chair and Reviewers,

Thank you for your feedback. As the discussion period approaches its end, we summarize the additional results addressing the main concerns:

Computational efficiency (all reviewers). We developed SAVE-Flash by adding adaptive candidate selection while retaining Q-SelfYes and OT allocation. It achieves 69.07 average accuracy, compared with 67.76 for AKS and 68.36 for WFS-SB, with lower scoring FLOPs and scoring-plus-answering model runtime. It requires neither an auxiliary scorer nor additional training.

Evidence preservation (3Vf7). Three volunteers assessed the selected frames, yielding complete ratings for 95 cases. SAVE increases unanimously sufficient evidence from 53.68% to 64.21% over uniform sampling, with improvements in cases involving abrupt scene changes and temporally distant events.

Allocation comparisons (44vZ, eiPX). Using identical relevance scores, candidate frames, and frame budgets, SAVE outperforms AKS with both SigLIP and Q-SelfYes. SAVE also achieves 69.74 average accuracy versus 68.99 for weighted temporal quantiles and 68.68 for weighted temporal K-means.

Design choices (eiPX). We evaluated alternative Softmax temperatures, normalization schemes, and an absolute-distance objective. These ablations support the default Softmax and squared-distance formulation.

Longest-video performance (44vZ). Our analysis locates the Qwen3.5 gap primarily in questions requiring local visual details within a short segment, where broader temporal coverage may offer less benefit. SAVE ranks second on the Long subset when averaged across the other three backbones.

Detailed results are provided in our individual responses. We will incorporate SAVE-Flash and its evaluation into the revised manuscript.

Before the discussion closes, could the Area Chair and reviewers let us know whether any concerns remain, or whether additional experiments or analyses would help clarify the evaluation? We would welcome the opportunity to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.