acceptodds
Desk rejected at ICLR 2027

Transport What Matters: Evidence-Directed Optimal Transport for Long Video Understanding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Long video understanding often requires reducing thousands of observations to a small frame budget that a vision–language model can process. Existing selectors prioritize query relevance, diversity, or adaptive sampling, but leave implicit how evidence from discarded observations should be represented by the retained set. We introduce Evidence-Directed Optimal Transport (E-DOT), a training-free framework that casts frame selection as transporting task-conditioned evidence mass from dense video observations to a sparse support of frames. Query relevance over dense observations and temporal extent define the source mass. A directed ground cost combines temporal proximity with sparse local-helpfulness estimates from a frozen model: transport to a nearby frame that preserves or improves source evidence is cheap, whereas losses in relevance or helpfulness are penalized. Relaxing the destination marginal lets multiple sources share representatives, allowing the support to concentrate on evidence that remains poorly covered. The resulting support-selection objective admits monotone-submodular greedy optimization and coverage characterizations. Across four long-video question answering benchmarks and three answerers, E-DOT outperforms all compared frame selectors under a fixed answerer budget, while matched ablations isolate the contributions of local helpfulness, directed transport geometry, and coverage-aware support selection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Frame Selection, Long Video Understanding, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Lh3pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces E-DoT, a training-free frame selector for long-video understanding. E-DoT ranks frames independently, it models how selected frames can represent discarded observations. Frames are described by time, query relevance, and sparse local-helpfulness estimates. A directed replacement cost penalizes losses in these evidence signals rather than just selecting the frames, while a coverage objective selects complementary representatives under a fixed frame budget.

Critical strengths
  • The replacement formulation is well motivated, for long videos where independently selecting relevant frames can produce redundant observations due to the fixed number of selection. Distinguishing the importance of an observation from how well another frame represents it gives the method a clear rationale. Directionality is also an intuitive decision a frame exposing additional answer evidence may replace a weaker previous observation more safely than the reverse.

  • The results are promising and well presented, with relevant ablations supporting the design. The comparisons isolate directed versus symmetric costs and coverage updates versus one-shot selection. The observer-free results and controls using the same observer information also help establish that the gains are not solely due to additional VLM calls. Evaluation across multiple benchmarks and answerers strengthens the practical contribution.

  • The authors establish the transport reduction and greedy guarantee, while explicitly showing that low transport loss does not guarantee preservation of answer-critical evidence.

Critical weaknesses
  • Although, the replacement relation is defined through scalar relevance scores, which do not establish that one frame preserves another’s evidence. However, close frames can have similar scores while exposing different facts specially for fast movement of fine grained motion that may occur in long video. Additionally, helpfulness is measured for three-frame neighborhoods and interpolated onto individual candidates. The optimization guarantees therefore apply to coverage of these proxy coordinates, rather than semantic evidence preservation. This does not negate the empirical gains, but limits the interpretation of the replacement claim.

  • The full computational tradeoff remains unclear. Fixing the answerer at 32 frames does not equalize total inference cost because the full method adds up to eight three-frame observer calls. The observer-free variant and matched controls are useful, but the reported selection-stage timings do not establish end-to-end latency.

  • The replacement cost assumes normalized time, relevance, and helpfulness are directly comparable. However, scaling them to [0,1] does not guarantee this, and the appendix notes that rescaling affects the objective. A sensitivity analysis could test how robust this is.

Questions for authors

1- Beyond aggregate QA accuracy, what supports the assumption that a nearby frame with higher relevance and helpfulness can preserve the source frame’s answer-bearing information?

2- What is the end-to-end cost of the full configuration relative to the observer-free variant and the strongest baseline? It would be helpful if the cost is broken down to relevance scoring, observer calls, selection etc...

6 comments
AuthorsOpenReview ↗

Response to Reviewer Lh3p (1/3)

Thank you for your careful review. We respond to your concerns below.

W1: Similar relevance scores and interpolated neighborhood helpfulness do not ensure that nearby frames preserve the same evidence, especially during brief changes. The guarantees concern proxy coverage rather than semantic preservation.

A1: Thank you for recognizing the motivation behind modeling frame replacement. Allowing a nearby frame to serve as a low-cost representative when neither relevance nor helpfulness decreases is a deliberate design choice.

1. The replacement cost uses task-related semantic signals. Relevance measures how well the visual content matches the question, while helpfulness measures how much the three-frame context supports answering it. These signals therefore capture some task-related semantic information. For example, a later frame within the same event may show the required text or action more clearly, and we want to allow this improvement. Eq. (5) retains temporal distance and uses one-sided penalties for decreases in either evidence score. Taking the maximum prevents a loss in one component from being masked by the others (Lines 212–223).

2. This content-conditioned proxy does not fully determine whether one frame preserves another's semantic evidence. Different facts can receive similar scores, and neighborhood helpfulness and its interpolation may miss brief changes. This is a tradeoff between the cost of sparse observations and how precisely they describe the evidence. The paper explicitly discusses this tradeoff in Section 3.1 and Appendix H. Eq. (19) characterizes the worst-case one-sided proxy loss between two frames; Theorem 1 / Eq. (20) extends this characterization to the selected set. Both statements concern the specified class of monotone Lipschitz functions of these proxy coordinates and do not guarantee complete preservation of semantic evidence (Appendix B, Lines 819–821). We will further clarify this meaning of “replacement.” For direct evidence-retention tests, see our response to Q1 below.

W2 & Q2: A fixed 32-frame answerer budget does not equalize total inference cost. Report end-to-end latency for B=8, B=0, and WFS-SB, with a breakdown of the main stages.

A2: Fixing the answerer input at 32 frames does not equalize total inference cost. We therefore add end-to-end measurements.

1. We measure the end-to-end runtime of the full pipeline. We use Video-MME without subtitles: 100 videos per duration group (short, medium, and long), with two questions per video, totaling 600 questions. The answerer is Qwen2.5-VL-7B, and the observer is Qwen2.5-VL-3B. Models are preloaded and warmed up. We run the complete pipeline three times for each question, including video reading, relevance scoring, observation, selection, and answering, without reusing cached per-sample results. All methods and repetitions for a given question run on the same single NVIDIA A100 GPU.

Times are means in s/question. E-DOT parentheses give BB; extra time is E-DOT (8) minus the row latency.

MethodLatency (s)Extra time (s)
WFS-SB rerun35.7483.271
BLIP relevance Top-KK36.4672.552
E-DOT (0)36.6902.329
E-DOT (8)39.019—

2. Separate diagnostic runs measure the time spent in each stage. Mean times for individual stages are shown below, in seconds per question.

StageWFS-SBE-DOT (0)E-DOT (8)
Video reading and decoding15.26915.49316.127
Relevance scoring14.91916.88116.922
Observer, excluding video reading001.438
Frame selection1.0850.0120.012
Final answering3.0113.0823.078
Other0.0040.0570.062

We average end-to-end latency over three complete runs per question; separate diagnostic runs identify the time spent in each stage. They are not an exact decomposition of the end-to-end runs and therefore need not sum to the reported means.

These full-method timings use different relevance models: BLIP for E-DOT and BLIP-2 for WFS-SB. Similar candidate FPS therefore does not imply equal scoring time.

The observer takes 2.148 seconds per question, including its own video reading. Of this, approximately 0.710 seconds is video reading and is already included in the decoding row. Adding it again would count the same video-reading time twice. Actual execution records show an average of 7.71 observer calls and 23.11 image inputs per question at B=8B=8.

The full configuration adds 2.329 seconds per question over the observer-free variant and 3.271 seconds per question over WFS-SB. The stage-level diagnostics show that video reading and relevance scoring account for most of the runtime. Frame-selection time alone therefore does not represent the end-to-end cost of the method. For latency by video duration, see our response to Reviewer PQ5o, W2 & Q2.

AuthorsOpenReview ↗

Response to Reviewer Lh3p (2/3)

W3: Normalization to [0,1] does not make time, relevance, and helpfulness semantically comparable. Test sensitivity to their relative scales.

A3: Thank you for raising this concern. We agree that normalizing time, relevance, and helpfulness to [0,1][0,1] only aligns their numerical ranges. Equal numerical differences can still have different meanings across the three components, and rescaling a component can change the objective. We therefore add a component-weight sensitivity analysis to examine how changes in relative scale affect the quality of the selected frames.

1. We halve or double one component's weight while keeping the others fixed. With B=8B=8 and K=32K=32, we vary the weights of the three cost components from the default (λt,λr,λh)=(1,1,1)(\lambda_t,\lambda_r,\lambda_h)=(1,1,1). We halve or double one weight at a time and leave the other two at 1. For each configuration, we divide all weighted costs by the common upper bound U=max⁡(λt,λr,λh)U=\max(\lambda_t,\lambda_r,\lambda_h):

=\frac{1}{U}\max\\!\left\\{ \lambda_t|t_i-t_j|, \lambda_r[r_i-r_j]_+, \lambda_h[h_i-h_j]_+ \right\\}.$$ This uniform scaling keeps costs in $[0,1]$ while preserving the relative component weights and the minimizers of the weighted objective. We therefore retain the greedy initialization $d_i=1$. Source masses, candidates, evidence scores, and all other settings are held fixed, and all configurations use the same evaluation samples. BSR measures coverage of answer-relevant scenes, and BDS measures the distribution of sampled frames across scenes. Both metrics are reported on a 0–100 scale. | Weights (time, relevance, helpfulness) | Video-MME BSR | Video-MME BDS | LongVideoBench BSR | LongVideoBench BDS | |:---:|:---:|:---:|:---:|:---:| | (1, 1, 1) | 80.4 | 84.5 | 72.8 | 77.7 | | (0.5, 1, 1) | 80.0 | 84.1 | 73.4 | 78.4 | | (2, 1, 1) | 80.4 | 84.4 | 73.1 | 77.8 | | (1, 0.5, 1) | 80.5 | 84.2 | 73.6 | 78.6 | | (1, 2, 1) | 80.2 | 84.3 | 73.5 | 78.2 | | (1, 1, 0.5) | 80.8 | 84.9 | 72.5 | 78.2 | | (1, 1, 2) | 80.0 | 84.5 | 73.5 | 78.4 | **2. Both metrics remain relatively stable within the tested range.** Based on the displayed values, the largest absolute BSR changes from the default are 0.4 points on Video-MME and 0.8 points on LongVideoBench. The corresponding maximum BDS changes are 0.4 and 0.9 points. These results show that answer-relevant scene coverage and the distribution of sampled frames across scenes remain relatively stable when a single component's weight is halved or doubled, providing empirical support for robustness to scale changes within this range. For the CLIP and SigLIP score mappings used in Table 5, see our response to Reviewer qpUS, Q3.
AuthorsOpenReview ↗

Response to Reviewer Lh3p (3/3)

Q1: Beyond QA accuracy, what supports the assumption that nearby frames with higher relevance and helpfulness preserve the source frame's answer cues?

A4: Thank you for highlighting this important distinction. The matched ablations in Table 2(b) support the contributions of the directed cost and coverage updates to QA performance. The scene-coverage evaluation in Appendix G.1 and the visualizations in Appendix I further support the value of the final selected frames as answer evidence. However, these results do not directly test whether a representative preserves specific answer cues from a discarded source frame. We therefore add two evaluations of actual source-to-representative assignments, examining retention of evidence intervals and specific visible cues.

We compare each actual representative with a control matched for temporal distance from the same source. Both evaluations use the final selected frames with B=8B=8 and K=32K=32. Following Eq. (9), we assign each source frame to its minimum-cost representative in this set. We include only assignments where the source is not selected, the representative is within 4 seconds of the source, and neither relevance nor helpfulness decreases. These conditions apply only to sample selection for the supplementary evaluation.

1. We check whether the representative stays within the source frame's annotated answer-evidence interval. For each benchmark, we sample questions from 100 distinct videos and use the existing answer-relevant scene annotations. We evaluate source frames inside an annotated interval with positive source weight, ai>0a_i>0. Each control comes from the same question and is matched for temporal distance, without filtering on relevance or helpfulness. Scene annotations are used only for evaluation.

Pairs: eligible assignments; Videos: included/sampled. Retention is reported in %, and differences in percentage points.

DatasetPairsVideosRepresentativeControlDifference [95% CI]
Video-MME1,93784 / 10087.2%68.5%+18.7 [9.1, 28.5]
LongVideoBench1,09977 / 10070.4%58.4%+12.0 [1.7, 22.4]

We first calculate retention rates within each video, then average them with equal weight per video. Confidence intervals for the differences use a paired bootstrap at the video level. With equal weight per video, actual representatives more often stay within the source frame's evidence interval on both benchmarks. When frame pairs receive equal weight instead, the differences become smaller and their confidence intervals include zero. Interval retention tests whether a representative stays within the same answer-relevant segment; the next evaluation examines retention of specific visible cues.

2. We check whether a visible answer cue fixed in advance from the source frame is fully retained. We separately sample 48 videos, 24 per benchmark, with no overlap with the first evaluation. Using only the source frame, question, and reference answer, we first fix one task-relevant cue that is actually visible in the source. We then check whether the representative and control each retain it in full. For example, if the source contains readable text naming a location, we check whether that text is still readable in each target frame.

Controls are again matched for temporal distance. Here, we also require at least one of their relevance or helpfulness scores to be at least 0.1 below the source frame's corresponding score, to test local candidates with a clear score difference. Reference answers are used only for evaluation, and cue-retention judgments are confirmed by two evaluators. Of the 48 source frames, 32 contain a cue that can be evaluated; the other 16 samples are recorded separately. Partial retention does not count as full retention.

Videos: evaluable/sampled. Entries show full cue retention by representatives and lower-score controls, as counts and percentages.

DatasetVideosRepresentativeControl
Video-MME16 / 2415/16 (93.8%)7/16 (43.8%)
LongVideoBench16 / 2414/16 (87.5%)8/16 (50.0%)
Combined32 / 4829/32 (90.6%)15/32 (46.9%)

The two evaluators' review shows that actual representatives more often fully retain the fixed visible task cue in samples with an evaluable cue. Together with the interval evaluation using equal weight per video, these results provide empirical support beyond aggregate QA accuracy for using nearby frames as representatives when neither relevance nor helpfulness decreases. This conclusion is limited to the stated filters and evaluation procedures; it does not guarantee preservation of all answer-relevant information from arbitrary source frames.

Reviewer_PQ5oOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper slove multi-frame selection from a novel perspective—taking it as an evidence-transport (optimal-transport) problem—and the method is innovative and practical. The experiments and the corresponding ablations are enough, have improvements across different backbone VLMs and different benchmarks.

Critical strengths

1 The transport formulation reduces to a source-weighted coverage objective with a standard greedy approximation guarantee. The accompanying theoretical analysis clarifies the behavior and optimization properties of the proposed selector.

2 E-DOT is evaluated on four long-video QA benchmarks with three answerers and compared against recent selectors under matched frame budgets. It achieves the best reported accuracy in all twelve answerer–benchmark settings. The ablations examine directionality, observer feedback, relevance-weighted source importance, and coverage updates, while paired tests support most gains for Qwen2.5-VL-7B.

Critical weaknesses
  1. E-DOT consistently performs well, but its gains over the strongest baseline are modest in several settings, sometimes below one point. The paper also reports that WFS-SB remains stronger on single-scene LongVideoBench questions, while E-DOT provides larger gains on multi-scene questions. It would be helpful to characterize this applicability boundary more clearly and explain which question or video types benefit most from directional, coverage-aware selection.

2.The full method requires up to eight additional calls to a frozen 3B observer, while the observer-free variant is already competitive and outperforms WFS-SB on average across all three answerers. The additional helpfulness signal provides relatively modest and occasionally inconsistent gains across benchmarks and backbones. Please report the end-to-end computational cost of the observer calls and discuss when the full configuration is preferable to the cheaper observer-free variant.

Questions for authors
  1. E-DOT consistently performs well, but its gains over the strongest baseline are modest in several settings, sometimes below one point. The paper also reports that WFS-SB remains stronger on single-scene LongVideoBench questions, while E-DOT provides larger gains on multi-scene questions. It would be helpful to characterize this applicability boundary more clearly and explain which question or video types benefit most from directional, coverage-aware selection.

  2. The full method requires up to eight additional calls to a frozen 3B observer, while the observer-free variant is already competitive and outperforms WFS-SB on average across all three answerers. The additional helpfulness signal provides relatively modest and occasionally inconsistent gains across benchmarks and backbones. Please report the end-to-end computational cost of the observer calls and discuss when the full configuration is preferable to the cheaper observer-free variant.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PQ5o (1/2)

Thank you for your thoughtful review. We respond to your concerns below.

W1 & Q1: Gains are sometimes modest, and WFS-SB remains stronger on single-scene LongVideoBench questions. Clarify which video and question types benefit most from E-DOT and where its advantages are limited.

A1: Thank you for this helpful suggestion. Under a fixed frame budget, E-DOT is more likely to help when relevant evidence is spread across a video and candidate frames differ in the quality of the evidence they provide.

1. E-DOT shows larger gains on multi-scene questions in the LongVideoBench subset. On the scene-annotated subset, E-DOT (B=8B=8) improves over WFS-SB by 3.3 percentage points on multi-scene questions but answers 15 fewer of the 802 single-scene questions correctly (approximately 1.9 points lower). In this setting, its advantage mainly appears on questions that require coverage of several answer-relevant scenes. E-DOT has no QA-accuracy advantage over WFS-SB on this single-scene subset.

2. Additional Video-MME results show larger gains over WFS-SB on long videos and reasoning questions. We use Qwen2.5-VL-7B with K=32K=32, consistent with the main table. Gains are smaller on short and medium-length videos, and neither counting nor summarization improves.

The duration results below report accuracy in percent, with 900 questions in each group. E-DOT parentheses denote BB.

DurationWFS-SBE-DOT (0)E-DOT (8)
Short74.374.974.8
Medium64.664.764.9
Long53.754.956.6

These duration results agree with Figures 3 and 6. We will correct the accompanying Video-MME summary in Appendix F.3 to gains over Uniform of 2.1, 4.1, and 6.3 points, computed from unrounded counts. For long videos, the gain is 57 correct answers (509 versus 452 out of 900), or approximately 6.33 points; subtracting the rounded table entries gives 6.4.

The question-type table below uses Qwen2.5-VL-7B, whereas the question-type panel in Figure 3 uses InternVL3-8B. We combine perception and recognition into one category.

Question typeQuestionsWFS-SB (%)E-DOT (8) (%)Difference (pp)
Reasoning97258.560.8+2.3
Perception and recognition99869.570.6+1.1
OCR13974.877.0+2.2
Counting26841.841.4-0.4
Summarization32378.678.0-0.6
Overall270064.265.4+1.2

3. These observations are consistent with the design, but do not isolate each component's contribution. Coverage updates reduce repeated selection of content that is already well represented. The directed cost allows nearby frames with higher relevance and local helpfulness to represent weaker frames. We will clarify where the method helps in the revision, keeping the conclusions within the video and question types tested here.

AuthorsOpenReview ↗

Response to Reviewer PQ5o (2/2)

W2 & Q2: Up to eight additional 3B observer calls yield modest, sometimes inconsistent gains over the competitive B=0 variant. Report the end-to-end overhead and explain when B=8 is worthwhile.

A2: Thank you for this suggestion. We agree that the observer's accuracy gains need to be weighed against its additional cost.

1. The full configuration increases mean end-to-end latency by approximately 6.3% over the observer-free variant. We evaluate Video-MME without subtitles using 100 videos from each duration group (short, medium, and long), with two questions per video, totaling 600 questions. The answerer is Qwen2.5-VL-7B with 32 input frames, and the observer is Qwen2.5-VL-3B. Models are preloaded and warmed up. We run the complete pipeline three times for each question without reusing cached per-sample results at the application level. All methods and repetitions for a given question run on the same single NVIDIA A100 GPU.

Times are means in s/question. E-DOT parentheses give BB; extra time is E-DOT (8) minus the row latency.

MethodLatency (s)Extra time (s)
WFS-SB rerun35.7483.271
BLIP relevance Top-KK36.4672.552
E-DOT (0)36.6902.329
E-DOT (8)39.019—

In a separate diagnostic run, the observer takes 2.148 seconds per question on average, including its own video reading. The average workload is 7.71 observer calls and 23.11 image inputs per question. The end-to-end measurements already include the observer, so this diagnostic time should not be added again. For stage timings, see our response to Reviewer Lh3p, W2 & Q2.

2. The relative latency overhead is lower for long videos in this test. The table combines latency measurements from 600 questions with accuracy changes from the full 2,700-question Video-MME evaluation. The two evaluations therefore use different sample sizes.

Times are in s/question; both changes compare B=8B=8 with B=0B=0.

DurationWFS-SBE-DOT (0)E-DOT (8)ΔTime (s, %)ΔAcc. (pp)
Short6.1466.0627.986+1.924 s (+31.7%)−0.1
Medium18.74918.94221.450+2.508 s (+13.2%)+0.2
Long82.34885.06687.620+2.554 s (+3.0%)+1.7
Mean35.74836.69039.019+2.329 s (+6.3%)+0.6

For short-video applications with strict response-time requirements, B=0B=0 is more attractive.

3. The observer can identify answer evidence that relevance scores alone do not distinguish. Several frames may all be relevant to a question, while only some show the key action or detail. The paper's case studies show that feedback can sometimes correct an answer by changing just one selected frame, illustrating the value of additional observation for selecting key evidence.

thanks for the auther reponse, I will maintain my score as weak acceptance.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer PQ5o,

Thank you for reviewing our responses and for your constructive feedback. If there are any concerns we have not fully addressed, we would be happy to discuss them further.

Thanks and regards.

Reviewer_qpUSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes E-DOT, a training-free frame selector for long-video QA. Each fixed fps candidate gets a BLIP relevance score and a helpfulness score interpolated from multiple Yes/No readouts of a frozen VLM. A candidate can represent a nearby frame at low cost when neither of its scores is lower, and a greedy routine picks the fixed number of frames that minimize the relevance-weighted cost of the best representative for every candidate. The paper reports the best accuracy among the compared selectors on four benchmarks by using three VLMs ().

Critical strengths

The method is simple and training-free, and the selection step runs in well under a second per question. The derivations are correct. The evaluation covers four benchmarks and three answerers, and the appendix reports results that go against the method, such as Top-K winning on LVBench and the target-KL refit matching greedy selection.

Critical weaknesses

The main weaknesses are as follows.

  • Eq. (5) sees only time and two scalars, so a nearby frame with equal or higher scores can replace any frame whatever it shows, and since source mass, cost, and probe placement all depend on the relevance scorer, the observer can only reorder frames the scorer already rated high, which matches the flat LVBench hit rate. Both scores are also conditioned on the answer choices, and every benchmark is multiple choice. Nothing separates the gain due to that choice text, and open-ended selection is untested although MLVU’s generation tasks were available.

  • GIFT [1] already selects frames with a one-directional substitute relation, which contradicts the first contribution as stated, and SeViLA [3] introduced the yes-probability frame score used in Eq. (2). A.I.R. [2] spends a similar VLM budget per question and reports InternVL3-8B results within 0.2 points of E-DOT and ahead on two benchmarks, yet it is absent from Table 1. None of the three is cited, so please clarify the novelty against them.

  • The paper does not clarify how any baseline was run, and the WFS-SB paper reports higher numbers for WFS-SB than Table 1 on the same uniform baseline. Against those published values E-DOT loses on two of three benchmarks with InternVL3-8B, and most MDP3 and AKS cells match entries in the WFS-SB paper’s table rather than a shared re-run. The claim of winning all twelve settings cannot stand on this evidence.

  • Plain BLIP Top-K is within 0.2 points of observer-free E-DOT on average and ahead on LVBench, and with the observer the gap stays under one point. Removing relevance weighting costs far more than removing the directed cost, and the target-KL refit matches greedy selection, so neither the transport view nor the solver explains the results. The one-shot ranking control scores below uniform sampling, so the large gain credited to coverage updates comes from a broken control.

  • Only B=8 with Qwen2.5-VL-7B is tested against WFS-SB, and one of the four gains survives correction for multiple tests, while the other two answerers win by a few questions and lose LongVideoBench accuracy when the observer is added. The 1/130 speed figure compares two solvers on a stage that costs 0.03 seconds, while BLIP scoring is not timed and the observer adds over ten seconds per question. Neither the margins nor the speed advantage is established.

  • Several figures are hard to read at print size. The 3D plots in Figure 2 are about 2 cm wide with unreadable axis labels, the radar chart in Figure 3 starts its axis at 45 and labels only one series, and the case-study strips in Figures 7 to 12 compress 32 ticks into about one centimeter and separate shared from exclusive frames by shade alone.

[1] GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding, CVPR'26. [2] A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering, ICLR'26. [3] Self-Chained Image-Language Model for Video Localization and Question Answering, NeurIPS'23.

Questions for authors

During the rebuttal, in addition to resolving concerns for major weaknesses, I would like to hear the answers for the following questions.

  • For each baseline cell in Tables 1 and 7, was the number re-run or copied, and with which code, candidate pool, and query text?

  • How many frames does each baseline receive on these videos? Table 15 implies that on the other questions WFS-SB answers 703 correctly against 698 for E-DOT (B=8), so please report all selectors by duration group.

  • How are CLIP and SigLIP scores mapped to [0,1] in Table 5?

  • Are the selected frames given to the answerer as a video or as a list of images, and does the answerer receive their timestamps? Selected frames are unevenly spaced while uniform frames are not, so this affects the temporal reasoning categories and the comparison with uniform sampling.

7 comments
AuthorsOpenReview ↗

Response to Reviewer qpUS (1/5)

Thank you for your careful review. We respond to your concerns below.

W1: Scalar scores may not preserve answer evidence, and the observer may only reorder highly relevant frames. Separate the gains from answer-choice text and clarify the untested open-ended setting.

A1: Relevance and helpfulness are content-conditioned proxies for replacement quality. Different answer-relevant facts can have similar scores; event identity and cross-frame relations may not be recoverable from them (Lines 147–150 and 819–821).

1. We test whether actual representatives retain source-frame evidence. With B=8B=8 and K=32K=32, we evaluate assignments where the source is unselected, the representative is within 4 seconds, and neither score decreases. Evidence-interval retention exceeds distance-matched controls by 18.7 points on Video-MME and 12.0 on LongVideoBench, with equal weight per video. With equal weight per frame pair, gains shrink and the CIs include zero. A separate review by two evaluators finds full retention of a fixed visible cue in 29/32 representatives versus 15/32 distance-matched controls with a score decrease of at least 0.1. These 32 evaluable samples come from 48 sampled videos. Full protocols and results are in our response to Reviewer Lh3p, Q1.

Relevance measures frame–question alignment; helpfulness estimates the support of a three-frame context. A later frame in the same event may show the required text or action more clearly. Eq. (5) allows this improvement by retaining temporal distance and penalizing only score decreases; the maximum prevents one loss from being masked by another. Eq. (19) gives the worst-case one-sided loss for proxy functions that are nondecreasing in relevance and helpfulness and 1-Lipschitz. Each selection updates the remaining source-weighted loss (Eqs. (9)–(11); see W2 below). With candidates and scores fixed, directed costs and coverage updates improve QA accuracy on all three ablation benchmarks (Table 2(b)).

2. Helpfulness changes representative selection within the full candidate pool. Candidates cover the full video without a hard BLIP Top-KK filter, and the solver evaluates marginal gains for every candidate. Relevance influences source weights and probe placement; helpfulness changes replacement costs. The probe schedule also uses neighboring visual-embedding differences and a similarity penalty relative to selected blocks (Appendix A.4).

On LVBench, adding the observer replaces an average of 6.04 of the 32 selected frames, and 77.24% of the newly selected frames lie outside BLIP’s Top-32. Thus, the selection changes introduced by observer feedback are not confined to the highest-ranked 32 candidates. These supplementary statistics are reported here.

LVBench hit rate changes from 55.2% to 55.4% with feedback (Table 16). It records whether any selected frame falls in a reference interval, without identifying its answer cues or relevance rank. On LongVideoBench, feedback improves scene coverage (BSR) from 0.704 to 0.728 (Table 14).

3. E-DOT remains ahead when selection uses only the question. We compare selection with and without choice text:

MethodText input during frame selectionVideo-MMELongVideoBench
WFS-SBQuestion only63.363.1
WFS-SBQuestion + all choices64.264.4
E-DOTQuestion only (neither scoring module uses choices)64.364.5
E-DOTQuestion + all choices (both scoring modules)65.464.8

With question-only selection, E-DOT leads by 1.0 and 1.4 points, so its advantage on these benchmarks does not depend entirely on choice text. Selection-query settings are detailed in W3 & Q1 below.

4. Our main evaluation focuses on multiple-choice QA. To address this concern, we additionally test the observer-free variant on two open-ended MLVU-Test tasks. On 46 Subplot and 40 VideoSum questions, the three 32-frame methods share the local Qwen2.5-VL answerer and generation settings. We use the official grading prompts and rules with gpt-4.1-2025-04-14 (temperature 0).

MethodInputSubplotVideoSumG-Avg
GPT-4o†0.5 fps6.8004.9405.870
Uniform32 frames5.7924.7505.271
WFS-SB‡32 frames6.1414.4505.296
E-DOT (B=0B=0)32 frames6.4764.9505.713

† Official MLVU-Test report, for reference only; not rerun or rescored. ‡ Official implementation and default selection parameters.

E-DOT improves both tasks, with G-Avg gains of 0.442 over Uniform and 0.417 over WFS-SB.

AuthorsOpenReview ↗

Response to Reviewer qpUS (2/5)

W2: Clarify novelty relative to GIFT's directed substitution and SeViLA's Yes-probability scoring, and address the missing A.I.R. comparison.

A2: Thank you for pointing out these works. We will add the citations and revise wording that may imply we introduced directed replacement for frame selection.

1. E-DOT uses free-target sparse OT to optimize a source-weighted coverage objective. Relevance and temporal extent determine source importance. Temporal distance and decreases in relevance or local helpfulness determine the directed replacement cost. Target mass is allocated freely among KK retained frames, allowing multiple sources to share a representative. The OT objective reduces exactly to (Sections 3.2–3.4, Eqs. (3)–(9)):

D(S)=∑iaimin⁡j∈Scij→,∣S∣=K.D(S)=\sum_i a_i\min_{j\in S}c^\rightarrow_{ij},\qquad |S|=K.

Each new frame is valued by its reduction in the remaining weighted loss, so sources already well represented contribute less to later selections. The coverage gain is monotone and submodular, allowing greedy optimization with an approximation guarantee (Section 3.5). Appendix B characterizes this loss within the defined proxy-function class.

2. The methods differ in their selection objectives and use of feedback.

  • GIFT scores each candidate by its irreplaceability relative to more relevant candidates, then removes selected frames and recomputes the remaining candidates' scores after each round (GIFT, Eqs. (4)–(5), Algorithm 1). E-DOT keeps all source observations in the objective and retains the coverage provided by selected frames when evaluating later candidates.
  • SeViLA scores individual frames using Yes probabilities and selects Top-KK. E-DOT estimates contextual helpfulness from sparse three-frame neighborhoods, uses it in directed transport costs, and optimizes the selected set. We will cite SeViLA near Eq. (2).
  • A.I.R. uses VLM feedback to verify candidates and expand sampling during iterative search. E-DOT estimates helpfulness under a separate observation budget before optimizing a fixed set objective. Its coverage updates require no further observer calls.

Table 2(b) tests directed costs and coverage updates with the same candidates and scores; Table 2(c) tests directed costs and source weights without an observer.

3. We add a direct comparison with GIFT. Within each answerer setting, we match the evaluation data, answerer preprocessing, and final 32-frame budget. We will add the GIFT results to the main experiments. Accuracy is in percent; E-DOT parentheses denote BB.

AnswererMethodVideo-MMELongVideoBenchMLVULVBench
Qwen2.5-VL-7BGIFT64.663.268.647.2
Qwen2.5-VL-7BE-DOT (0)64.864.470.648.9
InternVL3-8BGIFT66.961.973.353.5
InternVL3-8BE-DOT (0)67.962.974.354.3
LLaVA-OneVision-7BGIFT61.260.367.747.9
LLaVA-OneVision-7BE-DOT (0)61.661.168.449.2

The observer-free variant outperforms GIFT in these settings, showing that the improvements do not depend entirely on additional observer calls.

Per-question means are reported as Video-MME / LongVideoBench.

StatisticGIFTE-DOT (0)
Share of selected frames within 2 s of another selected frame (%)74.63 / 70.8234.45 / 27.25
Distinct relevant scenes covered1.189 / 0.8641.342 / 0.986
BSR0.738 / 0.6210.798 / 0.704

E-DOT B=0B=0 has a lower share of temporally adjacent frames and covers more relevant scenes on both benchmarks.

4. We adapt A.I.R. to a fixed 32-frame answerer budget. A.I.R. uses the same VLM for analysis and answering. Its InternVL3-8B configuration analyzes an average of 36.5 frames on Video-MME and supplies 24.8 frames to the answerer. E-DOT uses a separate Qwen2.5-VL-3B observer and supplies 32 frames to the answerer.

Under the common QA protocol above, we use EVA scores to fill any shortfall in A.I.R.'s selections to 32 frames. We label this the “A.I.R. + EVA fill variant” to distinguish it from the original adaptive frame-count configuration. MLVU uses the macro-average over seven task categories. Accuracy is reported in percent.

AnswererMethodVideo-MMELongVideoBenchMLVU
Qwen2.5-VL-7BA.I.R. + EVA fill variant63.363.170.6
Qwen2.5-VL-7BE-DOT (8)65.464.872.1
InternVL3-8BA.I.R. + EVA fill variant66.162.274.3
InternVL3-8BE-DOT (8)68.162.674.7

A.I.R.'s public implementation and configuration differ from the paper's description, so our runs do not exactly reproduce its setting. The adapted variant's scores do not establish the best performance the original method could achieve.

AuthorsOpenReview ↗

Response to Reviewer qpUS (3/5)

W3 & Q1: Clarify baseline sources, discrepancies, implementations, candidate pools, and query text.

A3: Thank you for checking the baseline results. The affected MDP³ entries in Table 1 and AKS entries in Table 7 retained published values compiled in the WFS-SB paper without clear source labels. We apologize and will replace them with our reruns, updating the associated averages, comparisons, and reused entries in Table 5.

1. The corrected selector comparisons use our reruns. This covers Uniform, MDP³, FOCUS, and WFS-SB in Table 1, and AKS and BOLT in Table 7. The three starred foundation-model rows retain their original papers' results and protocols as background references.

MDP³:

AnswererVideo-MMELongVideoBenchMLVU
Qwen2.5-VL-7B63.3 (63.8)60.4 (60.0)66.7 (66.2)
InternVL3-8B66.4 (66.8)60.9 (60.9)73.9 (74.0)
LLaVA-OneVision-7B58.5 (-)59.0 (-)68.2 (-)

AKS:

AnswererVideo-MMELongVideoBenchMLVU
Qwen2.5-VL-7B63.1 (64.0)63.6 (63.2)66.0 (67.2)
InternVL3-8B66.2 (66.3)61.9 (61.5)74.3 (74.2)
LLaVA-OneVision-7B58.4 (-)59.3 (-)67.5 (-)

Accuracy (%) at K=32K=32: our reruns, with WFS-SB's reported values in parentheses. A dash means no corresponding 32-frame result.

2. For the main-table QA comparisons, all selectors share the evaluation protocol and shortfall preprocessing (Q2), while retaining their native candidate construction and selection algorithms. We use official implementations and environment configurations, fixing the final answerer budget at K=32K=32. For query-dependent selectors, inputs consist of the question and all candidate choices, with no correct-answer label provided. “≈1 FPS” refers to official candidate sampling, and “Q + choices” denotes our selection setup using the question and all answer choices.

MethodWFS-SBAKSFOCUSBOLTMDP³GIFTAIRE-DOT
≈1 FPS✓✓✗✓✗✗✗✓
Q + choices✓✓*✓*✓*✓✓✓✓

* These inspected paths originally use question-only queries; our reruns add all choices. For AKS, this refers to BLIP/CLIP scoring.

FOCUS uses adaptive coarse-to-fine sampling; MDP³ uses a fixed number of uniform candidates; GIFT limits candidates by configuration; AIR starts at 2 or 3 FPS before adaptive filtering. Candidate-pool size differs from the final 32-frame budget.

3. Matching Uniform accuracy does not imply identical selected frames. Uniform bypasses relevance scoring. Adaptive selection depends on the similarity curve, which can change with scoring batch size and numerical settings. Main-table comparisons retain each method's relevance model; Table 5 additionally compares E-DOT and WFS-SB with the same BLIP-2 model.

We compare our WFS-SB rerun's complete 32-frame selections with the frame indices reported by WFS-SB:

DatasetSamplesExact matchesSamples with differencesMean differing frames¹Frame overlap²
Video-MME270016511049 (38.85%)0.74 / 3297.69%
LongVideoBench1337784553 (41.36%)1.07 / 3296.66%
MLVU217411511023 (47.06%)2.57 / 3291.97%

¹ Differing frames = 32 minus shared indices. ² Overlap = shared indices / 32. Both average over all questions, including exact matches. High overlap can coexist with different selections on many questions.

4. Numerical diagnostics show scoring sensitivity. With fixed inputs and weights, changing batch size altered Top-32 sets in 4/30 videos under default FP16, replacing one frame each time; FP32 with TF32 off preserved the sets in this sample. Deterministic settings did not remove cross-batch differences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.