- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces E-DoT, a training-free frame selector for long-video understanding. E-DoT ranks frames independently, it models how selected frames can represent discarded observations. Frames are described by time, query relevance, and sparse local-helpfulness estimates. A directed replacement cost penalizes losses in these evidence signals rather than just selecting the frames, while a coverage objective selects complementary representatives under a fixed frame budget.
Critical strengths
-
The replacement formulation is well motivated, for long videos where independently selecting relevant frames can produce redundant observations due to the fixed number of selection. Distinguishing the importance of an observation from how well another frame represents it gives the method a clear rationale. Directionality is also an intuitive decision a frame exposing additional answer evidence may replace a weaker previous observation more safely than the reverse.
-
The results are promising and well presented, with relevant ablations supporting the design. The comparisons isolate directed versus symmetric costs and coverage updates versus one-shot selection. The observer-free results and controls using the same observer information also help establish that the gains are not solely due to additional VLM calls. Evaluation across multiple benchmarks and answerers strengthens the practical contribution.
-
The authors establish the transport reduction and greedy guarantee, while explicitly showing that low transport loss does not guarantee preservation of answer-critical evidence.
Critical weaknesses
-
Although, the replacement relation is defined through scalar relevance scores, which do not establish that one frame preserves another’s evidence. However, close frames can have similar scores while exposing different facts specially for fast movement of fine grained motion that may occur in long video. Additionally, helpfulness is measured for three-frame neighborhoods and interpolated onto individual candidates. The optimization guarantees therefore apply to coverage of these proxy coordinates, rather than semantic evidence preservation. This does not negate the empirical gains, but limits the interpretation of the replacement claim.
-
The full computational tradeoff remains unclear. Fixing the answerer at 32 frames does not equalize total inference cost because the full method adds up to eight three-frame observer calls. The observer-free variant and matched controls are useful, but the reported selection-stage timings do not establish end-to-end latency.
-
The replacement cost assumes normalized time, relevance, and helpfulness are directly comparable. However, scaling them to [0,1] does not guarantee this, and the appendix notes that rescaling affects the objective. A sensitivity analysis could test how robust this is.
Questions for authors
1- Beyond aggregate QA accuracy, what supports the assumption that a nearby frame with higher relevance and helpfulness can preserve the source frame’s answer-bearing information?
2- What is the end-to-end cost of the full configuration relative to the observer-free variant and the strongest baseline? It would be helpful if the cost is broken down to relevance scoring, observer calls, selection etc...
6 comments
Response to Reviewer Lh3p (2/3)
W3: Normalization to [0,1] does not make time, relevance, and helpfulness semantically comparable. Test sensitivity to their relative scales.
A3: Thank you for raising this concern. We agree that normalizing time, relevance, and helpfulness to only aligns their numerical ranges. Equal numerical differences can still have different meanings across the three components, and rescaling a component can change the objective. We therefore add a component-weight sensitivity analysis to examine how changes in relative scale affect the quality of the selected frames.
1. We halve or double one component's weight while keeping the others fixed. With and , we vary the weights of the three cost components from the default . We halve or double one weight at a time and leave the other two at 1. For each configuration, we divide all weighted costs by the common upper bound :
=\frac{1}{U}\max\\!\left\\{ \lambda_t|t_i-t_j|, \lambda_r[r_i-r_j]_+, \lambda_h[h_i-h_j]_+ \right\\}.$$ This uniform scaling keeps costs in $[0,1]$ while preserving the relative component weights and the minimizers of the weighted objective. We therefore retain the greedy initialization $d_i=1$. Source masses, candidates, evidence scores, and all other settings are held fixed, and all configurations use the same evaluation samples. BSR measures coverage of answer-relevant scenes, and BDS measures the distribution of sampled frames across scenes. Both metrics are reported on a 0–100 scale. | Weights (time, relevance, helpfulness) | Video-MME BSR | Video-MME BDS | LongVideoBench BSR | LongVideoBench BDS | |:---:|:---:|:---:|:---:|:---:| | (1, 1, 1) | 80.4 | 84.5 | 72.8 | 77.7 | | (0.5, 1, 1) | 80.0 | 84.1 | 73.4 | 78.4 | | (2, 1, 1) | 80.4 | 84.4 | 73.1 | 77.8 | | (1, 0.5, 1) | 80.5 | 84.2 | 73.6 | 78.6 | | (1, 2, 1) | 80.2 | 84.3 | 73.5 | 78.2 | | (1, 1, 0.5) | 80.8 | 84.9 | 72.5 | 78.2 | | (1, 1, 2) | 80.0 | 84.5 | 73.5 | 78.4 | **2. Both metrics remain relatively stable within the tested range.** Based on the displayed values, the largest absolute BSR changes from the default are 0.4 points on Video-MME and 0.8 points on LongVideoBench. The corresponding maximum BDS changes are 0.4 and 0.9 points. These results show that answer-relevant scene coverage and the distribution of sampled frames across scenes remain relatively stable when a single component's weight is halved or doubled, providing empirical support for robustness to scale changes within this range. For the CLIP and SigLIP score mappings used in Table 5, see our response to Reviewer qpUS, Q3.Response to Reviewer Lh3p (3/3)
Q1: Beyond QA accuracy, what supports the assumption that nearby frames with higher relevance and helpfulness preserve the source frame's answer cues?
A4: Thank you for highlighting this important distinction. The matched ablations in Table 2(b) support the contributions of the directed cost and coverage updates to QA performance. The scene-coverage evaluation in Appendix G.1 and the visualizations in Appendix I further support the value of the final selected frames as answer evidence. However, these results do not directly test whether a representative preserves specific answer cues from a discarded source frame. We therefore add two evaluations of actual source-to-representative assignments, examining retention of evidence intervals and specific visible cues.
We compare each actual representative with a control matched for temporal distance from the same source. Both evaluations use the final selected frames with and . Following Eq. (9), we assign each source frame to its minimum-cost representative in this set. We include only assignments where the source is not selected, the representative is within 4 seconds of the source, and neither relevance nor helpfulness decreases. These conditions apply only to sample selection for the supplementary evaluation.
1. We check whether the representative stays within the source frame's annotated answer-evidence interval. For each benchmark, we sample questions from 100 distinct videos and use the existing answer-relevant scene annotations. We evaluate source frames inside an annotated interval with positive source weight, . Each control comes from the same question and is matched for temporal distance, without filtering on relevance or helpfulness. Scene annotations are used only for evaluation.
Pairs: eligible assignments; Videos: included/sampled. Retention is reported in %, and differences in percentage points.
| Dataset | Pairs | Videos | Representative | Control | Difference [95% CI] |
|---|---|---|---|---|---|
| Video-MME | 1,937 | 84 / 100 | 87.2% | 68.5% | +18.7 [9.1, 28.5] |
| LongVideoBench | 1,099 | 77 / 100 | 70.4% | 58.4% | +12.0 [1.7, 22.4] |
We first calculate retention rates within each video, then average them with equal weight per video. Confidence intervals for the differences use a paired bootstrap at the video level. With equal weight per video, actual representatives more often stay within the source frame's evidence interval on both benchmarks. When frame pairs receive equal weight instead, the differences become smaller and their confidence intervals include zero. Interval retention tests whether a representative stays within the same answer-relevant segment; the next evaluation examines retention of specific visible cues.
2. We check whether a visible answer cue fixed in advance from the source frame is fully retained. We separately sample 48 videos, 24 per benchmark, with no overlap with the first evaluation. Using only the source frame, question, and reference answer, we first fix one task-relevant cue that is actually visible in the source. We then check whether the representative and control each retain it in full. For example, if the source contains readable text naming a location, we check whether that text is still readable in each target frame.
Controls are again matched for temporal distance. Here, we also require at least one of their relevance or helpfulness scores to be at least 0.1 below the source frame's corresponding score, to test local candidates with a clear score difference. Reference answers are used only for evaluation, and cue-retention judgments are confirmed by two evaluators. Of the 48 source frames, 32 contain a cue that can be evaluated; the other 16 samples are recorded separately. Partial retention does not count as full retention.
Videos: evaluable/sampled. Entries show full cue retention by representatives and lower-score controls, as counts and percentages.
| Dataset | Videos | Representative | Control |
|---|---|---|---|
| Video-MME | 16 / 24 | 15/16 (93.8%) | 7/16 (43.8%) |
| LongVideoBench | 16 / 24 | 14/16 (87.5%) | 8/16 (50.0%) |
| Combined | 32 / 48 | 29/32 (90.6%) | 15/32 (46.9%) |
The two evaluators' review shows that actual representatives more often fully retain the fixed visible task cue in samples with an evaluable cue. Together with the interval evaluation using equal weight per video, these results provide empirical support beyond aggregate QA accuracy for using nearby frames as representatives when neither relevance nor helpfulness decreases. This conclusion is limited to the stated filters and evaluation procedures; it does not guarantee preservation of all answer-relevant information from arbitrary source frames.
Response to Reviewer Lh3p (1/3)
Thank you for your careful review. We respond to your concerns below.
W1: Similar relevance scores and interpolated neighborhood helpfulness do not ensure that nearby frames preserve the same evidence, especially during brief changes. The guarantees concern proxy coverage rather than semantic preservation.
A1: Thank you for recognizing the motivation behind modeling frame replacement. Allowing a nearby frame to serve as a low-cost representative when neither relevance nor helpfulness decreases is a deliberate design choice.
1. The replacement cost uses task-related semantic signals. Relevance measures how well the visual content matches the question, while helpfulness measures how much the three-frame context supports answering it. These signals therefore capture some task-related semantic information. For example, a later frame within the same event may show the required text or action more clearly, and we want to allow this improvement. Eq. (5) retains temporal distance and uses one-sided penalties for decreases in either evidence score. Taking the maximum prevents a loss in one component from being masked by the others (Lines 212–223).
2. This content-conditioned proxy does not fully determine whether one frame preserves another's semantic evidence. Different facts can receive similar scores, and neighborhood helpfulness and its interpolation may miss brief changes. This is a tradeoff between the cost of sparse observations and how precisely they describe the evidence. The paper explicitly discusses this tradeoff in Section 3.1 and Appendix H. Eq. (19) characterizes the worst-case one-sided proxy loss between two frames; Theorem 1 / Eq. (20) extends this characterization to the selected set. Both statements concern the specified class of monotone Lipschitz functions of these proxy coordinates and do not guarantee complete preservation of semantic evidence (Appendix B, Lines 819–821). We will further clarify this meaning of “replacement.” For direct evidence-retention tests, see our response to Q1 below.
W2 & Q2: A fixed 32-frame answerer budget does not equalize total inference cost. Report end-to-end latency for B=8, B=0, and WFS-SB, with a breakdown of the main stages.
A2: Fixing the answerer input at 32 frames does not equalize total inference cost. We therefore add end-to-end measurements.
1. We measure the end-to-end runtime of the full pipeline. We use Video-MME without subtitles: 100 videos per duration group (short, medium, and long), with two questions per video, totaling 600 questions. The answerer is Qwen2.5-VL-7B, and the observer is Qwen2.5-VL-3B. Models are preloaded and warmed up. We run the complete pipeline three times for each question, including video reading, relevance scoring, observation, selection, and answering, without reusing cached per-sample results. All methods and repetitions for a given question run on the same single NVIDIA A100 GPU.
Times are means in s/question. E-DOT parentheses give B; extra time is E-DOT (8) minus the row latency.
2. Separate diagnostic runs measure the time spent in each stage. Mean times for individual stages are shown below, in seconds per question.
We average end-to-end latency over three complete runs per question; separate diagnostic runs identify the time spent in each stage. They are not an exact decomposition of the end-to-end runs and therefore need not sum to the reported means.
These full-method timings use different relevance models: BLIP for E-DOT and BLIP-2 for WFS-SB. Similar candidate FPS therefore does not imply equal scoring time.
The observer takes 2.148 seconds per question, including its own video reading. Of this, approximately 0.710 seconds is video reading and is already included in the decoding row. Adding it again would count the same video-reading time twice. Actual execution records show an average of 7.71 observer calls and 23.11 image inputs per question at B=8.
The full configuration adds 2.329 seconds per question over the observer-free variant and 3.271 seconds per question over WFS-SB. The stage-level diagnostics show that video reading and relevance scoring account for most of the runtime. Frame-selection time alone therefore does not represent the end-to-end cost of the method. For latency by video duration, see our response to Reviewer PQ5o, W2 & Q2.