- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SPAN, a method for efficiently selecting a limited number of frames to provide to a VLM for Video QA. The model is trained on a large-scale QA dataset without temporal annotations. It learns to combine representations of short temporal segments through weighted summation to approximate the representation of a longer segment. This enables representations of longer segments to be constructed from short segments without additional model forward passes for each longer segment. Given the resulting pool of segments at different temporal scales, SPAN uses tree-structured knapsack optimization to allocate the frame budget and select frames. The proposed method is evaluated on four Video QA benchmarks and two VLM families. SPAN reports an average improvement of 5.47 points over uniform sampling and outperforms existing methods that do not use temporal annotations in 18 of the 20 model–benchmark combinations under the eight-frame setting.
Critical strengths
-
Training using only large-scale Video QA data. Temporal grounding annotations require substantial human effort and are costly to obtain at scale. The paper instead learns from more readily available Video QA data without temporal annotations. This enables training on approximately 200,000 examples while keeping the training objective closely aligned with the downstream Video QA task.
-
Efficient segment composition through self-distillation. Frame selection involves a combinatorial search space, and evaluating candidate subsets with large models can be prohibitively expensive. The proposed self-distillation objective learns to combine short-clip representations to reconstruct the full video's predicted representation. This allows longer segments to be represented and scored without a separate model forward pass for every candidate temporal window.
-
Strong performance under tight frame budgets. The experimental results show that SPAN can achieve comparable or better performance using only eight frames than uniform sampling with substantially more frames. Since the self-attention cost grows quadratically with the number of input tokens, these results suggest that SPAN can reduce the downstream VLM's computational cost while maintaining answer accuracy.
Critical weaknesses
-
Lack of direct evaluation of learned segment relevance The paper learns segment relevance from Video QA data without temporal annotations. The authors explicitly state that they validate segment relevance only through downstream VLM accuracy and leave direct validation using CG-Bench’s clue-level grounding annotations to future work (page 9, lines 481–482). Thus, direct experimental evidence of how closely the learned relevance aligns with the actual evidence intervals remains limited. Directly evaluating segment relevance and selected frames against these annotations would provide stronger support for the proposed learning approach.
-
Insufficient ablation studies of the core learning approach The paper uses self-distillation to make a weighted combination of short-clip representations reconstruct the full video’s predicted representation. However, no comparison without self-distillation is reported, making it difficult to isolate the contribution of this objective. Comparing the proposed approach with an alternative that directly aligns the composed representation with the ground-truth answer embedding would also help validate the choice of distillation target.
-
Limited ability of the composition operator to capture temporal order across clips The proposed composition operator uses a weighted sum of base-clip representations. When these representations are held fixed, permuting the clips produces the same composed representation. For example, pouring water and then drinking it differs temporally from drinking and then pouring water, yet the composition operator alone cannot distinguish these sequences. Beyond temporal information within individual base clips, the paper should analyze how this limitation affects performance on questions requiring temporal ordering across multiple clips.
Questions for authors
Q1. Additional validation of the learning approach Could you provide comparisons against settings without self-distillation and with the composed clip representation aligned directly to the ground-truth answer text embedding? These comparisons are important for determining whether the performance gains stem from the proposed self-distillation objective.
Q2. Clarification on temporal order Do the base-clip representations or relevance computations incorporate absolute clip positions within the video or ordering information across clips? If not, is there an analysis showing how the composition operator’s permutation invariance affects performance on questions requiring temporal ordering across multiple clips?
Q3. Additional validation of segment relevance Could you provide results evaluating how closely SPAN’s segment relevance scores and selected frames align with the actual evidence intervals? If possible, please distinguish the evaluation of SPAN’s relevance scores alone from that of the final frame selection incorporating CLIP scores and knapsack search.
6 comments
(2/2) Author Response to Reviewer RRRs
W3, Q2. (Limited ability of the composition operator to capture temporal order across clips ) We agree with the reviewer. Compose is a query-weighted sum and is therefore invariant to the order of clips within a segment. Temporal order is captured only within each jointly encoded 8-frame base clip and by the downstream VLM, which receives the selected frames in chronological order.
The question-type breakdown on MLVU (Qwen3-VL-8B, K = 8) reflects this limitation:
| Question type | # Questions | Uniform (%) | SPAN (%) | Δ (pts) |
|---|---|---|---|---|
| Counting Problem | 271 | 24.35 | 44.65 | +20.30 |
| Object Recognition | 138 | 51.45 | 69.57 | +18.12 |
| Attribute Perception | 333 | 53.75 | 71.17 | +17.42 |
| Action Recognition | 231 | 59.74 | 71.00 | +11.26 |
| Emotion & Social Perception | 43 | 60.47 | 69.77 | +9.30 |
| OCR Problems | 14 | 42.86 | 50.00 | +7.14 |
| Object Reasoning | 31 | 67.74 | 74.19 | +6.45 |
| Spatial Perception | 235 | 60.43 | 66.81 | +6.38 |
| Information Synopsis | 179 | 79.33 | 84.36 | +5.03 |
| Temporal Perception | 431 | 55.68 | 58.93 | +3.25 |
| Action Reasoning | 120 | 66.67 | 67.50 | +0.83 |
| Spatial Reasoning | 5 | 80.00 | 80.00 | 0.00 |
| Temporal Reasoning | 89 | 69.66 | 66.29 | −3.37 |
| Miscellaneous | 54 | 62.96 | 70.37 | +7.41 |
| Overall | 2174 | 55.70 | 65.41 | +9.71 |
SPAN improves on uniform sampling in 12 of 14 question types, with the largest gains (11–20 points) on recognition, attribute, and counting questions. Gains are smallest on temporally oriented types. Temporal Reasoning, which covers before/after and cause-effect questions, is the only type below uniform, by three questions out of 89.
We will add this analysis to the manuscript, extending it to other frame budgets and VLM backbones. Making composition order-aware, for example by adding encoding of the temporal position of clips, is left for future work.
Additional temporal grounding evaluation on LV-Haystack
To complement the temporal grounding on CG-Bench, we also evaluate on LV-Haystack (Ye Jinhui et al., 2025), which annotates keyframes containing the evidence needed to answer the question. We use its Haystack-LVBench subset. For each method, we take the top-K ranked frames, with K in {8, 32}.
Evaluation Metrics: Following T*’s setting, a predicted frame counts as correct if it lies within 5 seconds of any reference keyframe, and a reference keyframe counts as found if any predicted frame lies within 5 seconds of it. We report temporal precision (the fraction of predicted frames that are correct), recall (the fraction of reference keyframes found), and F1, each computed per question and averaged over questions. Since most questions have only one or two reference keyframes, precision falls as K grows, so recall is the more informative measure at larger K.
| Method | precision (K=8) | precision (K=32) | recall (K=8) | recall (K=32) | F1 (K=8) | F1 (K=32) |
|---|---|---|---|---|---|---|
| Uses grounding supervision | ||||||
| VideoITG | 11.10 | 6.25 | 24.11 | 40.58 | 14.02 | 10.27 |
| No grounding supervision | ||||||
| AKS | 3.24 | 1.80 | 16.10 | 32.84 | 5.16 | 3.35 |
| BOLT | 2.18 | 1.98 | 10.67 | 34.17 | 3.48 | 3.67 |
| SPAN (ours) | 6.31 | 3.48 | 16.99 | 36.42 | 8.22 | 6.04 |
Here as well, we observe that SPAN outperforms the best annotation-free baseline. Its F1 is 3.06-4.74 points better at the K=8 budget. At a higher budget of K=32, its recall is 2.25-3.58 points higher. Furthermore, it reaches 90% of VideoITG’s recall at K=32.
Ye, Jinhui, et al., T*: Re-thinking temporal search for long-form video understanding, 2025
(1/2) Author Response to Reviewer RRRs
W1, Q3. (Lack of direct evaluation of learned segment relevance) This is the evaluation we deferred to future work, as noted at lines 481–482. We have now run it and will replace that limitation statement with the results below. All methods rank the same set of candidate frames, and each method's top-K frames, with K as in Tables 1–2, form its prediction. Temporal IoU is computed between the predicted and ground-truth frame sets. Following the evaluation setting (Chen et al., 2025), we report mIoU and rec@IoU, with the latter averaged over IoU thresholds from 0.1 to 0.5. CG-Bench accuracy, averaged across the five VLM backbones, is computed from Tables 1, 2, 6, and 7. For scale, we also report mIoU and rec@IoU of 5.73 and 8.12 for GPT-4o, and 3.64 and 5.11 for Qwen2-VL-72B. These models predict interval timestamps rather than top-K frames, so they are reference points, not direct comparisons.
Note that a set of K frames is penalised for every frame outside the clue, and its IoU with a clue of different length is bounded above, so scores need not increase with K.
We can observe the following -
We will add this evaluation to the main paper. Furthermore, the gain achieved by SPAN due to the learned segment-level signal is also observed in Figure 2 (Right), which varies λsegment on EgoSchema while the rest of the pipeline is held fixed. At λsegment = 0, no segment score is used, and selection reduces to top-K frames by CLIP. This is the weakest setting, and at K = 8 and K = 16 it falls below uniform sampling. Introducing SPAN’s segment relevance improves accuracy across all K=8, 16, 32.
W2, Q1. (Missing ablation on self-distillation) In SPAN, the relevance scorer's parameters enter the objective only through the composition weights in Eqs. 6–8. The prediction losses L_L1 and L_InfoNCE act on the full-video representation and do not provide gradients to the scorer. Setting λself−distill=0 therefore leaves the scorer untrained, so self-distillation is the sole source of learned relevance. The appropriate comparison is against an untrained scorer, which we report below using the K×1 setting of Table 5 (MLVU, Qwen3-VL-8B, K = 8, 32), where frames are selected via flat top-Kbase clips (one frame each), isolating the quality of the relevance signal itself.
The learned scorer improves on uniform sampling by 3.73-4.14 points in this setting, whereas removing self-distillation yields the weakest performance, with little to no improvement over uniform sampling.