acceptodds
Under review as a conference paper at ICLR 2027

Segments Divide, Frames Decide: Composable Relevance for Long Videos from QA Alone

Abstract

A vision-language model (VLM) sees a long video only through a handful of frames, so the choice of frames determines what it can answer. Queries differ in the temporal scale of the evidence they need, yet existing relevance signals operate at a single, fixed scale. Selectors that require no temporal grounding annotations score each frame in isolation, missing motion and event structure. Supervised grounding methods capture this structure but require such annotations and expensive model passes. This paper introduces SPAN: Segment relevance via Propagated, Annotation-free, Nested composition, which scores relevance at multiple temporal scales without grounding labels, using a query-conditioned composition operator that propagates relevance from base clips to arbitrarily coarse segments. Its weighting and composition functions read only the defined base-clip representations and the query to construct segments and obtain their relevance, so the same operator applies unchanged across scales. Self-distillation learns this operator from video-question-answer tuples alone: composed clips must reconstruct the video's own representation. Finally, a nested, tree-structured knapsack allocates SPAN's frame budget across the segment hierarchy. Across four long-video benchmarks and two VLM families, SPAN improves on uniform sampling by an average of 5.47 points and beats the strongest annotation-free baseline in 18/20 settings. It recovers over 95.0% of the performance achieved by a selector that runs a VLM during selection, without running one itself. For long-video question answering, this makes query-aware frame selection a lightweight plug-in step, thereby letting any frozen VLM answer from relevant evidence without temporal grounding annotations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VideoQA, Unsupervised Temporal Grounding, VLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RRRsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPAN, a method for efficiently selecting a limited number of frames to provide to a VLM for Video QA. The model is trained on a large-scale QA dataset without temporal annotations. It learns to combine representations of short temporal segments through weighted summation to approximate the representation of a longer segment. This enables representations of longer segments to be constructed from short segments without additional model forward passes for each longer segment. Given the resulting pool of segments at different temporal scales, SPAN uses tree-structured knapsack optimization to allocate the frame budget and select frames. The proposed method is evaluated on four Video QA benchmarks and two VLM families. SPAN reports an average improvement of 5.47 points over uniform sampling and outperforms existing methods that do not use temporal annotations in 18 of the 20 model–benchmark combinations under the eight-frame setting.

Critical strengths
  1. Training using only large-scale Video QA data. Temporal grounding annotations require substantial human effort and are costly to obtain at scale. The paper instead learns from more readily available Video QA data without temporal annotations. This enables training on approximately 200,000 examples while keeping the training objective closely aligned with the downstream Video QA task.

  2. Efficient segment composition through self-distillation. Frame selection involves a combinatorial search space, and evaluating candidate subsets with large models can be prohibitively expensive. The proposed self-distillation objective learns to combine short-clip representations to reconstruct the full video's predicted representation. This allows longer segments to be represented and scored without a separate model forward pass for every candidate temporal window.

  3. Strong performance under tight frame budgets. The experimental results show that SPAN can achieve comparable or better performance using only eight frames than uniform sampling with substantially more frames. Since the self-attention cost grows quadratically with the number of input tokens, these results suggest that SPAN can reduce the downstream VLM's computational cost while maintaining answer accuracy.

Critical weaknesses
  1. Lack of direct evaluation of learned segment relevance The paper learns segment relevance from Video QA data without temporal annotations. The authors explicitly state that they validate segment relevance only through downstream VLM accuracy and leave direct validation using CG-Bench’s clue-level grounding annotations to future work (page 9, lines 481–482). Thus, direct experimental evidence of how closely the learned relevance aligns with the actual evidence intervals remains limited. Directly evaluating segment relevance and selected frames against these annotations would provide stronger support for the proposed learning approach.

  2. Insufficient ablation studies of the core learning approach The paper uses self-distillation to make a weighted combination of short-clip representations reconstruct the full video’s predicted representation. However, no comparison without self-distillation is reported, making it difficult to isolate the contribution of this objective. Comparing the proposed approach with an alternative that directly aligns the composed representation with the ground-truth answer embedding would also help validate the choice of distillation target.

  3. Limited ability of the composition operator to capture temporal order across clips The proposed composition operator uses a weighted sum of base-clip representations. When these representations are held fixed, permuting the clips produces the same composed representation. For example, pouring water and then drinking it differs temporally from drinking and then pouring water, yet the composition operator alone cannot distinguish these sequences. Beyond temporal information within individual base clips, the paper should analyze how this limitation affects performance on questions requiring temporal ordering across multiple clips.

Questions for authors

Q1. Additional validation of the learning approach Could you provide comparisons against settings without self-distillation and with the composed clip representation aligned directly to the ground-truth answer text embedding? These comparisons are important for determining whether the performance gains stem from the proposed self-distillation objective.

Q2. Clarification on temporal order Do the base-clip representations or relevance computations incorporate absolute clip positions within the video or ordering information across clips? If not, is there an analysis showing how the composition operator’s permutation invariance affects performance on questions requiring temporal ordering across multiple clips?

Q3. Additional validation of segment relevance Could you provide results evaluating how closely SPAN’s segment relevance scores and selected frames align with the actual evidence intervals? If possible, please distinguish the evaluation of SPAN’s relevance scores alone from that of the final frame selection incorporating CLIP scores and knapsack search.

6 comments
AuthorsOpenReview ↗

(1/2) Author Response to Reviewer RRRs

W1, Q3. (Lack of direct evaluation of learned segment relevance) This is the evaluation we deferred to future work, as noted at lines 481–482. We have now run it and will replace that limitation statement with the results below. All methods rank the same set of candidate frames, and each method's top-K frames, with K as in Tables 1–2, form its prediction. Temporal IoU is computed between the predicted and ground-truth frame sets. Following the evaluation setting (Chen et al., 2025), we report mIoU and rec@IoU, with the latter averaged over IoU thresholds from 0.1 to 0.5. CG-Bench accuracy, averaged across the five VLM backbones, is computed from Tables 1, 2, 6, and 7. For scale, we also report mIoU and rec@IoU of 5.73 and 8.12 for GPT-4o, and 3.64 and 5.11 for Qwen2-VL-72B. These models predict interval timestamps rather than top-K frames, so they are reference points, not direct comparisons.

MethodmIoU (K=8)mIoU (K=32)rec@IoU (K=8)rec@IoU (K=32)Avg. QA acc. (K=8)Avg. QA acc. (K=32)
Uses grounding supervision
VideoITG4.373.754.753.0535.9136.91
No grounding supervision
AKS0.980.780.190.0031.7532.87
BOLT0.520.990.070.0731.2033.49
SPAN (ours)3.233.403.673.0834.7635.84

Note that a set of K frames is penalised for every frame outside the clue, and its IoU with a clue of different length is bounded above, so scores need not increase with K.

We can observe the following -

  • SPAN grounds better than the annotation-free baselines. Its mIoU is more than three times that of the stronger baseline at both budgets, and its rec@IoU is 3.08–3.67 against at most 0.19.
  • SPAN recovers most of the grounding of a supervised selector. Without temporal labels and without using a VLM in the selection loop, it reaches 74% (K = 8) and 91% (K = 32) of VideoITG's mIoU, and matches its rec@IoU at K = 32.
  • Grounding and QA accuracy agree. The ordering SPAN > AKS/BOLT is the same for grounding and for CG-Bench accuracy, and SPAN outperforms AKS and BOLT on CG-Bench for all five backbones at both budgets (Tables 1 and 6).

We will add this evaluation to the main paper. Furthermore, the gain achieved by SPAN due to the learned segment-level signal is also observed in Figure 2 (Right), which varies λsegment\lambda_{segment} on EgoSchema while the rest of the pipeline is held fixed. At λsegment\lambda_{segment} = 0, no segment score is used, and selection reduces to top-K frames by CLIP. This is the weakest setting, and at K = 8 and K = 16 it falls below uniform sampling. Introducing SPAN’s segment relevance improves accuracy across all K=8, 16, 32.


W2, Q1. (Missing ablation on self-distillation) In SPAN, the relevance scorer's parameters enter the objective only through the composition weights in Eqs. 6–8. The prediction losses L_L1 and L_InfoNCE act on the full-video representation and do not provide gradients to the scorer. Setting λself−distill=0\lambda_{self-distill} = 0 therefore leaves the scorer untrained, so self-distillation is the sole source of learned relevance. The appropriate comparison is against an untrained scorer, which we report below using the K×1 setting of Table 5 (MLVU, Qwen3-VL-8B, K = 8, 32), where frames are selected via flat top-Kbase clips (one frame each), isolating the quality of the relevance signal itself.

Relevance scorerK = 8K = 32
Uniform sampling55.7060.49
Untrained scorer (λself−distill=0\lambda_{self-distill} = 0)55.1560.63
SPAN (self-distillation)59.4364.63

The learned scorer improves on uniform sampling by 3.73-4.14 points in this setting, whereas removing self-distillation yields the weakest performance, with little to no improvement over uniform sampling.

AuthorsOpenReview ↗

(2/2) Author Response to Reviewer RRRs

W3, Q2. (Limited ability of the composition operator to capture temporal order across clips ) We agree with the reviewer. Compose is a query-weighted sum and is therefore invariant to the order of clips within a segment. Temporal order is captured only within each jointly encoded 8-frame base clip and by the downstream VLM, which receives the selected frames in chronological order.

The question-type breakdown on MLVU (Qwen3-VL-8B, K = 8) reflects this limitation:

Question type# QuestionsUniform (%)SPAN (%)Δ (pts)
Counting Problem27124.3544.65+20.30
Object Recognition13851.4569.57+18.12
Attribute Perception33353.7571.17+17.42
Action Recognition23159.7471.00+11.26
Emotion & Social Perception4360.4769.77+9.30
OCR Problems1442.8650.00+7.14
Object Reasoning3167.7474.19+6.45
Spatial Perception23560.4366.81+6.38
Information Synopsis17979.3384.36+5.03
Temporal Perception43155.6858.93+3.25
Action Reasoning12066.6767.50+0.83
Spatial Reasoning580.0080.000.00
Temporal Reasoning8969.6666.29−3.37
Miscellaneous5462.9670.37+7.41
Overall217455.7065.41+9.71

SPAN improves on uniform sampling in 12 of 14 question types, with the largest gains (11–20 points) on recognition, attribute, and counting questions. Gains are smallest on temporally oriented types. Temporal Reasoning, which covers before/after and cause-effect questions, is the only type below uniform, by three questions out of 89.

We will add this analysis to the manuscript, extending it to other frame budgets and VLM backbones. Making composition order-aware, for example by adding encoding of the temporal position of clips, is left for future work.

AuthorsOpenReview ↗

Additional temporal grounding evaluation on LV-Haystack

To complement the temporal grounding on CG-Bench, we also evaluate on LV-Haystack (Ye Jinhui et al., 2025), which annotates keyframes containing the evidence needed to answer the question. We use its Haystack-LVBench subset. For each method, we take the top-K ranked frames, with K in {8, 32}.

Evaluation Metrics: Following T*’s setting, a predicted frame counts as correct if it lies within 5 seconds of any reference keyframe, and a reference keyframe counts as found if any predicted frame lies within 5 seconds of it. We report temporal precision (the fraction of predicted frames that are correct), recall (the fraction of reference keyframes found), and F1, each computed per question and averaged over questions. Since most questions have only one or two reference keyframes, precision falls as K grows, so recall is the more informative measure at larger K.

Methodprecision (K=8)precision (K=32)recall (K=8)recall (K=32)F1 (K=8)F1 (K=32)
Uses grounding supervision
VideoITG11.106.2524.1140.5814.0210.27
No grounding supervision
AKS3.241.8016.1032.845.163.35
BOLT2.181.9810.6734.173.483.67
SPAN (ours)6.313.4816.9936.428.226.04

Here as well, we observe that SPAN outperforms the best annotation-free baseline. Its F1 is 3.06-4.74 points better at the K=8 budget. At a higher budget of K=32, its recall is 2.25-3.58 points higher. Furthermore, it reaches 90% of VideoITG’s recall at K=32.


Ye, Jinhui, et al., T*: Re-thinking temporal search for long-form video understanding, 2025

Reviewer_ijuzOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to learn query-conditioned multi-scale segment relevance from video-question-answer pairs without temporal grounding annotations. A self-distillation objective supports compositional segment representations, while a hierarchical knapsack algorithm allocates the frame budget. Experiments across four benchmarks and two VLM families show consistent improvements over annotation-free baselines.

Critical strengths

Learning reusable multi-scale segment relevance without temporal annotations is an interesting and potentially valuable contribution.

The evaluation is extensive, including multiple benchmarks, model scales, frame budgets, and comparisons with a temporally supervised selector.

Critical weaknesses

Insufficient validation of learned relevance. Segment relevance is evaluated primarily through downstream QA accuracy. The self-distillation objective does not directly establish that its scores identify the correct temporal evidence.

Unclear generalization and evaluation protocol. Training uses relatively short videos, while inference targets hour-long videos through tiled segment hierarchies. Furthermore, the frame/segment weights are tuned separately for each benchmark, raising concerns about validation-set dependence and generalization.

Questions for authors

Were the benchmark-specific hyperparameters selected using strictly held-out validation data, separate from the reported evaluation sets?

Can the learned segment relevance be evaluated directly using available temporal grounding annotations, rather than relying exclusively on downstream QA accuracy?

5 comments

W1, Q2. (Insufficient validation of learned relevance. ) We thank the reviewer for this point and have now evaluated SPAN directly against CG-Bench's ground-truth temporal grounding intervals. All methods rank the same set of candidate frames, and each method's top-K frames, with K as in Tables 1–2, form its prediction. Temporal IoU is computed between the predicted and ground-truth frame sets. Following the evaluation setting (Chen et al., 2025), we report mIoU and rec@IoU. CG-Bench accuracy, averaged across the five VLM backbones, is computed from Tables 1, 2, 6, and 7. For scale, we also report mIoU and rec@IoU of 5.73 and 8.12 for GPT-4o, and 3.64 and 5.11 for Qwen2-VL-72B. These models predict interval timestamps rather than top-K frames, so they are reference points, not direct comparisons.

MethodmIoU (K=8)mIoU (K=32)rec@IoU (K=8)rec@IoU (K=32)Avg. QA acc. (K=8)Avg. QA acc. (K=32)
Uses grounding supervision
VideoITG4.373.754.753.0535.9136.91
No grounding supervision
AKS0.980.780.190.0031.7532.87
BOLT0.520.990.070.0731.2033.49
SPAN (ours)3.233.403.673.0834.7635.84

We can observe the following -

  • SPAN grounds better than the annotation-free baselines. Its mIoU is more than three times that of the stronger baseline at both budgets, and its rec@IoU is 3.08–3.67 against at most 0.19.
  • SPAN recovers most of the grounding of a supervised selector. Without temporal labels and without using VLM in the selection loop, it reaches 74% (K = 8) and 91% (K = 32) of VideoITG's mIoU, and matches its rec@IoU at K = 32.
  • Grounding and QA accuracy agree. The ordering SPAN > AKS/BOLT is the same for grounding and for CG-Bench accuracy, and SPAN outperforms AKS and BOLT on CG-Bench for all five backbones at both budgets (Tables 1 and 6).

We will add this evaluation to the main paper. Furthermore, the gain achieved by SPAN due to the learned segment-level signal is also observed in Figure 2 (Right), which varies λsegment\lambda_{segment} on EgoSchema while the rest of the pipeline is held fixed. At λsegment\lambda_{segment} = 0, no segment score is used, and selection reduces to top-K frames by CLIP. This is the weakest setting, and at K = 8 and K = 16 it falls below uniform sampling. Introducing SPAN’s segment relevance improves accuracy across all K=8, 16, 32.

We agree that the self-distillation objective does not guarantee that composition weights localise evidence, but we empirically observe that relevance scores learned from QA pairs alone localise annotated evidence better than frame-level scoring and are close to those of a selector trained with grounding labels.


W2. (Unclear generalization) SPAN is currently trained on videos of at most 128 frames, and training on longer videos would let it learn coarser-scale relevance directly. We note this in our Limitations and leave it to future work. Three points nonetheless support the hour-scale results.

  • First, short-video training reflects the supervision available. LLaVA-Video-178K comprises videos up to 3 minutes. The supervised VideoITG is also trained and evaluated using the same protocol: trained on VideoITG-40K, built from LLaVA-Video, and evaluated on longer videos.
  • Second, the operator is never applied beyond its training scale. With D=4D = 4, a root segment spans 24×8=1282^4 \times 8 = 128 frames; longer videos are covered by tiling such segments, and only the parameter-free knapsack allocates budget across tiles.
  • Third, gains persist on long videos. On CG-Bench, where videos are at least 10 minutes long, SPAN is the best annotation-free method.

W2, Q1. (Unclear evaluation) The weight λsegment\lambda_{segment} is tuned via a grid search on a small held-out set for each benchmark and frame budget K. To test whether SPAN's gains depend on this tuning, we fixed a single global value, λsegment\lambda_{segment} = 0.75, across all benchmarks. Results for Qwen3-VL-8B at K = 8:

BenchmarkStrongest annotation-free baselineSPAN (tuned)SPAN (global λsegment\lambda_{segment} = 0.75)
MLVU61.6865.4164.95
EgoSchema59.261.461.40
LongVideoBench52.3653.4553.45
CG-Bench34.7737.2736.37
Average52.0054.3854.04

With the global value, SPAN loses an average of 0.34 points relative to the tuned setting. The conclusion does not hinge on the particular global value - average accuracies for λsegment\lambda_{segment} = 0.1, 0.25, 0.5, 0.9 are 52.82, 53.21, 53.84, and 53.16, respectively, and it still outperforms the best annotation-free baseline. This analysis supports the conclusion that tuning helps when a validation split is available, but SPAN does not depend on it. In its absence, a broad middle range for λsegment\lambda_{segment} works.

AuthorsOpenReview ↗

Additional temporal grounding evaluation on LV-Haystack

To complement the temporal grounding on CG-Bench, we also evaluate on LV-Haystack (Ye Jinhui et al., 2025), which annotates keyframes containing the evidence needed to answer the question. We use its Haystack-LVBench subset. For each method, we take the top-K ranked frames, with K in {8, 32}.

Evaluation Metrics: Following T*’s setting, a predicted frame counts as correct if it lies within 5 seconds of any reference keyframe, and a reference keyframe counts as found if any predicted frame lies within 5 seconds of it. We report temporal precision (the fraction of predicted frames that are correct), recall (the fraction of reference keyframes found), and F1, each computed per question and averaged over questions. Since most questions have only one or two reference keyframes, precision falls as K grows, so recall is the more informative measure at larger K.

Methodprecision (K=8)precision (K=32)recall (K=8)recall (K=32)F1 (K=8)F1 (K=32)
Uses grounding supervision
VideoITG11.106.2524.1140.5814.0210.27
No grounding supervision
AKS3.241.8016.1032.845.163.35
BOLT2.181.9810.6734.173.483.67
SPAN (ours)6.313.4816.9936.428.226.04

Here as well, we observe that SPAN outperforms the best annotation-free baseline. Its F1 is 3.06-4.74 points better at the K=8 budget. At a higher budget of K=32, its recall is 2.25-3.58 points higher. Furthermore, it reaches 90% of VideoITG’s recall at K=32.


Ye, Jinhui, et al., T*: Re-thinking temporal search for long-form video understanding, 2025

AuthorsOpenReview ↗

Following up on our response

Dear Reviewer ijuz, thank you for your feedback. As the discussion period is closing soon, we wanted to check whether our rebuttal addressed your concerns. We would be glad to clarify anything further.

Reviewer_BobDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SPAN, a query-conditioned frame selection method for long-video question answering that learns multi-scale temporal relevance without grounding annotations. SPAN uses self-distillation to learn clip relevance weights by reconstructing the full video's answer-predictive representation from weighted clip embeddings. A hierarchical knapsack algorithm then allocates a limited frame budget across relevant segments (Figs. 1, 3). Across four benchmarks and two VLM families, SPAN outperforms annotation-free baselines in most settings while approaching the performance of a temporally supervised method (Tables 1 and 2). The key takeaway is that learning segment relevance from QA supervision can improve frame selection efficiency without requiring explicit temporal grounding annotations.

Critical strengths
  1. Annotation-free relevance learning. SPAN learns query-conditioned segment relevance directly from video-QA supervision, eliminating the need for temporal grounding annotations.

  2. Good empirical results. SPAN outperforms the strongest annotation-free baseline in 18/20 settings (Table 1) and achieves over 95% of VideoITG's accuracy without temporal supervision (Table 2).

Critical weaknesses
  1. Lack of direct temporal-grounding evaluation. The paper's central contribution is learning meaningful segment relevance from QA supervision. However, the evaluation relies entirely on downstream QA accuracy (Tables 1 and 2), which does not establish whether the selected frames correspond to the actual evidence needed to answer a question. Existing benchmarks such as LV-Haystack from T* provide annotated reference keyframes for directly evaluating temporal search quality. Without such evaluations, it is difficult to assess whether SPAN genuinely learns accurate temporal relevance.

  2. Limited evaluation of complex long-form video reasoning. Although the paper evaluates videos extending to two hours (Table 1), it does not include HourVideo, which specifically evaluates long-form visual and temporal reasoning. Results on HourVideo would provide stronger evidence that the proposed method can handle complex reasoning across distant temporal segments.

=====

T* paper: Ye, Jinhui, et al. "Re-thinking temporal search for long-form video understanding." CVPR 2025

Chandrasegaran, Keshigeyan, et al. "Hourvideo: 1-hour video-language understanding." Advances in Neural Information Processing Systems 37 (2024): 53168-53197.

Questions for authors

Please see the weaknesses list.

I’m happy to increase my rating if both concerns are addressed.

====

Post Rebuttal

Thank you for the additional results. The new LV-Haystack and CG-Bench evaluations address my temporal-grounding concern, which was also raised by the other two reviewers. The HourVideo results address my concern on complex long-form video reasoning, showing consistent gains over uniform sampling, though margins over the strongest baseline are small for larger models. Please add these results to the main paper rather than the Appendix. I have raised my score.

4 comments

W1. (Lack of direct temporal-grounding evaluation) We thank the reviewer for this point and have now evaluated SPAN directly against CG-Bench's ground-truth temporal grounding intervals. We use CG-Bench because it is already part of our QA evaluation, so grounding and accuracy are measured on the same questions and selections. All methods rank the same set of candidate frames, and each method's top-K frames, with K as in Tables 1–2, form its prediction. Temporal IoU is computed between the predicted and ground-truth frame sets. Following the evaluation setting (Chen et al., 2025), we report mIoU and rec@IoU, with the latter averaged over IoU thresholds from 0.1 to 0.5. CG-Bench accuracy, averaged across the five VLM backbones, is computed from Tables 1, 2, 6, and 7. For scale, we also report mIoU and rec@IoU of 5.73 and 8.12 for GPT-4o, and 3.64 and 5.11 for Qwen2-VL-72B. These models predict interval timestamps rather than top-K frames, so they are reference points, not direct comparisons.

MethodmIoU (K=8)mIoU (K=32)rec@IoU (K=8)rec@IoU (K=32)Avg. QA acc. (K=8)Avg. QA acc. (K=32)
Uses grounding supervision
VideoITG4.373.754.753.0535.9136.91
No grounding supervision
AKS0.980.780.190.0031.7532.87
BOLT0.520.990.070.0731.2033.49
SPAN (ours)3.233.403.673.0834.7635.84

Note that a set of K frames is penalised for every frame predicted outside the clue, and its IoU with a clue of different length is bounded above, so scores need not increase with K.

We can observe the following -

  • SPAN grounds better than the annotation-free baselines. Its mIoU is more than three times that of the stronger baseline at both budgets, and its rec@IoU is 3.08–3.67 against at most 0.19.
  • SPAN recovers most of the grounding of a supervised selector. Without temporal labels and without using VLM in the selection loop, it reaches 74% (K = 8) and 91% (K = 32) of VideoITG's mIoU, and matches its rec@IoU at K = 32.
  • Grounding and QA accuracy agree. The ordering SPAN > AKS/BOLT is the same for grounding and for CG-Bench accuracy, and SPAN outperforms AKS and BOLT on CG-Bench for all five backbones at both budgets (Tables 1 and 6).

We will add this evaluation to the main paper. Furthermore, the gain achieved by SPAN due to the learned segment-level signal is also observed in Figure 2 (Right), which varies λsegment\lambda_{segment} on EgoSchema while the rest of the pipeline is held fixed. At λsegment\lambda_{segment} = 0, no segment score is used, and selection reduces to top-K frames by CLIP. This is the weakest setting, and at K = 8 and K = 16 it falls below uniform sampling. Introducing SPAN’s segment relevance improves accuracy across all K=8, 16, 32.


W2. (Limited evaluation of complex long-form video reasoning) We thank the reviewer for the suggestion and provide the following evaluation on HourVideo -

BudgetUniformBoltAKSSPAN(Ours)
Qwen3-VL-2B
K=831.8131.9831.4633.54
K=3232.8433.0233.3635.18
Qwen3-VL-4B
K=835.0034.3135.6135.78
K=3236.9138.4636.5638.46
Qwen3-VL-8B
K=833.5433.7934.4935.61
K=3235.7036.2137.0837.51
VideoLLaMA3-7B
K=832.2433.3632.1533.45
K=3231.8933.4531.8933.79

We can observe that SPAN is the best method in all settings. It improves on uniform sampling by 0.78–2.34 points (mean 1.67). Its margins over the strongest annotation-free baseline are up to 1.82 points. The trend from Table 1 carries over to HourVideo as well. We will add these results to the final manuscript.

AuthorsOpenReview ↗

Additional temporal grounding evaluation on LV-Haystack

To complement the temporal grounding on CG-Bench, we also evaluate on LV-Haystack (Ye Jinhui et al., 2025), which annotates keyframes containing the evidence needed to answer the question. We use its Haystack-LVBench subset. For each method, we take the top-K ranked frames, with K in {8, 32}.

Evaluation Metrics: Following T*’s setting, a predicted frame counts as correct if it lies within 5 seconds of any reference keyframe, and a reference keyframe counts as found if any predicted frame lies within 5 seconds of it. We report temporal precision (the fraction of predicted frames that are correct), recall (the fraction of reference keyframes found), and F1, each computed per question and averaged over questions. Since most questions have only one or two reference keyframes, precision falls as K grows, so recall is the more informative measure at larger K.

Methodprecision (K=8)precision (K=32)recall (K=8)recall (K=32)F1 (K=8)F1 (K=32)
Uses grounding supervision
VideoITG11.106.2524.1140.5814.0210.27
No grounding supervision
AKS3.241.8016.1032.845.163.35
BOLT2.181.9810.6734.173.483.67
SPAN (ours)6.313.4816.9936.428.226.04

Here as well, we observe that SPAN outperforms the best annotation-free baseline. Its F1 is 3.06-4.74 points better at the K=8 budget. At a higher budget of K=32, its recall is 2.25-3.58 points higher. Furthermore, it reaches 90% of VideoITG’s recall at K=32.


Ye, Jinhui, et al., T*: Re-thinking temporal search for long-form video understanding, 2025

Reviewer_BobDOpenReview ↗

Reviewer Response

Thank you for the additional results. The new LV-Haystack and CG-Bench evaluations address my temporal-grounding concern, which was also raised by the other two reviewers. The HourVideo results address my concern on long-form video understanding, showing consistent gains over uniform sampling, though margins over the strongest baseline are small for larger models.

Please add these results to the main paper rather than the Appendix.

I have raised my score.

Thank you for reconsidering our work and raising your score. We appreciate your feedback and will incorporate it in the manuscript. Please let us know if there are any remaining questions; we would be happy to clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.