acceptodds
Under review as a conference paper at ICLR 2027

In-Context Search with Masked Resampling for Long-Video Temporal Grounding

Abstract

Video large language models (Video LLMs) localize moments in short videos with high accuracy, yet struggle when the target lasts a few seconds within hours of video. Existing multi-stage methods for long videos, with or without Video LLMs, search from coarse to fine, but refine within a single contiguous window at a time; if that window misses the target, the refinement is wasted. We instead bridge the coarse and fine stages with an in-context search that carries several candidates from one stage to the next. Since a Video LLM outputs only the window it is most confident about, we propose masked resampling, which hides the frames of each predicted window from the model and lets it answer again, so that a single pass over the video yields several distinct candidates. The candidates are joined into a montage, a sequence of short clips shown with their original timestamps, on which we find that Video LLMs ground without any adaptation. Montages also remove the need to feed a uniformly downsampled whole video to the Video LLM as its coarse input: an embedder instead filters out clearly irrelevant content, and the Video LLM searches a montage of the most relevant clips, which retains the target in most cases at a higher per-frame resolution. The resulting method, MaRS, requires no training and improves three Video LLMs on four long-video benchmarks at the same visual budget, including models already trained on long videos. When long-video training data are available, fine-tuning on montages reduces long-video training to training on short inputs and brings further gains: with MaRS, a fine-tuned 4B model outperforms larger fine-tuned Video LLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video large language models, video temporal grounding, long-video understanding, masked resampling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_So24OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the failure modes of coarse-to-fine long-video temporal grounding caused by early single-window commitment, this paper proposes MaRS, a training-free in-context search framework. It first stitches retrieved relevant clips into a timestamped "montage" for direct Video LLM grounding, and introduces masked resampling on the key-value cache to efficiently draw multiple candidate windows across rounds. Experiments show MaRS consistently boosts Video LLMs across four long-video benchmarks.

Critical strengths

1.Directly addresses early search failures with well-grounded mechanistic evidence.

2.Leverages the training-free grounding capability on retrieved clips and extends it to multi-candidate iterative search.

Critical weaknesses
  1. Computational fairness is not sufficiently established. The paper fixes the visual token budget per grounding pass, whereas FOCUS and EXPLORE require two and three passes, respectively, alongside retrieval, query prefilling, and additional decoding. Without end-to-end latency measurements or comparisons under matched total compute, the results do not establish an advantage at equivalent overall computational cost.

  2. The mechanism underlying cache masking lacks direct validation. Blocking query and answer tokens from attending to selected window tokens does not necessarily remove their information, as the remaining cached representations may already encode it through prefix self-attention. A comparison with explicitly removing the selected windows and re-encoding the input would clarify how this approximation affects candidate diversity, grounding accuracy, and computational cost.

  3. Controlled comparisons with the closest search methods are missing. Despite comparisons across multiple backbone models and internal ablations, the paper does not directly compare against closely related approaches, such as the retrieve-then-ground pipeline of Seo & Kim, using the same retriever, grounding model, and total computational budget.

  4. The technical novelty appears limited. Concatenating retrieved clips, excluding previously selected regions to generate alternative candidates, and retaining multiple hypotheses during coarse-to-fine refinement are relatively straightforward design choices; grounding over multiple retrieved windows with a frozen Video LLM also has precedent. The more distinctive contribution lies in cache-level masked resampling and its integration into iterative search, but the paper has not sufficiently demonstrated that this combination constitutes a substantial methodological advance.

Questions for authors

Please address the concerns raised in Critical Weaknesses. Additionally, Appendix B.1 notes that the external 128K baselines in Table 2 were evaluated on 525/715 VUE-TR/VUE-TR-V2 queries, whereas your evaluation uses 512/691 queries due to video availability. These results therefore do not constitute a strictly controlled comparison. Could you re-evaluate the relevant baseline models on the same 512/691-query subsets? Since masked resampling is stochastic, please also report results across multiple random seeds, including the number of runs and mean ± standard deviation for the key comparisons. I would like to raising my rating if the clarifications and additional evidence adequately address these concerns.

7 comments
AuthorsOpenReview ↗

Computational fairness (Weakness 1) - Part 1

Response. Thank you for this comment. We measured the time of each step, with retrieval included, and we added a comparison at a matched grounding cost. Table 8 gives the results. We added Table 8 to the appendix of the revised paper.

Table 8. Cost per query. We use 87 videos that are longer than 17 minutes, with one query for each video: 8 from Ego4D-NLQ, 16 from MomentSeeker, 28 from VUE-TR and 35 from VUE-TR-V2. These videos have a length of 18 to 119 minutes, with a mean of 51 minutes. Time is GPU time on one A100 GPU. We do not include video decoding for any method.

ModelMethodTPF for Emb.Visual tokensInput tokensRetrieval (s)Grounding passes (s)Draws (s)Peak GPU memory (GB)mIoU, mean of 4 benchmarks (%)
TimeLens-8BOne pass, whole video, 28K (official setup)-25,15029,241–5.5–24.727.5
TimeLens-8BOne pass, whole video, 128K (official setup)-118,682122,774–40.8–52.128.8
TimeLens-8BCoarse-to-fine, UniTime-style (2 passes)-41,26846,275–8.8–28.329.2
TimeLens-8BFocus (2 passes)76845,99748,23235.59.30.330.535.8
TimeLens-8BFocus (2 passes)12845,99748,23210.69.30.330.535.1
TimeLens-8BExplore (3 passes)76863,49970,80535.513.61.630.935.2
TimeLens-8BExplore (3 passes)12863,49970,80510.613.61.630.935.1
TimeLens2-8BOne pass, whole video, 28K (official setup)-22,55832,799–6.6–26.834.5
TimeLens2-8BOne pass, whole video, 128K (official setup)-114,822125,063–42.3–54.537.2
TimeLens2-8BCoarse-to-fine, UniTime-style (2 passes)-41,92153,544–10.5–28.335.2
TimeLens2-8BFocus (2 passes)76842,03844,11335.58.60.630.538.4
TimeLens2-8BFocus (2 passes)12842,03844,11310.68.60.630.538.0
TimeLens2-8BExplore (3 passes)76856,84763,85135.512.33.130.938.5
TimeLens2-8BExplore (3 passes)12856,84763,85110.612.33.130.938.5
  • TPF for Emb.: the maximum number of visual tokens for each frame pair in the embedder. The embedder scores each clip one time. 768 is the default (the paper setting), and 128 is a lighter setting.
  • Retrieval (s): the time for one video. Retrieval runs one time for each video. The other times are for one query. The videos of the four benchmarks have a mean length of 8 to 25 minutes, so retrieval takes less time for a typical benchmark video than in this table.
  • Retrieval runs with the fastest batch size of each embedder setting (16 for 768, 64 for 128).
  • Visual tokens and input tokens are totals over all passes. Input tokens: the visual tokens plus the text tokens. The text includes the timestamps. Peak GPU memory: the memory of the grounding model.
  • mIoU: the mean over the four benchmarks of the revised Table 2, where each one-pass baseline uses its official evaluation setup and MaRS is used for videos longer than 192 s (see our response to Weakness 4 of Reviewer YDSb). For the rows with 128, the mean is from Table 9.
  • The rows with 128 use the grounding times, tokens and memory of the rows with 768. We measured only the retrieval time and the mIoU again.
AuthorsOpenReview ↗

Computational fairness (Weakness 1) - Part 2

Results.

  1. Matched grounding cost. We added a coarse-to-fine baseline that follows the two-stage inference of UniTime with the same grounders. It uses two passes and about 41K visual tokens in total, which is close to the total of Focus (42K to 46K). Its grounding time (8.8 to 10.5 s) is also close to that of Focus (9.2 to 9.6 s). At this matched grounding cost, Focus is 2.8 to 6.6 mIoU points above it.
  2. Compared with one pass. MaRS is more accurate than one pass, at a grounding cost between one 28K pass and one 128K pass. Over all passes, Focus and Explore use 42K to 63K visual tokens, 9 to 15 s of grounding time and at most 31 GB of GPU memory.
    • One 28K pass uses 23K to 25K visual tokens, 5.5 to 6.6 s and 25 to 27 GB. Focus and Explore increase its mean mIoU by 3.5 to 8.3 points.
    • One 128K pass uses 115K to 119K visual tokens, 41 to 42 s and 52 to 55 GB. Focus and Explore increase its mean mIoU by 0.8 to 7.0 points, with about half of the visual tokens or less and less than 40% of the grounding time.
  3. Retrieval overhead. With 768 tokens for each frame pair, retrieval takes 35.5 s for each of these videos (mean length 51 minutes). This is the default setting of the official code of the embedder, and we used it during the development of MaRS. This budget is larger than necessary: with 128 tokens for each frame pair, retrieval takes 10.6 s, and the mean mIoU changes by 0.7 points or less. Retrieval runs one time for each video, and all queries on this video use its clip embeddings again. Each query then needs only the grounding time of 9 to 15 s (grounding passes and draws), which is less than 40% of the time of one 128K pass. Results in the main paper keep the default of 768 tokens per frame pair. We added the full results with 128 tokens to the appendix (Table 9).
  4. Query prefilling and additional decoding. The masked draws use the KV cache of the montage again, so they do not encode the video again. All draws together take 0.3 to 3.1 s for each query. This time includes the prefilling of the query and the decoding of the answers.
AuthorsOpenReview ↗

Validation of cache masking (Weakness 2)

Response. Thank you for this suggestion. We ran this comparison. The strict variant removes the windows of the earlier answers, encodes the remaining montage again, and then generates the answer. Table 14 compares it with masked resampling on candidate diversity, grounding accuracy and computational cost.

Table 14. Masked resampling (KV cache used again) and strict masking (montage encoded again). TimeLens-8B, 14K visual tokens for each pass, ablation set (Ego4D-NLQ dev and MomentSeeker long, 1,707 queries). The candidates are the first answer and 8 draws. The final accuracy is the mIoU of MaRS-Explore.

MeasureMasked resampling (ours)Strict masking
Distinct candidates (of 9)5.806.12
Recall: a candidate overlaps the target (%)71.874.3
Recall: a candidate has IoU ≥ 0.3 with the target (%)35.437.6
Final mIoU, Ego4D-NLQ dev (%)18.917.9
Final mIoU, MomentSeeker long (%)11.011.6
Time for each draw (s)0.18 to 0.201.95 to 3.79
  1. Candidate diversity. Masked resampling keeps the diversity of strict masking. It gives 95% of the distinct candidates of strict masking, and 94% to 97% of its recall.
  2. Grounding accuracy. Strict masking does not increase the final accuracy. Over the two sets (1,707 queries), the final mIoU is 17.9 with masked resampling and 17.1 with strict masking.
  3. Computational cost. Masked resampling is 10 to 20 times faster for each draw. A masked draw uses the KV cache again, so its time does not increase with the token budget. A strict draw encodes the montage again, so its time increases from about 2 s at 14K tokens to about 3.8 s at 28K tokens.

Thus masked resampling gives the benefit of strict masking at a small part of its cost. We revised Section 3.2 to describe exactly what the masking does, and we added the strict variant and Table 14 to the appendix of the revised paper.

Reviewer_YDSbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MaRS, a training-free framework for long-video temporal grounding that combines retrieval-based montage construction with masked resampling. Specifically, an embedding model first retrieves relevant video clips to construct a montage, and a Video LLM then performs multi-round coarse-to-fine grounding over multiple candidate windows. To efficiently obtain diverse candidates, the proposed masked resampling mechanism prevents the model from directly attending to previously predicted windows while reusing the cached key-value representations. The authors further introduce montage-based fine-tuning using GRPO to adapt Video LLMs to retrieved video clips. Experiments on multiple long-video grounding benchmarks demonstrate consistent improvements across different Video LLMs, with additional gains after fine-tuning.

Critical strengths
  1. The proposed masked resampling mechanism provides an interesting way to generate diverse temporal candidates while reusing the KV cache, reducing the overhead of repeated predictions. The improved candidate recall provides empirical evidence for the effectiveness of this design.

  2. Strong empirical performance. The proposed method consistently improves multiple Video LLMs across several long-video grounding benchmarks.

  3. The paper also provides relatively comprehensive ablation studies on retrieval models, candidate numbers, search rounds, and training inputs, which support the effectiveness of the overall framework.

Critical weaknesses
  1. Limited methodological novelty. The paper's first claimed contribution, identifying that existing coarse-to-fine methods for long-video temporal grounding refine within a single contiguous window, is not entirely accurate. Several earlier methods, such as CONE [1] and SOONet [2], already retain multiple high-confidence coarse proposals for subsequent refinement. Therefore, maintaining multiple candidates across coarse-to-fine stages is not a new insight. Furthermore, the proposed montage-based fine-tuning mainly applies existing GRPO training to retrieved video clips, without introducing a specific optimization mechanism for long-video temporal grounding. Overall, the methodological novelty appears largely limited to masked resampling, while the other contributions are relatively incremental.

[1] CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding. ACL 2023.

[2] Scanning Only Once: An End-to-end Framework for Fast Temporal Grounding in Long Videos. ICCV 2023.

  1. Potential information leakage in masked resampling. The proposed masking mechanism may not fully eliminate information from previously predicted windows. During the initial prefilling stage, video tokens are processed without masking, allowing later video tokens to attend to earlier ones through causal self-attention. Consequently, even if the query and answer tokens are subsequently prevented from directly attending to the masked window, they may still indirectly access its information through the cached representations of later video tokens. It is suggested to compare their implementation with a strict masking baseline that removes the selected window and recomputes the KV cache by prefilling the remaining video tokens and query.

  2. Limited ability to correct incomplete or inaccurate temporal boundaries. The proposed multi-round refinement appears unable to recover event regions outside previously predicted windows once those regions are discarded. For example, if the model initially predicts only a discriminative subsegment within the ground-truth event, subsequent refinement may further narrow the prediction rather than recover the complete temporal extent. Similarly, inaccurate initial boundaries may be difficult to correct when relevant surrounding frames are excluded from subsequent montages. This limitation raises concerns about the method's ability to achieve accurate temporal boundary localization, rather than merely identifying a relevant portion of the event.

  3. Lack of evaluation on short-video temporal grounding benchmarks. Although the paper focuses on long videos, the proposed framework should also be evaluated on standard short-video benchmarks, such as Charades-STA and ActivityNet Captions, to establish its general applicability. In particular, mandatory multi-round coarse-to-fine refinement may introduce undesirable effects when the original video already contains sufficient temporal information. Repeatedly narrowing the candidate windows could encourage the model to identify the most discriminative part of an event rather than its complete temporal extent. Without such evaluations, it remains unclear whether MaRS consistently improves temporal grounding or introduces performance degradation in shorter videos.

  4. Lack of fair end-to-end inference cost comparisons. The paper emphasizes comparisons under the same visual token budget per pass. However, MaRS involves multiple search rounds, additional visual encoding, repeated prefilling and decoding, and embedding-based retrieval. Therefore, the total inference cost can be substantially higher than that of single-pass baselines. Using the same per-pass visual token budget does not guarantee a fair comparison in terms of overall computation. The paper should report end-to-end inference latency, total computational cost, and peak GPU memory, including retrieval overhead, and compare against existing methods under matched total inference budgets.

Questions for authors
  1. Could the authors evaluate a strict masking variant that removes previously predicted windows and recomputes the video KV cache before generating subsequent predictions? How does this compare with the proposed KV-cache reuse strategy in terms of candidate diversity, recall, and final grounding accuracy?

  2. How does MaRS recover complete temporal boundaries when an initial prediction covers only a subsegment of the ground-truth event? Could the authors provide a quantitative analysis of such cases, particularly when the initial predictions have high semantic relevance but incomplete temporal coverage?

  3. Could the authors evaluate MaRS on Charades-STA and ActivityNet Captions? In particular, does repeated coarse-to-fine refinement negatively affect the localization of complete events in short videos, compared with single-pass grounding?

  4. Could the authors provide end-to-end inference time, total computational cost, and peak GPU memory comparisons under matched overall inference budgets, including the overhead of embedding-based retrieval and all search rounds?

7 comments
AuthorsOpenReview ↗

Novelty (Weakness 1) - Part 1

Response. Thank you for these references. CONE and SOONet keep several candidates after the coarse step. Our claim is about how these candidates are used afterwards.

  • Group 1 (CONE, SOONet). These methods score each candidate against the query separately, and the final prediction comes from inside one candidate.
  • Group 2 (UniTime, ReVisionLLM). UniTime (code) and ReVisionLLM (paper) refine over several steps, but each step operates inside one contiguous window.

Thus, in both groups, the final prediction comes from within one contiguous window, as the first contribution in the revised paper states. In contrast, MaRS combines the two properties that no method above has together: several candidates and several steps. Group 1 has several candidates but one per-candidate refinement step. Group 2 has several refinement steps but one window. In MaRS, each step gets several candidates from the frozen Video LLM and passes them to the next step to be refined in context. We give the three differences below.

1. Compared to Group 1, MaRS conducts in-context comparison, not candidate-wise ranking. CONE and SOONet score each candidate on its own against the query. In MaRS, the Video LLM reads all retrieved clips together in one pass and compares them in context. This gives a higher recall (Table 12).

Table 12. First choice: one pass over all clips, or one clip at a time (% of queries where the choice overlaps the ground truth). Both methods get the same top-32 retrieved clips. The reranker (Qwen3-VL-Reranker-8B) scores each clip alone and takes the clip with the highest score. The Video LLM reads all 32 clips in one pass and gives its first answer. For each query, the shorter choice is padded to the length of the longer choice. Thus the reranker value depends on the Video LLM in the same row.

Video LLMSetReranker, one clip at a timeVideo LLM, all clips in one pass
TimeLens-8BEgo4D-NLQ dev35.841.5
TimeLens-8BMomentSeeker long33.336.7
TimeLens2-8BEgo4D-NLQ dev37.345.6
TimeLens2-8BMomentSeeker long39.149.8

2. Compared to Groups 1 and 2, MaRS grounds in a montage of several candidates, not in one segment. In the methods of both groups, the final answer comes from one candidate segment. In MaRS, the final pass grounds the query in a montage of several candidates. Table 13 compares the two directly. Figure 5 and Table 4 of the paper also show that the final pass is more accurate with several candidates than with the first answer only.

Table 13. Final pass: one top-ranked segment, or a montage of several candidates (mIoU, %). One segment: the reranker selects the best of the top-32 clips, and the Video LLM grounds the query in a 64 s segment around it. MaRS-Focus: the Video LLM grounds the query in a 64 s montage of its first answer and two masked draws. Both use TimeLens-8B with the same prompt and 14K visual tokens.

SetOne top-ranked segmentMontage of several candidates (MaRS-Focus)
Ego4D-NLQ dev16.519.2
MomentSeeker long9.510.8

3. Compared to Group 1, MaRS repeats the search. Compared to Group 2, each round keeps several candidates. The methods of Group 1 have one refinement step. The methods of Group 2 repeat the refinement, but inside one contiguous window. MaRS-Explore repeats the search (for example 512 s, then 128 s, then 64 s) and passes several candidates from each round to the next. Table 4 of the paper shows that a second round helps only when several candidates go into it.

We revised the abstract, the introduction, Figure 1 and the related work to state these differences explicitly, and we added Tables 12 and 13 to the appendix. Figure 1(a) now shows both groups of existing methods: methods that retrieve several windows and commit to the top-scored one, and methods that pick one coarse window and refine it by zooming in.

AuthorsOpenReview ↗

Novelty (Weakness 1) - Part 2

Montage-based fine-tuning. The contribution is the training input, which is specific to long videos.

  1. It removes the difference between the training input and the test input. With the same GRPO objective, token budget and number of steps, training on montages is 3.3 mIoU points above training on the downsampled whole video on Ego4D-NLQ, and 4.3 points above it on TACoS (Table 6). The same models, tested without further training on MomentSeeker long, give a gain of 5.0 points.
  2. It needs no long-context training stage. Recent long-video grounders use such a stage. TimeLens2 runs supervised fine-tuning on sequences of up to 100K tokens, and it reports a decrease of 3.3 mIoU points on Ego4D-NLQ and 4.0 points on VUE-TR-V2 when this length is 16K. VideoChat3 has a long-video tuning stage with sequences of up to 98K tokens. In our pipeline, each training sample has at most 14K visual tokens.
  3. It is effective. With no long-context stage, our 4B model gets 21.2 mIoU on Ego4D-NLQ. TimeLens2-8B, which is two times larger, gets 19.0 (Table 3).

We revised Section 3.4, the paragraph "Montage-based fine-tuning" in Section 4.3 and the third contribution in the introduction to state point 2: montage-based fine-tuning needs no long-context training stage.

AuthorsOpenReview ↗

Strict masking (Weakness 2, Question 1)

Response. Thank you for this suggestion. We ran the strict masking variant: for each draw, it removes the windows of the earlier answers, encodes the remaining montage again, and then generates the answer. Table 14 compares it with masked resampling on candidate diversity, recall, final accuracy and time.

Table 14. Masked resampling (KV cache used again) and strict masking (montage encoded again). TimeLens-8B, 14K visual tokens for each pass, ablation set. The candidates are the first answer and 8 draws. The final accuracy is the mIoU of MaRS-Explore.

MeasureMasked resampling (ours)Strict masking
Distinct candidates (of 9)5.806.12
Recall: a candidate overlaps the target (%)71.874.3
Recall: a candidate has IoU ≥ 0.3 with the target (%)35.437.6
Final mIoU, Ego4D-NLQ dev (%)18.917.9
Final mIoU, MomentSeeker long (%)11.011.6
Time for each draw (s)0.18 to 0.201.95 to 3.79
  1. Masked resampling keeps the diversity of strict masking. It gives 95% of the distinct candidates of strict masking, and 94% to 97% of its recall.
  2. Strict masking does not increase the final accuracy. Over the two sets (1,707 queries), the final mIoU is 17.9 with masked resampling and 17.1 with strict masking.
  3. Masked resampling is 10 to 20 times faster for each draw. A masked draw uses the KV cache again, so its time does not increase with the token budget. A strict draw encodes the montage again, so its time increases from about 2 s at 14K tokens to about 3.8 s at 28K tokens.

Thus masked resampling gives the benefit of strict masking at a small part of its cost. We revised Section 3.2 to describe exactly what the masking does, and we added the strict variant and Table 14 to the appendix of the revised paper.

Reviewer_u7KpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MaRS, a multi-stage framework for long-video temporal grounding. It first retrieves relevant video clips to construct a montage, then uses masked resampling to generate multiple candidate temporal windows. These candidates are combined into a new montage for further refinement. The method can work without additional training and shows improvements across several long-video benchmarks.

Critical strengths
  1. The paper addresses an important challenge in long-video temporal grounding.

  2. Masked resampling is an interesting way to generate diverse temporal candidates without retraining the VLM.

  3. The method shows promising improvements across multiple datasets and VLM backbones.

Critical weaknesses
  1. Inference efficiency: The method requires multiple VLM decoding steps and search rounds. Although KV-cache reuse reduces repeated computation, the actual inference latency and computational overhead are not sufficiently clear.

  2. Limited analysis of masked resampling: Masking previously predicted windows may remove important visual context needed to understand the video. The paper shows improvements in candidate coverage, but does not fully explain how useful candidates are distributed across different resampling iterations.

  3. Fixed-length clip segmentation: The initial montage is constructed from fixed 4-second clips. Such segmentation may split continuous actions or semantically related events, potentially losing important temporal context. The effect of this design choice is not sufficiently investigated.

Questions for authors
  1. Could the authors provide a detailed inference latency and computational cost comparison with single-pass and other multi-stage methods, including the overhead of retrieval, masked resampling, and KV-cache operations?

  2. Could the authors report the percentage of queries for which the ground truth is first covered by the 1st, 2nd, 3rd, and later candidates? This would help demonstrate the contribution of each resampling iteration. Additionally, how does masking affect predictions that require information from previously masked regions?

  3. Could the authors provide an ablation comparing fixed 4-second clips with different clip lengths or adaptive, semantic-aware temporal segmentation? This would help determine whether fixed segmentation breaks important temporal information.

4 comments
AuthorsOpenReview ↗

Inference cost (Weakness 1, Question 1) - Part 1

Response. Thank you for this question. We measured the cost of each step, for MaRS and for the one-pass baselines. Table 8 gives the results. We added Table 8 to the appendix of the revised paper.

Table 8. Cost per query. We use 87 videos that are longer than 17 minutes, with one query for each video: 8 from Ego4D-NLQ, 16 from MomentSeeker, 28 from VUE-TR and 35 from VUE-TR-V2. These videos have a length of 18 to 119 minutes, with a mean of 51 minutes. Time is GPU time on one A100 GPU. We do not include video decoding for any method.

ModelMethodTPF for Emb.Visual tokensInput tokensRetrieval (s)Grounding passes (s)Draws (s)Peak GPU memory (GB)mIoU, mean of 4 benchmarks (%)
TimeLens-8BOne pass, whole video, 28K (official setup)-25,15029,241–5.5–24.727.5
TimeLens-8BOne pass, whole video, 128K (official setup)-118,682122,774–40.8–52.128.8
TimeLens-8B (UniTime-Style)Coarse-to-fine (2 passes)-41,26846,275–8.8–28.329.2
TimeLens-8BFocus (2 passes)76845,99748,23235.59.30.330.535.8
TimeLens-8BFocus (2 passes)12845,99748,23210.69.30.330.535.1
TimeLens-8BExplore (3 passes)76863,49970,80535.513.61.630.935.2
TimeLens-8BExplore (3 passes)12863,49970,80510.613.61.630.935.1
TimeLens2-8BOne pass, whole video, 28K (official setup)-22,55832,799–6.6–26.834.5
TimeLens2-8BOne pass, whole video, 128K (official setup)-114,822125,063–42.3–54.537.2
TimeLens2-8B (UniTime-Style)Coarse-to-fine (2 passes)-41,92153,544–10.5–28.335.2
TimeLens2-8BFocus (2 passes)76842,03844,11335.58.60.630.538.4
TimeLens2-8BFocus (2 passes)12842,03844,11310.68.60.630.538.0
TimeLens2-8BExplore (3 passes)76856,84763,85135.512.33.130.938.5
TimeLens2-8BExplore (3 passes)12856,84763,85110.612.33.130.938.5
  • TPF for Emb.: the maximum number of visual tokens for each frame pair in the embedder. 768 is the default (the paper setting), and 128 is a lighter setting.
  • Retrieval (s): the time for one video. Retrieval runs one time for each video. The other times are for one query. The videos of the four benchmarks have a mean length of 8 to 25 minutes, so retrieval takes less time for a typical benchmark video than in this table.
  • Input tokens: the visual tokens plus the text tokens. The text includes the timestamps. Peak GPU memory: the memory of the grounding model.
  • mIoU: the mean over the four benchmarks of the revised Table 2, where each one-pass baseline uses its official evaluation setup and MaRS is used for videos longer than 192 s (see our response to Weakness 4 of Reviewer YDSb). For the rows with 128, the mean is from Table 9.
  • The rows with 128 use the grounding times, tokens and memory of the rows with 768.
AuthorsOpenReview ↗

Inference cost (Weakness 1, Question 1) - Part 2

Results.

  1. Masked resampling is cheap. The draws use the KV cache of the montage again, so they do not encode the video again. All draws together take 0.3 to 3.1 s (2 to 10 draws) for each query, which is 3% to 20% of the grounding time (grounding passes and draws). The time to load the KV cache is part of this time.
  2. MaRS is more accurate than one pass, at a grounding cost between one 28K pass and one 128K pass. Focus and Explore use 42K to 63K visual tokens, 9 to 15 s of grounding time and at most 31 GB of GPU memory.
    • One 28K pass uses 23K to 25K visual tokens, 5.5 to 6.6 s and 25 to 27 GB. Focus and Explore increase its mean mIoU by 3.5 to 8.3 points.
    • One 128K pass uses 115K to 119K visual tokens, 41 to 42 s and 52 to 55 GB. Focus and Explore increase its mean mIoU by 0.8 to 7.0 points, with about half of the visual tokens or less and less than 40% of the grounding time.
  3. Retrieval adds a cost that is easy to reduce. With 768 tokens for each frame pair, retrieval takes 35.5 s for each of these videos (mean length 51 minutes). This is the default setting of the official code of the embedder, and we used it during the development of MaRS. This budget is larger than necessary: with 128 tokens for each frame pair, retrieval takes 10.6 s, and the mean mIoU changes by 0.7 points or less. Retrieval runs one time for each video, and all queries on this video use its clip embeddings again. Each query then needs only the grounding time of 9 to 15 s (grounding passes and draws), which is less than 40% of the time of one 128K pass. Retrieval can also run on several GPUs, because the embedder encodes each clip separately. Results in the main paper keep the default of 768 tokens per frame pair. We added the full results with 128 tokens to the appendix (Table 9). The main conclusions do not change: Focus and Explore stay above one 28K pass in mIoU on all four benchmarks, for both TimeLens-8B and TimeLens2-8B. They also stay above one 128K pass on all four benchmarks for TimeLens-8B, and on Ego4D-NLQ and VUE-TR-V2 for TimeLens2-8B.
  4. Other multi-stage methods. We added a coarse-to-fine baseline that follows the two-stage inference of UniTime with the same grounders (rows "UniTime-Style" in Table 8). The first pass reads the whole video with 28K tokens and selects a segment of 32 s clips. The second pass reads this segment again at 2 fps. This baseline takes 8.8 to 10.5 s for each query, which is close to the grounding time of Focus (9.2 to 9.6 s). It increases the mean mIoU of one 28K pass by 0.7 to 1.7 points. Focus and Explore are 2.8 to 6.6 points above it. The baseline keeps one segment from its first pass; MaRS keeps several candidates.
AuthorsOpenReview ↗

Contribution of each draw and masked context (Weakness 2, Question 2)

Response. Thank you for these questions. We answer them with the records of our main experiments on the ablation set.

Which candidate first covers the target. Table 10 gives, for each query, the first candidate that overlaps the ground truth. The candidates are in the order that the model gives them: the first answer, then draws 1 to 8 of the first search round of Explore.

Table 10. First candidate that overlaps the ground truth (% of queries).

ModelSetFirst answerDraw 1Draw 2Draws 3–8All draws
TimeLens-8BEgo4D dev37.310.17.921.539.5
TimeLens-8BMomentSeeker long29.59.74.321.335.3
TimeLens2-8BEgo4D dev39.612.78.721.442.8
TimeLens2-8BMomentSeeker long40.67.77.218.833.7
VideoChat3-4BEgo4D dev33.310.26.320.236.7
VideoChat3-4BMomentSeeker long26.64.86.816.928.5
  1. Every draw adds new targets. In 28.5% to 42.8% of the queries, a draw is the first candidate that reaches the target. In these queries, only the draws bring the target into the next round.
  2. The first draws add the most. On average, later draws add fewer new targets. This agrees with the sweep of the number of candidates in the paper (Figure 5).

How masking affects predictions that need masked context.

  1. Masked resampling aims at recall, not at the final boundaries. Each draw looks for another place that is relevant to the query, often from local visual cues, which need little context from the masked windows. In the paper, the draws cover more of the ground truth than random windows of the same length (Table 5).
  2. The mask affects only the draws, not the final prediction. The final pass reads all candidates without a mask, and the masked windows are among these candidates. Thus the final prediction can use the information in the masked regions.
  3. Figure 6 shows a real example. In Figure 6 of the revised appendix, the query depends on an event that the masked window contains. Draw 2 still finds the target from a local cue, without that context. The final pass then reads both windows without a mask and gives the correct span (IoU 0.67).

We added Table 10 and Figure 6 to the appendix of the revised paper.

AuthorsOpenReview ↗

Fixed-length clips (Weakness 3, Question 3)

Response. Thank you for this suggestion. We ran the ablation with clip lengths of 4 s, 8 s, 16 s, 32 s and 64 s. Table 11 gives the results.

Table 11. mIoU (%) for different clip lengths. The embedder scores clips of the given length. The first-pass input is always 128 s (the top 32, 16, 8, 4 or 2 clips). We use the ablation setting of the paper: 14K visual tokens for each pass, on the ablation set. "One pass" is one grounding pass on this input. MaRS-Focus uses the same input.

Clip lengthInput overlaps the target (%)TimeLens-8B, one passTimeLens-8B, MaRS-FocusTimeLens2-8B, one passTimeLens2-8B, MaRS-Focus
4 s (paper)81.516.517.717.319.3
8 s77.716.818.317.419.7
16 s70.716.717.217.018.8
32 s64.016.017.216.618.3
64 s54.914.815.115.516.5
  1. MaRS is not sensitive to the clip length from 4 s to 32 s. Compared with 4 s clips, the mIoU of MaRS-Focus changes by 1.0 point or less.
  2. Short clips find the target more frequently. The input overlaps the target for 81.5% of the queries with 4 s clips, and for 54.9% with 64 s clips. Adjacent selected clips also make one continuous segment in the montage, so an action that is longer than one clip stays whole. Thus the montage adapts the segment length to the content.
  3. Very long clips decrease the accuracy. With 64 s clips, the mIoU of MaRS-Focus is 2.6 to 2.8 points lower than with 4 s clips.
  4. The search restores the context that a clip boundary cuts. For the final pass, MaRS pads each candidate with the video around it (Appendix A.2). Thus the final montage shows a continuous segment around each candidate, not separate clips. As a result, MaRS-Focus is above one pass on the same input for all five clip lengths and both models.

We added Table 11 to the appendix of the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our responses

We thank the Area Chair for handling our submission, and Reviewers u7Kp, YDSb and So24 for their time and for their careful and constructive comments.

We also thank the reviewers for the strengths that they pointed out:

  • Problem. The paper addresses an important challenge in long-video temporal grounding (u7Kp), and it directly addresses early search failures with well-grounded mechanistic evidence (So24).
  • Method. Masked resampling is an interesting way to generate diverse temporal candidates without retraining the Video LLM (u7Kp), and it uses the KV cache again, with the improved candidate recall as evidence for this design (YDSb). The method extends training-free grounding on retrieved clips to a multi-candidate iterative search (So24).
  • Results. The method consistently improves several Video LLMs on several long-video benchmarks (u7Kp, YDSb), and the ablation studies on retrieval models, candidate numbers, search rounds and training inputs support the effectiveness of the framework (YDSb).
  • Presentation. The paper is clear and easy to follow (u7Kp, YDSb).

We answered each comment in a separate reply to each reviewer. We also uploaded a revised paper, in which all changes are in blue. This note gives a short summary of our replies.

1. Inference cost (u7Kp, YDSb, So24). We measured the cost of each step of MaRS and of the one-pass baselines: the number of tokens, the time and the peak GPU memory, with retrieval and the masked draws as separate items. We also added a coarse-to-fine baseline at a matched grounding cost. The results show that the masked draws add little cost, and that the gain of MaRS does not come from more computation.

2. Relation to prior methods and novelty (YDSb, So24). We made clear how MaRS differs from the methods that keep several candidates and from the methods that refine over several steps. We added controlled comparisons that test these differences directly. We revised the abstract, the introduction, Figure 1 and the related work to state these differences.

3. Masked resampling (u7Kp, YDSb, So24). We ran the strict masking variant that the reviewers asked for, and we compared it with masked resampling in candidate diversity, recall, final accuracy and time. We also reported which candidate first reaches the target, with an example. We revised Section 3.2 to describe exactly what the mask does.

4. Controlled comparison with Seo & Kim (So24). We ran this comparison with the same retriever, the same Video LLM and the same GPU time.

5. Same VUE queries and random seeds (So24). We ran the baselines with the large token budget on our own VUE queries, and we ran MaRS with several random seeds. We reported the mean and the standard deviation.

6. Other concerns.

  • Clip length (u7Kp): we added an ablation over the clip length.
  • Incomplete first answers (YDSb): we added an analysis of these cases, with an example.
  • Short videos (YDSb): we evaluated MaRS on the short-video benchmarks and stated how MaRS handles short videos.
  • Montage-based fine-tuning (YDSb): we made clear that it needs no long-context training stage.

Revised paper. The new results are in Appendix B.2 to B.10. In Table 2, the rows that we evaluated now use the official evaluation setup of each one-pass baseline. The conclusions of the paper do not change.

We are glad to answer any further question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.