- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the failure modes of coarse-to-fine long-video temporal grounding caused by early single-window commitment, this paper proposes MaRS, a training-free in-context search framework. It first stitches retrieved relevant clips into a timestamped "montage" for direct Video LLM grounding, and introduces masked resampling on the key-value cache to efficiently draw multiple candidate windows across rounds. Experiments show MaRS consistently boosts Video LLMs across four long-video benchmarks.
Critical strengths
1.Directly addresses early search failures with well-grounded mechanistic evidence.
2.Leverages the training-free grounding capability on retrieved clips and extends it to multi-candidate iterative search.
Critical weaknesses
-
Computational fairness is not sufficiently established. The paper fixes the visual token budget per grounding pass, whereas FOCUS and EXPLORE require two and three passes, respectively, alongside retrieval, query prefilling, and additional decoding. Without end-to-end latency measurements or comparisons under matched total compute, the results do not establish an advantage at equivalent overall computational cost.
-
The mechanism underlying cache masking lacks direct validation. Blocking query and answer tokens from attending to selected window tokens does not necessarily remove their information, as the remaining cached representations may already encode it through prefix self-attention. A comparison with explicitly removing the selected windows and re-encoding the input would clarify how this approximation affects candidate diversity, grounding accuracy, and computational cost.
-
Controlled comparisons with the closest search methods are missing. Despite comparisons across multiple backbone models and internal ablations, the paper does not directly compare against closely related approaches, such as the retrieve-then-ground pipeline of Seo & Kim, using the same retriever, grounding model, and total computational budget.
-
The technical novelty appears limited. Concatenating retrieved clips, excluding previously selected regions to generate alternative candidates, and retaining multiple hypotheses during coarse-to-fine refinement are relatively straightforward design choices; grounding over multiple retrieved windows with a frozen Video LLM also has precedent. The more distinctive contribution lies in cache-level masked resampling and its integration into iterative search, but the paper has not sufficiently demonstrated that this combination constitutes a substantial methodological advance.
Questions for authors
Please address the concerns raised in Critical Weaknesses. Additionally, Appendix B.1 notes that the external 128K baselines in Table 2 were evaluated on 525/715 VUE-TR/VUE-TR-V2 queries, whereas your evaluation uses 512/691 queries due to video availability. These results therefore do not constitute a strictly controlled comparison. Could you re-evaluate the relevant baseline models on the same 512/691-query subsets? Since masked resampling is stochastic, please also report results across multiple random seeds, including the number of runs and mean ± standard deviation for the key comparisons. I would like to raising my rating if the clarifications and additional evidence adequately address these concerns.
7 comments
Computational fairness (Weakness 1) - Part 2
Results.
- Matched grounding cost. We added a coarse-to-fine baseline that follows the two-stage inference of UniTime with the same grounders. It uses two passes and about 41K visual tokens in total, which is close to the total of Focus (42K to 46K). Its grounding time (8.8 to 10.5 s) is also close to that of Focus (9.2 to 9.6 s). At this matched grounding cost, Focus is 2.8 to 6.6 mIoU points above it.
- Compared with one pass. MaRS is more accurate than one pass, at a grounding cost between one 28K pass and one 128K pass. Over all passes, Focus and Explore use 42K to 63K visual tokens, 9 to 15 s of grounding time and at most 31 GB of GPU memory.
- One 28K pass uses 23K to 25K visual tokens, 5.5 to 6.6 s and 25 to 27 GB. Focus and Explore increase its mean mIoU by 3.5 to 8.3 points.
- One 128K pass uses 115K to 119K visual tokens, 41 to 42 s and 52 to 55 GB. Focus and Explore increase its mean mIoU by 0.8 to 7.0 points, with about half of the visual tokens or less and less than 40% of the grounding time.
- Retrieval overhead. With 768 tokens for each frame pair, retrieval takes 35.5 s for each of these videos (mean length 51 minutes). This is the default setting of the official code of the embedder, and we used it during the development of MaRS. This budget is larger than necessary: with 128 tokens for each frame pair, retrieval takes 10.6 s, and the mean mIoU changes by 0.7 points or less. Retrieval runs one time for each video, and all queries on this video use its clip embeddings again. Each query then needs only the grounding time of 9 to 15 s (grounding passes and draws), which is less than 40% of the time of one 128K pass. Results in the main paper keep the default of 768 tokens per frame pair. We added the full results with 128 tokens to the appendix (Table 9).
- Query prefilling and additional decoding. The masked draws use the KV cache of the montage again, so they do not encode the video again. All draws together take 0.3 to 3.1 s for each query. This time includes the prefilling of the query and the decoding of the answers.
Validation of cache masking (Weakness 2)
Response. Thank you for this suggestion. We ran this comparison. The strict variant removes the windows of the earlier answers, encodes the remaining montage again, and then generates the answer. Table 14 compares it with masked resampling on candidate diversity, grounding accuracy and computational cost.
Table 14. Masked resampling (KV cache used again) and strict masking (montage encoded again). TimeLens-8B, 14K visual tokens for each pass, ablation set (Ego4D-NLQ dev and MomentSeeker long, 1,707 queries). The candidates are the first answer and 8 draws. The final accuracy is the mIoU of MaRS-Explore.
| Measure | Masked resampling (ours) | Strict masking |
|---|---|---|
| Distinct candidates (of 9) | 5.80 | 6.12 |
| Recall: a candidate overlaps the target (%) | 71.8 | 74.3 |
| Recall: a candidate has IoU ≥ 0.3 with the target (%) | 35.4 | 37.6 |
| Final mIoU, Ego4D-NLQ dev (%) | 18.9 | 17.9 |
| Final mIoU, MomentSeeker long (%) | 11.0 | 11.6 |
| Time for each draw (s) | 0.18 to 0.20 | 1.95 to 3.79 |
- Candidate diversity. Masked resampling keeps the diversity of strict masking. It gives 95% of the distinct candidates of strict masking, and 94% to 97% of its recall.
- Grounding accuracy. Strict masking does not increase the final accuracy. Over the two sets (1,707 queries), the final mIoU is 17.9 with masked resampling and 17.1 with strict masking.
- Computational cost. Masked resampling is 10 to 20 times faster for each draw. A masked draw uses the KV cache again, so its time does not increase with the token budget. A strict draw encodes the montage again, so its time increases from about 2 s at 14K tokens to about 3.8 s at 28K tokens.
Thus masked resampling gives the benefit of strict masking at a small part of its cost. We revised Section 3.2 to describe exactly what the masking does, and we added the strict variant and Table 14 to the appendix of the revised paper.
Computational fairness (Weakness 1) - Part 1
Response. Thank you for this comment. We measured the time of each step, with retrieval included, and we added a comparison at a matched grounding cost. Table 8 gives the results. We added Table 8 to the appendix of the revised paper.
Table 8. Cost per query. We use 87 videos that are longer than 17 minutes, with one query for each video: 8 from Ego4D-NLQ, 16 from MomentSeeker, 28 from VUE-TR and 35 from VUE-TR-V2. These videos have a length of 18 to 119 minutes, with a mean of 51 minutes. Time is GPU time on one A100 GPU. We do not include video decoding for any method.