acceptodds
Under review as a conference paper at ICLR 2027

Grounding All Moments in Long Videos via Retrieval, Reranking and Refinement

Abstract

Video temporal grounding aims to localize video segments that match a language query. However, existing methods typically assume a single occurrence per query, which is inadequate for long videos where the same event may appear multiple times. Existing benchmarks are also limited in this setting, as repeated valid moments are often missing from the annotations. To address this problem, we introduce AO-Bench, a human-refined benchmark for arbitrary-occurrence long video temporal grounding, where the goal is to localize all temporal spans that satisfy a query. We further propose a three-stage framework that retrieves candidate clips, reranks them, and refines their temporal boundaries. Our analysis shows that reranking is the key bottleneck, as it requires distinguishing true query-matching events from visually similar distractors. Therefore, we train the reranking module on AOVTG-66K generated by a dynamic multi-agent consensus process, in which multiple MLLM agents independently assess candidate clips, exchange feedback, and iteratively revise their decisions, while weakly supported or inconsistent predictions are filtered out to produce robust consensus labels. Experiments on AO-Bench show that ReGround achieves leading multi-occurrence grounding performance while remaining competitive in the single-occurrence setting, with lower inference cost than the multi-agent pipeline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long video temporal grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vieHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses arbitrary‑occurrence video temporal grounding for long‑form videos. The authors construct AO‑Bench, and propose a three‑stage Retrieve‑Rerank‑Refine pipeline. For the reranking stage, the authors propose Multi‑Agent Consensus Reranking (MACR), which uses multiple MLLM agents to evaluate candidate clips and aggregates their judgments.

Critical strengths

The human‑in‑the‑loop curation pipeline for AO‑Bench is carefully designed. The authors identify common annotation artifacts including missing event instances, boundary shifts and ambiguous queries, and provide inter‑annotator agreement statistics to support benchmark quality.

The distillation from heavy multi‑agent MACR into the lightweight ReGround model yields substantial speedup while retaining most multi‑occurrence performance benefits.

Critical weaknesses

The Retrieve‑Rerank‑Refine three‑stage paradigm is a standard coarse‑to‑fine pipeline widely adopted in prior works of long‑video temporal grounding or temporal action localization, e.g., [1-4]. The proposed method lacks technical novelity.

[1] Xiong, Zeyu, et al. "Rethinking video sentence grounding from a tracking perspective with memory network and masked attention." IEEE Transactions on Multimedia 26 (2024): 11204-11218.

[2] Fang, Xiang, et al. "Hierarchical local-global transformer for temporal sentence grounding." IEEE Transactions on Multimedia 26 (2023): 3263-3277.

[3] Long, Fuchen, et al. "Coarse-to-fine localization of temporal action proposals." IEEE Transactions on Multimedia 22.6 (2019): 1577-1590.

[4] Hu, Yufan, et al. "Learning proposal-aware re-ranking for weakly-supervised temporal action localization." IEEE Transactions on Circuits and Systems for Video Technology 34.1 (2023): 207-220.

Related works are sufficient. The authors only shortly survey long video temporal grounding benchmarks and methods. For methods, they simply category existing methods into traditional methods and MLLM-based methods. Is there really a fundamental difference between methods that use MLLMs and those that do not? Can the innovation of a previously published paper simply be reimplemented using an MLLM and then published as a new paper?

The computational complexity and inference time of the proposed model are not compared with those of existing methods.

Since the method relies on an MLLM, its generalization ability should be carefully evaluated. However, the paper does not provide any cross-dataset generalization experiments.

No ablation studying how instance granularity affects benchmark metrics and model ranking.

No comparison against simple baselines: e.g., take existing strong VTG models and let them output all high‑confidence proposals without modification.

There is insufficient analysis on how retrieval K balances recall, computation and downstream reranking load for arbitrary‑occurrence scenarios.

The arbitrary‑occurrence problem formulation is questionable and lacks well‑defined task boundaries. The core claim is that existing video temporal grounding benchmarks assume exactly one ground‑truth segment per query, which fails for real‑world long videos where events repeat. However, standard video temporal grounding never enforces a hard constraint that an event must occur exactly once. Most benchmarks simply only annotate one valid segment for each query due to annotation cost.

Questions for authors

What is the fundamental difference between Retrieve-Rerank-Refine and the extensive existing work on coarse-to-fine long-video temporal grounding and temporal action localization?

Does the construction of AO-Bench provide meaningful value to the field?

3 comments
AuthorsOpenReview ↗

Gentle Reminder for Your Feedback

Dear Reviewer vieH,

Thank you for your time and effort in reviewing our submission. We have carefully considered your comments and provided detailed responses. We look forward to your feedback.

Thank you for the authors' rebuttal, which addresses some of my concerns. I hope the above discussion and clarifications can be incorporated into the revised manuscript. I raise my score from 1 to 2.

Thank you for reading our rebuttal and for raising your score. We will incorporate the clarifications and additional results into the revised manuscript, specifically clearer contribution and task descriptions, expanded related work, and the additional experimental analyses.

To make the best use of the remaining discussion period, could you let us know if there are any remaining concerns that are not yet fully resolved?

Reviewer_eRwjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces a new benchmark for arbitraty-occurence video grounding tasks with long time duration. It proposes a method with 3 stages which can ground, rerank and refine potentiel clips, achieving leading performance.

Critical strengths

It addresses the issue of "a query potentially corresponding to multiple segments" in long videos, and creates a manually refined benchmark, AO-Bench.

Critical weaknesses

The novelty of the method is limitted.

Questions for authors

How does this work inspire related works and future studies?

Reviewer_4TCWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates arbitrary-occurrence video temporal grounding, which aims to localize all valid occurrences of a queried event in long videos. To address the incomplete annotations in existing single-occurrence benchmarks, the authors introduce AO-Bench, a human-refined benchmark constructed from Ego4D-NLQ and MomentSeeker with multiple temporal annotations per query. The paper further proposes a three-stage Retrieve-Rerank-Refine framework and identifies reranking as a key performance bottleneck through oracle-based analysis. To improve reranking, the authors introduce Multi-Agent Consensus Reranking (MACR), which combines judgments from multiple MLLM agents to generate reliable relevance labels and temporal boundaries. These annotations are used to construct AOVTG-66K, an instruction-tuning dataset for training ReGround, a unified model that performs reranking and temporal refinement efficiently.

Critical strengths

1. Important and well-motivated problem.
The paper addresses a meaningful limitation of existing long-video temporal grounding benchmarks, where incomplete annotations of recurring events may lead to incorrect evaluation penalties. The proposed arbitrary-occurrence grounding setting is practically relevant and provides a useful direction for evaluating long-video understanding beyond the conventional single-occurrence assumption.


2. Systematic benchmark construction.
The authors construct AO-Bench by manually auditing and refining existing annotations from Ego4D-NLQ and MomentSeeker. The annotation process addresses multiple types of errors, including missing occurrences, inaccurate temporal boundaries, and ambiguous queries. The inclusion of both repetitive egocentric activities and diverse video domains, along with explicit annotation guidelines and inter-annotator agreement statistics, is a valuable aspect of the benchmark construction.


3. Extensive experiments and insightful component-level analysis.
The paper provides comprehensive evaluations of existing VTG models and MLLMs under both single- and multi-occurrence settings. In particular, the oracle-based analysis of the retrieval, reranking, and refinement stages provides useful insights into the limitations of the evaluated grounding pipeline. The proposed MACR framework and its distillation into ReGround further demonstrate a practical approach to improving multi-occurrence grounding performance with reduced inference cost.

Critical weaknesses

1. Insufficient validation of annotation completeness in AO-Bench.

AO-Bench relies on a human-in-the-loop annotation process involving only two annotators to identify missing occurrences and refine existing annotations. However, it is unclear whether this process is sufficient to ensure that all valid moments are exhaustively annotated, especially given the relatively long video durations. Although the authors report inter-annotator agreement for error classification and temporal boundaries, these metrics do not directly measure annotation completeness. In particular, both annotators could miss the same valid occurrences.

Since exhaustive annotation is a central contribution of AO-Bench, more details about the annotation protocol and additional validation of occurrence-level completeness would strengthen the reliability of the benchmark. For example, an independent re-annotation audit on a subset of videos could help estimate the extent of remaining missing occurrences.


2. Potential retrieval-induced selection bias in AOVTG-66K.

While the oracle analysis in Table 3 provides evidence that reranking is the primary bottleneck in the evaluated pipeline, I have a separate concern regarding the construction of AOVTG-66K. The training dataset is generated by applying MACR to the top-10 candidate clips retrieved by UniTime. However, Table 7 shows that UniTime still misses a substantial portion of valid occurrences in multi-occurrence settings (under top-5 retrieval).

Since MACR can only annotate retrieved candidates, difficult positive occurrences that are consistently missed by UniTime may be underrepresented in the training data. This could introduce a selection bias toward easily retrievable moments, potentially limiting the diversity and generalizability of the learned supervision. Although improving retrieval may not be the primary bottleneck for the evaluated inference pipeline, it remains important to assess retrieval coverage during training data construction.

Questions for authors
  1. Annotation completeness of AO-Bench.
    Given that AO-Bench relies on a human-in-the-loop annotation process involving two annotators, how do the authors ensure that all valid occurrences are exhaustively annotated? Could the authors clarify whether the annotators independently identified occurrences, how disagreements were resolved, and whether any additional validation was conducted to assess annotation completeness?

  1. Sufficiency of UniTime for AOVTG-66K construction.
    While the oracle analysis suggests that reranking is the primary bottleneck, I am concerned about the reliance on a single retriever during training data construction. Table 7 shows that UniTime misses a substantial portion of valid occurrences in multi-occurrence settings. Is UniTime's top-10 retrieval sufficient to provide diverse and representative training samples for AOVTG-66K? Have the authors investigated whether combining candidates from multiple retrievers could improve the coverage of challenging positive moments, compared to relying solely on UniTime?

  1. Qualitative differences between single- and multi-occurrence supervision.
    Beyond predicting a different number of moments, I am curious whether models trained on the same video sources with single-occurrence versus multi-occurrence annotations exhibit qualitatively different prediction behaviors. For instance, do they show different preferences for particular occurrences, such as earlier, more salient, or visually distinctive moments? Such an analysis could provide additional insights into how annotation strategies affect model behavior beyond the number of predicted moments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.