acceptodds
Under review as a conference paper at ICLR 2027

Zero- and Multi-Event Temporal Grounding with Multimodal Large Language Model

Abstract

Video Temporal Grounding (VTG) aims to localize video segments matching a text query. Conventional VTG methods assume that each query corresponds to exactly one segment, an assumption violated in real-world scenarios where events may occur zero, once, or multiple times. We address this gap by formalizing Any-Moment Temporal Grounding (AMTG), which requires models to predict an arbitrary number of temporal segments, including none. To facilitate research on AMTG, we construct AnyMoment dataset, which includes queries with zero, single, and multiple ground-truth segments. Recently, reinforcement learning (RL) has emerged as an effective paradigm for optimizing multimodal large language models (MLLMs) for VTG. However, AMTG poses two additional challenges: recognizing distinct occurrences of the queried event from complex video content, and providing fine-grained learning signals for individual event predictions, as conventional sequence-level RL objectives assign a single advantage to the entire response and may therefore penalize correct predictions together with incorrect ones. To address these challenges, we propose Event-wise Grounding Optimization (EGO), an RL-based framework for AMTG. First, EGO employs a bivariate difficulty-aware sampling strategy that jointly characterizes the difficulty of recognizing event occurrences and locating their temporal extents, enabling RL to focus on samples that are challenging yet learnable. Second, we propose an occurrence-aware chain-of-thought reasoning strategy that identifies fine-grained descriptions of individual event occurrences and transforms them into discriminative subqueries, facilitating the localization of multiple similar or closely related events. Third, we introduce fine-grained event-level credit assignment, which computes event-specific advantages within each rollout group and assigns them to the corresponding output tokens, explicitly reinforcing correct event predictions while penalizing erroneous ones. Experiments show that EGO achieves state-of-the-art performance on AnyMoment and public multi-segment VTG benchmarks, demonstrating its effectiveness across diverse temporal grounding scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Xh6wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies video temporal grounding where a text query may correspond to zero, one, or multiple video segments. It introduces AnyMoment, a dataset covering all three cases, and proposes Event-wise Grounding Optimization (EGO), a reinforcement learning framework for this setting. EGO combines difficulty-aware data sampling, occurrence-specific reasoning, and event-level credit assignment to provide separate learning signals for individual predictions within a response. Experiments on AnyMoment and existing benchmarks show improvements in temporal localization and rejection of queries with no matching events, with event-level optimization particularly benefiting queries involving multiple occurrences.

Critical strengths
  • Practical problem setting. Supporting zero, one, or more matching segments makes temporal grounding more applicable to real-world videos.

  • Useful dataset contribution. AnyMoment includes both recurring events and challenging negative queries, enabling evaluation of localization and rejection. (This can be considered as a contribution only if the authors will release their dataset to the public.)

  • Well-motivated method. Event-level credit assignment addresses the limitation of applying a shared learning signal to correct and incorrect predictions within the same response.

  • Broad evaluation. Experiments cover multiple backbones and benchmarks, with ablations showing that event-level optimization is particularly helpful for queries with more occurrences

Critical weaknesses
  1. Limited coverage of recent strong baselines. The evaluation would benefit from comparisons with newer Gemini Flash/Pro and Seed models, particularly on AnyMoment, where no proprietary-model results are reported. These comparisons would help establish the benchmark’s difficulty and better contextualize EGO’s performance.

  2. Incomplete baseline inference details. Video input settings, decoding parameters, and output parsing rules are not fully specified for all baselines. Instructions for rejecting absent events also differ across prompts, raising concerns about reproducibility and comparability.

  3. Insufficient separation of data and methodological gains. Most baselines are not reported to be retrained on the same data, and the MUSEG comparison using the proposed data lacks training details. This makes it difficult to isolate the contribution of the proposed method.

  4. Limited transparency in data quality validation. The scope and sample size of human verification, as well as disagreement resolution, are unclear. Query and temporal annotation errors are not separately analyzed, and the relationship between overall and subgroup accuracy statistics requires clarification.

Questions for authors
  1. The paper lacks domain, action, and scene distributions, making it difficult to assess AnyMoment’s semantic coverage and potential bias toward specific repetitive actions.

  2. The dataset is claimed as a contribution. If the authors are not going to release the dataset, it is inappropriate to claim it as a contribution in my view.

4 comments

We thank the reviewer for recognizing the practical zero-, single-, and multi-event setting, the value of recurring events and challenging negative samples in AnyMoment, and the motivation for event-level credit assignment. We also appreciate your positive assessment of the evaluation across multiple backbones and benchmarks, especially the gains for samples with more occurrences.

W1: Recent Gemini Flash/Pro, GPT, and Seed baselines on AnyMoment and EGO's relative performance

11 Thank you for requesting these comparisons. We have added AnyMoment results for Seed-2.0-Lite, GPT-6.1-sol, Gemini-3.1-pro, and Gemini-3.8-flash. We also compare with MUSEG-7B, the second-best baseline that specialized multi-event grounding method. Our EGO-8B leads mIoU and [email protected] on both datasets and [email protected] on HACS. Gemini-3.1-pro has the highest [email protected] on InternVid, while GPT-6.1-sol has the highest RejF1 on both datasets. This comparison highlights EGO's localization strength alongside the strong rejection performance of frontier models.

ModelHACS mIoUHACS [email protected]HACS [email protected]HACS RejF1InternVid mIoUInternVid [email protected]InternVid [email protected]InternVid RejF1
Seed-2.0-Lite39.2147.1034.4245.4250.9961.6840.8768.82
GPT-6.1-sol34.4439.5924.9183.7854.1963.0947.9795.77
Gemini-3.1-pro36.2441.0226.2178.2655.9365.1949.2192.02
Gemini-3.8-flash40.1047.6830.7080.5655.1163.7447.6494.98
MUSEG-7B33.6039.5023.200.0040.2044.3024.900.30
EGO-8B (Ours)43.3050.0035.0075.0056.4064.2052.7093.50

22 We further compare the performance on multi-event samples. Multi-event grounding remains challenging: across the four strong frontier models, mIoU drops a lot on multi-event samples. The advantage of our EGO over the these frontier models is clearest in multi-event mIoU and [email protected]: it exceeds their strongest result in mIoU by 4.33 points on HACS and 2.58 on InternVid, and in [email protected] by 1.46 and 4.86 points, respectively. These results show that AnyMoment remains challenging even for frontier models, while EGO-8B is more effective at precisely localizing repeated event occurrences.

ModelHACS mIoUHACS [email protected]HACS [email protected]InternVid mIoUInternVid [email protected]InternVid [email protected]
Seed-2.0-Lite37.1645.0632.5442.7153.3833.59
GPT-6.1-sol32.2937.4723.1245.0754.9239.09
Gemini-3.1-pro34.0439.3024.1648.0460.1943.34
Gemini-3.8-flash37.7145.4628.7245.9454.6638.22
MUSEG-7B32.0937.6622.0434.8138.3221.72
EGO-8B (Ours)42.0448.9634.0050.6258.7248.20

W2: Baseline video inputs, decoding parameters, output parsing, and model-specific rejection instructions

11 Video inputs and decoding parameters. All the baselines and our models samples videos at 2 FPS, caps the input at 768 frames, and dynamically adjusts resolution to a budget of approximately 3,584 visual tokens, for a fair comparasion. Generation uses temperature=0, top-p = 1.0, repetition penalty = 1.05, and a maximum of 1,024 output tokens.

22 Output parsing. We preserve each grounding baseline's native training-time output format and parse its predicted temporal intervals using format-specific regular expressions. The parsing is robust across these native formats: no evaluated sample from any model was assigned a zero score due to parsing failure.

(3) Rejection instructions. Different methods have different default prompts and output formats. We preserve each method's native prompt structure, try alternative rejection instructions, and retain the stronger-performing version for that method. Consequently, the wording differs, while the task remains to return an empty prediction when the event is absent and all matching intervals when it is present. The selected native templates are described in Appendix E.4.

W3: Separating data and method gains: MUSEG retraining with matched data and budgets

We use our data to retrain MUSEG-7B, the second-best baseline that specializes in multi-event grounding. Our EGO and MUSEG w/ our data use identical training data and budgets, with the Qwen2.5-VL-7B backbone. SFT uses 46,940 AnyMoment samples; RL uses 7,500 samples: 2,500 HACS, 2,500 InternVid, and 2,500 Time-R1. Following Appendix C.4–C.5, SFT runs for one epoch with batch size 32 and learning rate 2×10⁻⁴; RL runs for two epochs with batch size 16, eight rollouts, and learning rate 10⁻⁶.

To further demonstrate the contribution of the proposed method, we additionally train Time-R1 with our data under the same training budgets. Comparing the original baselines with their w/ our data variants demonstrates the dataset's contribution, while EGO's further gains over both matched-data baselines demonstrate the method's contribution.

ModelHACS mIoUHACS [email protected]HACS [email protected]HACS RejF1InternVid mIoUInternVid [email protected]InternVid [email protected]InternVid RejF1QV-M2 mIoUQV-M2 [email protected]QV-M2 [email protected]
Time-R1-7B21.7026.7015.3018.4034.1037.7021.3032.1025.5032.2022.10
Time-R1-7B w/ our data34.6040.8022.1061.9044.6043.3026.9078.8031.9037.8028.60
MUSEG-7B33.6039.5023.200.0040.2044.3024.900.3031.0036.2026.60
MUSEG-7B w/ our data35.3041.9025.7024.8043.5047.5030.7055.2033.4038.6029.30
EGO-7B (Ours)41.3047.9031.5075.4050.4057.6042.4087.1038.0046.0032.90

Finally, the OMTG experiment in Table 3 also supports the effectiveness of our method beyond AnyMoment training data. Using the same OMTG-46K dataset and Qwen3-VL-4B backbone, our EGO-4B outperforms OMTG-4B on all reported metrics.

MethodC-Acc[email protected][email protected][email protected]mIoU
OMTG-4B55.6373.4665.4048.9661.24
EGO-4B (Ours)59.0675.7267.0350.8662.43

W4: Human-verification scope, disagreement resolution, query/temporal errors, and overall versus subgroup accuracy

11 The scope and sample size of human verification and disagreement resolution. All 6,200 test samples are human-verified. Annotators receive machine labels as a reference and inspect the video and annotations. They remove event annotations that do not match the query, adjust inaccurate temporal boundaries, and add missing matching events with their timestamps. Human judgments always take precedence when they disagree with MLLM annotations. A second annotator double-checks the resulting annotations. Any problems identified lead to reannotation, with the reannotated results used as the final labels. In practice, 99.8% of the non-zero-event test samples have their automatic event boundaries adjusted by humans.

22 Query and temporal annotation errors. Under our task definition, there is no separate query-error category. If a query does not match the video, annotators leave the query unchanged and correct the temporal annotation to an empty set. Such corrected cases account for 3.5% of samples.

33 Relationship between overall and subgroup accuracy. The 87% overall automatic-annotation accuracy and the target-type statistics refer to AnyMoment-InternVid. The reported rates are 96.5%, 82.6%, and 80.6% for zero-, single-, and multi-event samples. Weighting these rounded rates by InternVid's test composition 1,112zero−,788single−,and1,100multi−eventsamples1,112 zero-, 788 single-, and 1,100 multi-event samples gives approximately 87%, consistent with the overall estimate. HACS's source-specific accuracy is a separate statistic.

Q1: Domain, action, and scene distributions, semantic coverage, and bias toward repetitive actions

We supplement the source-wise domain and scene distributions below, alongside a summary of HACS action coverage. We acknowledge a bias toward human activities and repeatable actions. Our collection prioritizes repeated occurrences, which are readily available in human-activity videos such as sports and everyday routines.

Domain distribution

CategoryHACSInternVid
Human activities99.16%72.40%
Animal activities0.12%8.10%
Transportation and machinery0.50%7.60%
Objects and static displays0.22%10.47%
Other0.00%1.43%

Scene distribution

CategoryHACSInternVid
Indoor20.67%33.80%
Urban and streets3.80%10.10%
Rural and farmland0.44%4.63%
Natural10.89%15.23%
Sports42.74%15.44%
Water environments13.26%13.83%
Other8.20%6.97%

HACS action distribution. Our data provide broad coverage of HACS's original 200 action labels. We will include the detailed action-frequency distribution figure in the revised paper.

Q2: Public release of AnyMoment and the EGO method

Upon acceptance, we will publicly release the AnyMoment dataset, EGO models, and training/evaluation code.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewer Xh6w,

Thank you for your thoughtful review and constructive feedback. As the discussion period is approaching its end, we would greatly appreciate any feedback on our rebuttal. Please let us know if you have any remaining concerns, and we would be glad to discuss them further.

Best regards,

The Authors

Reviewer_URsnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AnyMoment, a dataset covering zero-, single-, and multi-event temporal grounding, and EGO, a training framework combining difficulty-based sampling, occurrence-aware reasoning, and event-level credit assignment. EGO matches predictions to ground-truth events across rollouts and combines local and global advantages. Experiments evaluate localization and rejection on AnyMoment, with additional evaluations on QV-M2, OMTG Bench, and Charades-TimeLens.

Critical strengths
  1. Useful coverage of absent and repeated events. AnyMoment jointly evaluates rejection and localization, addressing an important limitation of positive-only grounding benchmarks. Its manually inspected test set provides a useful resource for studying absent and repeated events.
  2. A well-motivated structural modification with relevant empirical evidence. Table 6 shows [email protected] improvements of 4.1 and 5.5 points for queries with three and at least four events, respectively. These results support the motivation for finer-grained feedback in multi-event prediction.
Critical weaknesses
  1. Limited novelty. Supporting zero events is useful, but the paper itself acknowledges earlier zero/multi-event settings. Occurrence-aware descriptions are also closely related to OMTG’s query-guided captions and explicit discriminability objective. More broadly, fine-grained advantages are established: Zoom-Zero, TimeThink, Credit the Right Box. The distinctive design appears to be ground-truth-event-aligned normalization across rollouts, together with unmatched/missed-event handling. Similarly, the CoT/no-CoT comparison does not establish the added value of occurrence-specific subqueries over existing descriptive reasoning.

  2. Insufficient validation of the central mechanism. Table 5 shows that local advantages add only 0.3 mIoU and 0.6 [email protected] over the global-only configuration. Although the high-event-count gains are encouraging, the core ablations cover one backbone and one dataset, without uncertainty estimates. More importantly, the global-only comparison establishes the benefit of adding local feedback, but does not isolate the benefit of event-specific grouping. The comparison with OMTG also changes the training recipe, limiting attribution of the improvement to the proposed optimization. Consequently, the evidence for the central algorithmic contribution remains inconclusive.

Questions for authors
  1. Experimental clarification: Please clarify the differences in SFT/RL data and initialization underlying Table 3, and whether an otherwise matched global-only EGO-4B result is available. For Table 6, please provide subgroup sizes and any available uncertainty estimates to assess the reliability of the reported gains.
  2. Independent contribution of occurrence-aware reasoning. What evidence shows that occurrence-specific subqueries improve grounding beyond descriptive CoT of the type used in OMTG? The CoT/no-CoT ablation does not distinguish these alternatives. Is a comparison available with the training data and RL objective held fixed?
  3. Rejection–localization trade-off. Appendix A.2 acknowledges that EGO sometimes rejects queries whose events are present. What are the false-rejection rates for single- and multi-event queries, alongside rejection precision and recall? These results would clarify whether improved rejection is achieved at a meaningful cost to retrieval of valid events.
6 comments

We thank the reviewer for the thoughtful assessment and for recognizing AnyMoment's useful coverage of absent and repeated events and the value of its manually inspected test set. We also appreciate your recognition that the event-count-stratified results in Table 6 support the motivation for finer-grained feedback in multi-event prediction.

W1: Novelty beyond earlier zero/multi-event settings

Our novelty lies in two aspects: the dataset and the method.

11 Data contribution. We introduce AnyMoment for zero-, single-, and multi-event temporal grounding, with rigorous quality control and a fully human-verified test set. Most of exsiting datasets lack queries with zero matching events, which leaves models without the ability to reject non-matching results. To our knowledge, the only earlier dataset jointly covering zero and multiple events, NExT-VMR, is not publicly available, limiting community research. We commit to publicly releasing AnyMoment upon acceptance.

22 Method contribution. Existing methods for multi-event grounding uses response-level RL optimization that assigns a shared advantage to all event predictions in an answer, even when some are correct and others are incorrect. We propose the event-level credit assignment that provides fine-grained learning signals for individual predictions, along with a bivariate difficulty-aware sampling and a occurrence-aware CoT that generates discriminative subqueries for localization.

W1, Q2: Novelty and independent contribution of occurrence-specific subqueries beyond OMTG-style descriptive CoT, with the training data and RL objective held fixed

11 OMTG's query-guided CoT densely describes the video without requiring every GT occurrence to be explicitly represented, so its reasoning can contain substantial content unrelated to the localization targets. For comparison, selecting the caption with the highest temporal IoU for each GT event in OMTG's CoT SFT data yields only 50.54% mIoU and 24.35% [email protected], whereas our GT-paired subqueries achieve 100% on both alignment metrics.

22 Our CoT SFT construction starts from each target GT event and specializes the original query with observable discriminative details to distinguish that occurrence from the others. This reduces ambiguity between repeated occurrences, and avoids unnecessary distractions and output from descriptions of unrelated video content.

33 We provide a direct comparison with the training data and RL objective held fixed, replacing occurrence-specific CoT with OMTG-style descriptive CoT. This controlled comparison uses AnyMoment-InternVid, the same Qwen2.5-VL-7B backbone and training budgets, and identical video-query training samples and ground-truth temporal annotations. Occurrence-specific CoT improves mIoU, [email protected], and [email protected] over OMTG-style descriptive CoT while reducing average output length from 474 to 105 tokens. This directly establishes its added value beyond the CoT/no-CoT comparison.

CoT formatmIoU[email protected][email protected]Avg. CoT tokens
OMTG-style CoT49.5856.3841.18474
Occurrence-specific CoT (Ours)50.4057.6042.40105

W1,W2: Novelty of GT-event grouping across rollouts and fine-grained credit assignment versus Zoom-Zero, TimeThink, and Credit the Right Box

We agree that fine-grained credit assignment is established. As the reviewer identifies, EGO's contribution is GT-event-aligned comparison across rollouts, with explicit handling of missed and unmatched predictions. In multi-event temporal grounding, rollouts predict different numbers of intervals at different temporal positions. GT matching establishes event correspondence across rollouts, and normalization within each GT group compares alternative predictions of the same event. This matters because event difficulty varies: combining different events in a single comparison group can favor easy events and give negative credit to an improved hard-event prediction whose absolute IoU remains below that group's mean.

11 Zoom-Zero separates advantages by reward type and assigns them to grounding versus answering tokens, where the grounding reward evaluates a single target event interval.

22 TimeThink scores temporal clues by their maximum IoU with GT evidence, but normalizes them in a single comparison group across events and rollouts. This can give improved predictions of difficult events low or negative local credit; missed GT events receive no explicit local credit.

33 Credit the Right Box, a concurrent work first posted in August 2026, handles variable-size object sets. Its MCR-GRPO normalizes predicted records' leave-one-out marginal contributions within each response. This compares different targets within the same rollout and can favor easy ones. EGO instead compares alternative predictions of the same GT event across rollouts.

Zoom-Zero does not define credit assignment among multiple temporal predictions, so we cannot directly compare its credit scheme in this multi-event setting. For TimeThink and Credit the Right Box, we adapt their credit assignment schemes to our temporal grounding task and compare them with our GT-event grouping. For analysis, we stratify AnyMoment-InternVid test samples by the SFT model's [email protected], using performance before RL as a measure of initial localization difficulty. EGO achieves the best overall localization scores, with its largest gains in the lowest-SFT-performance group. These results support the benefit of GT-aligned comparison, especially for difficult samples.

Local credit assignmentmIoU[email protected][email protected]
MCR-GRPO49.0756.8141.94
TimeThink-style49.0356.7340.90
EGO (Ours)50.4057.6042.40
SFT [email protected] groupLocal credit assignmentmIoU[email protected][email protected]
< 0.3MCR-GRPO32.1730.9018.82
< 0.3TimeThink-style31.4729.8817.59
< 0.3EGO (Ours)33.7833.7619.60
03˙,07˙0\.3, 0\.7
| MCR\-GRPO               | 45\.17     | 56\.78     | 39\.51     |

|

03˙,07˙0\.3, 0\.7
| TimeThink\-style        | 45\.29     | **56\.91** | 38\.71     |

|

03˙,07˙0\.3, 0\.7
| EGO \(Ours\)            | **45\.88** | 56\.44     | **39\.95** |

| > 0.7 | MCR-GRPO | 70.48 | 85.21 | 69.05 | | > 0.7 | TimeThink-style | 71.04 | 85.99 | 68.05 | | > 0.7 | EGO (Ours) | 71.82 | 84.69 | 69.15 |

W2,Q1: Local-advantage ablations on other datasets and backbones, including global-only EGO-4B

We add ablations on another OMTG Bench and QV-M2 and extend the AnyMoment-InternVid ablation to the Qwen3-VL-8B backbone. The table below compares performance with and without local advantages, including the matched global-only EGO-4B result on OMTG. Local advantages improve [email protected] and [email protected] in all five comparisons, extending the evidence for event-level feedback beyond the original backbone and dataset. Our gain from local advantages is more pronounced on OMTG, since the number of all GT events on OMTG is greater than or equal to 2.

Dataset / modelLocal advantagesmIoU[email protected][email protected]
AnyMoment-InternVid Qwen25˙−VL−7BQwen2\.5-VL-7Bw/o50.1257.0641.00
AnyMoment-InternVid Qwen25˙−VL−7BQwen2\.5-VL-7Bw/ (Ours)50.3557.6542.39
QV-M2 Qwen25˙−VL−7BQwen2\.5-VL-7Bw/o37.6644.7332.12
QV-M2 Qwen25˙−VL−7BQwen2\.5-VL-7Bw/ (Ours)38.0145.9532.93
AnyMoment-InternVid Qwen3−VL−8BQwen3-VL-8Bw/o55.1663.6549.59
AnyMoment-InternVid Qwen3−VL−8BQwen3-VL-8Bw/ (Ours)56.4064.2552.70
QV-M2 Qwen3−VL−8BQwen3-VL-8Bw/o41.9251.2335.97
QV-M2 Qwen3−VL−8BQwen3-VL-8Bw/ (Ours)42.8151.8736.32
OMTG Bench Qwen3−VL−4BQwen3-VL-4Bw/o61.8665.3648.41
OMTG Bench Qwen3−VL−4BQwen3-VL-4Bw/ (Ours)62.4367.0350.86

W2,Q1: Table 6 subgroup sizes and uncertainty by event count

We use paired bootstrap to assess uncertainty in performance gains. The table below reports the subgroup sizes and the 95% percentile intervals gain[95gain [95% CI] from 10,000 paired bootstrap resamples by GT event count for Qwen2.5-VL-7B on AnyMoment-InternVid, where a larger positive gain indicates a larger improvement and interval width reflects uncertainty. The [email protected] intervals are entirely positive for samples with 3 and ≥ 4 events, supporting the benefit of local feedback on samples containing more occurrences.

DatasetGT eventsSamplesΔmIoU [95% CI]Δ[email protected] [95% CI]Δ[email protected] [95% CI]
AnyMoment-InternVid1788+0.01
−12˙2,+12˙6-1\.22, +1\.26

| -0.99

−32˙6,+13˙9-3\.26, +1\.39

| +0.70

−16˙8,+31˙1-1\.68, +3\.11

| | AnyMoment-InternVid | 2 | 800 | -0.46

−15˙3,+06˙1-1\.53, +0\.61

| +0.67

−12˙3,+25˙9-1\.23, +2\.59

| +1.40

−04˙7,+32˙8-0\.47, +3\.28

| | AnyMoment-InternVid | 3 | 200 | +2.66

+09˙2,+44˙3+0\.92, +4\.43

| +4.09

+12˙6,+70˙8+1\.26, +7\.08

| +3.56

+01˙4,+70˙7+0\.14, +7\.07

| | AnyMoment-InternVid | ≥ 4 | 100 | +2.73

+00˙2,+53˙5+0\.02, +5\.35

| +5.48

+09˙8,+98˙9+0\.98, +9\.89

| +2.32

−15˙4,+62˙3-1\.54, +6\.23

|

Reviewer_DZk5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies video temporal grounding when a query may have zero, one, or multiple matching occurrences. It introduces AnyMoment, with 51,940 training queries and 6,200 manually inspected test queries. The proposed EGO framework selects training examples using count and localization difficulty, structures reasoning around occurrence-specific subqueries, and combines event-level advantages with response-level advantages in GRPO. Experiments cover the two AnyMoment subsets, QV-M2, OMTG Bench, and Charades-TimeLens. The paper reports improvements in localization and rejection, and examines the contribution of local advantages through ablations stratified by event count.

Critical strengths
  1. The event-level feedback is supported by a relevant diagnostic. In Table 6, adding local advantages improves [email protected] from 49.1 to 53.2 for three-event queries and from 46.6 to 52.1 for queries with at least four events; single-event [email protected] changes from 67.5 to 66.5. This pattern supports the central motivation more directly than an overall benchmark improvement: finer feedback is useful when a response contains several predictions of different quality.

  2. The dataset addresses a useful evaluation need. AnyMoment combines semantically close negative queries with single- and multi-event queries, draws on complementary content from HACS and InternVid, and includes a manually inspected test set. Retraining MUSEG with AnyMoment already improves its results in Table 2. This suggests that the resource can help existing methods adapt to the setting and has value beyond EGO itself.

Critical weaknesses
  1. Dataset reliability and difficulty need stronger evidence. The paper reports human inspection and double annotation, but coverage, correction of missing events, and adjudication of disagreements between annotators or between humans and the MLLM remain unclear. Omission and incorrect-annotation rates are not reported separately. AnyMoment also lacks Gemini-3-Pro, GPT, and Seed results; Table 3 compares Gemini and Seed only on OMTG. These gaps limit assessment of label reliability, benchmark difficulty, and EGO's advantage.

  2. Attributing the overall gain requires comparable training conditions. Table 2 includes retrained MUSEG, but C.5 mentions additional Time-R1 data without specifying whether the full training data and budgets are matched. Ablations already support event-level feedback; the remaining issue is how much of the overall EGO advantage is attributable to the method itself.

Questions for authors
  1. How thorough was human verification of the test labels? Please specify the coverage of independent double annotation and whether annotators saw machine labels. How were disagreements between annotators, and between human and MLLM annotations, adjudicated and the final labels revised? In particular, how were missing events identified and corrected? Report counts and rates for omissions, incorrect annotations, and both types of disagreement, specifying the denominators.

  2. How do strong general-purpose models perform? Please report AnyMoment results for Gemini-3-Pro, GPT, and Seed, broken down by zero-, single-, and multi-event queries, to establish benchmark difficulty and EGO's relative advantage.

  3. Attribution of gains. Please specify Time-R1 usage, amounts, and training budgets for EGO, retrained MUSEG, and the ablations. Under matched data and training conditions, what gains come from occurrence-specific versus ordinary CoT, and what changes when only the rejection reward is removed while retaining the same negative samples?

8 comments

We thank the reviewer for recognizing the diagnostic evidence supporting event-level feedback and the value of AnyMoment's semantically close negatives, complementary data sources, and manually inspected test set, including its usefulness beyond EGO demonstrated by retraining MUSEG.

W1,Q1: Dataset reliability and how thorough was human verification of the test labels?

11 Human verification, machine-label visibility, and annotation correction. All test samples are human-verified. Annotators receive machine labels as a reference and inspect the video and annotations. They remove event annotations that do not match the query, adjust inaccurate temporal boundaries, and add missing matching events with their timestamps. A second annotator double-checks the labels; any detected errors lead to reannotation, with the corrected results used as the final labels. In practice, 99.8% of non-zero-event test samples have their automatic event boundaries adjusted by humans. On AnyMoment-InternVid, overall automatic-annotation accuracy is 87%, using tIoU ≥ 0.7 for non-zero-event samples and correct absence decisions for zero-event samples. Most automatic annotations are already accurate, and human verification further refines temporal boundaries and corrects identified errors.

22 Disagreements between annotators and between human and MLLM annotations. Human–MLLM disagreements are resolved in favor of human judgments, and the automatic labels are revised accordingly. Human–human disagreements are resolved through the second-annotator double-check and reannotation procedure described in 11, with the corrected annotations retained as the final labels.

33 Clarification of independent double annotation. The independent double annotation described in Appendix B.3, Table 10, was conducted on 200 randomly sampled test samples to evaluate inter-annotator agreement. Two annotators independently labeled these samples. This agreement experiment covers only a subset of the test data, and all test samples undergo the human annotation and subsequent double-checking procedure described above.

44 Human–machine omission and incorrect-annotation rates. On AnyMoment-InternVid, comparing the initial automatic annotations with the final human-revised labels identifies omissions in 8.9% and incorrect annotations in 4.1% of test samples.

W1,Q2: Report AnyMoment results for Gemini-3-Pro, GPT, and Seed to establish benchmark difficulty and EGO's relative advantage.

Thank you for requesting these comparisons. We have added AnyMoment results for Seed-2.0-Lite, GPT-6.1-sol, Gemini-3.1-pro, and Gemini-3.8-flash. We also compare with MUSEG-7B, the second-best baseline that specializes in multi-event grounding.

11 Overall localization and zero-event rejection. EGO-8B leads mIoU and [email protected] on both datasets and [email protected] on HACS. Gemini-3.1-pro has the highest [email protected] on InternVid, while GPT-6.1-sol has the highest RejF1 on both datasets. This comparison highlights EGO's localization strength alongside the strong rejection performance of frontier models.

ModelHACS mIoUHACS [email protected]HACS [email protected]HACS RejF1InternVid mIoUInternVid [email protected]InternVid [email protected]InternVid RejF1
Seed-2.0-Lite39.2147.1034.4245.4250.9961.6840.8768.82
GPT-6.1-sol34.4439.5924.9183.7854.1963.0947.9795.77
Gemini-3.1-pro36.2441.0226.2178.2655.9365.1949.2192.02
Gemini-3.8-flash40.1047.6830.7080.5655.1163.7447.6494.98
MUSEG-7B33.6039.5023.200.0040.2044.3024.900.30
EGO-8B (Ours)43.3050.0035.0075.0056.4064.2052.7093.50

22 Single-event samples. Frontier general-purpose models perform strongly on single-event samples, while our EGO-8B still outperforms MUSEG-7B on all three single-event localization metrics on both datasets.

ModelHACS mIoUHACS [email protected]HACS [email protected]InternVid mIoUInternVid [email protected]InternVid [email protected]
Seed-2.0-Lite58.8466.5252.2462.5573.2651.04
GPT-6.1-sol55.3860.2642.3166.8974.4760.34
Gemini-3.1-pro57.6557.6946.1566.9172.1557.38
Gemini-3.8-flash63.3669.2350.0067.8876.3760.76
MUSEG-7B48.5356.8834.4047.6352.6929.37
EGO-8B (Ours)55.0260.0944.9164.4771.9758.98

33 Multi-event samples. Multi-event grounding remains challenging: across the four strong frontier models, mIoU drops by 18.87–25.65 points from single- to multi-event samples. The advantage of our EGO over the frontier models is clearest in multi-event mIoU and [email protected]: it exceeds their strongest result in mIoU by 4.33 points on HACS and 2.58 on InternVid, and in [email protected] by 1.46 and 4.86 points, respectively. These results show that AnyMoment remains challenging even for frontier models, while EGO-8B is more effective at precisely localizing repeated event occurrences.

ModelHACS mIoUHACS [email protected]HACS [email protected]InternVid mIoUInternVid [email protected]InternVid [email protected]
Seed-2.0-Lite37.1645.0632.5442.7153.3833.59
GPT-6.1-sol32.2937.4723.1245.0754.9239.09
Gemini-3.1-pro34.0439.3024.1648.0460.1943.34
Gemini-3.8-flash37.7145.4628.7245.9454.6638.22
MUSEG-7B32.0937.6622.0434.8138.3221.72
EGO-8B (Ours)42.0448.9634.0050.6258.7248.20

W2,Q3: Attribution of EGO's overall gains under comparable training conditions. Time-R1 usage and amounts, matched data and budgets for retrained baselines and ablations, occurrence-specific versus ordinary CoT, and removing only the rejection reward while retaining the same negative samples

Thank you for requesting these controlled comparisons. We clarify the training conditions and report the additional experiments below.

11 Time-R1 usage, matched training data and budgets, and baseline comparisons. Retrained MUSEG uses exactly the same data and training budgets as our EGO, including Time-R1 data. Both our EGO and retrained MUSEG use 46,940 AnyMoment samples for SFT and use 7,500 samples, including 2,500 AnyMoment-HACS, 2,500 AnyMoment-InternVid, and 2,500 Time-R1, for RL.

We additionally retrain Time-R1 with the same data and budgets. As shown in the table below, both Time-R1 and MUSEG improve over their original versions when trained with our data, demonstrating the contribution of AnyMoment. Under the same training data and budgets, EGO outperforms both retrained baselines, demonstrating the contribution of our method. Furthermore, the OMTG experiments in Table 3 show that our method remains effective on an external OMTG dataset, providing additional evidence for its effectiveness.

ModelHACS mIoUHACS [email protected]HACS [email protected]HACS RejF1InternVid mIoUInternVid [email protected]InternVid [email protected]InternVid RejF1QV-M2 mIoUQV-M2 [email protected]QV-M2 [email protected]
Time-R1-7B21.7026.7015.3018.4034.1037.7021.3032.1025.5032.2022.10
Time-R1-7B w/ our data34.6040.8022.1061.9044.6043.3026.9078.8031.9037.8028.60
MUSEG-7B33.6039.5023.200.0040.2044.3024.900.3031.0036.2026.60
MUSEG-7B w/ our data35.3041.9025.7024.8043.5047.5030.7055.2033.4038.6029.30
EGO-7B (Ours)41.3047.9031.5075.4050.4057.6042.4087.1038.0046.0032.90

22 Occurrence-specific versus ordinary CoT. We replace our occurrence-specific CoT with OMTG-style CoT while keeping training data, budgets, and the RL objective fixed. Occurrence-specific CoT outperforms both OMTG-style CoT and no CoT in mIoU, [email protected], and [email protected], demonstrating the benefit of reasoning about individual event occurrences for temporal localization. OMTG-style CoT first densely describes the entire video and then performs localization, producing substantial content unrelated to the queried event. These irrelevant descriptions introduce unnecessary distractions for localization and lengthen the output, increasing inference cost. Our CoT instead generates a specialized subquery for each matching occurrence, focusing reasoning on relevant events. The average output length is 474 tokens for OMTG-style CoT versus 105 tokens for ours. This shows that our CoT achieves better localization with shorter, more focused reasoning, reducing unnecessary output and its associated inference cost.

ConfigurationmIoU[email protected][email protected]RejF1
No CoT48.9055.4040.5086.50
OMTG-style CoT49.5856.3841.1887.40
Occurrence-specific CoT (Ours)50.4057.6042.4087.10

33 Removing only the rejection reward. We remove only the rejection reward, keeping the negative samples, other rewards, training data, and budgets unchanged. Localization performance remains comparable, while RejF1 decreases. This demonstrates that the explicit rejection reward improves absent-event rejection beyond including negative samples in training.

ConfigurationmIoU[email protected][email protected]RejF1
Without rejection reward50.4057.0841.6685.10
Full EGO50.4057.6042.4087.10

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.