- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies video temporal grounding where a text query may correspond to zero, one, or multiple video segments. It introduces AnyMoment, a dataset covering all three cases, and proposes Event-wise Grounding Optimization (EGO), a reinforcement learning framework for this setting. EGO combines difficulty-aware data sampling, occurrence-specific reasoning, and event-level credit assignment to provide separate learning signals for individual predictions within a response. Experiments on AnyMoment and existing benchmarks show improvements in temporal localization and rejection of queries with no matching events, with event-level optimization particularly benefiting queries involving multiple occurrences.
Critical strengths
-
Practical problem setting. Supporting zero, one, or more matching segments makes temporal grounding more applicable to real-world videos.
-
Useful dataset contribution. AnyMoment includes both recurring events and challenging negative queries, enabling evaluation of localization and rejection. (This can be considered as a contribution only if the authors will release their dataset to the public.)
-
Well-motivated method. Event-level credit assignment addresses the limitation of applying a shared learning signal to correct and incorrect predictions within the same response.
-
Broad evaluation. Experiments cover multiple backbones and benchmarks, with ablations showing that event-level optimization is particularly helpful for queries with more occurrences
Critical weaknesses
-
Limited coverage of recent strong baselines. The evaluation would benefit from comparisons with newer Gemini Flash/Pro and Seed models, particularly on AnyMoment, where no proprietary-model results are reported. These comparisons would help establish the benchmark’s difficulty and better contextualize EGO’s performance.
-
Incomplete baseline inference details. Video input settings, decoding parameters, and output parsing rules are not fully specified for all baselines. Instructions for rejecting absent events also differ across prompts, raising concerns about reproducibility and comparability.
-
Insufficient separation of data and methodological gains. Most baselines are not reported to be retrained on the same data, and the MUSEG comparison using the proposed data lacks training details. This makes it difficult to isolate the contribution of the proposed method.
-
Limited transparency in data quality validation. The scope and sample size of human verification, as well as disagreement resolution, are unclear. Query and temporal annotation errors are not separately analyzed, and the relationship between overall and subgroup accuracy statistics requires clarification.
Questions for authors
-
The paper lacks domain, action, and scene distributions, making it difficult to assess AnyMoment’s semantic coverage and potential bias toward specific repetitive actions.
-
The dataset is claimed as a contribution. If the authors are not going to release the dataset, it is inappropriate to claim it as a contribution in my view.
4 comments
W3: Separating data and method gains: MUSEG retraining with matched data and budgets
We use our data to retrain MUSEG-7B, the second-best baseline that specializes in multi-event grounding. Our EGO and MUSEG w/ our data use identical training data and budgets, with the Qwen2.5-VL-7B backbone. SFT uses 46,940 AnyMoment samples; RL uses 7,500 samples: 2,500 HACS, 2,500 InternVid, and 2,500 Time-R1. Following Appendix C.4–C.5, SFT runs for one epoch with batch size 32 and learning rate 2×10⁻⁴; RL runs for two epochs with batch size 16, eight rollouts, and learning rate 10⁻⁶.
To further demonstrate the contribution of the proposed method, we additionally train Time-R1 with our data under the same training budgets. Comparing the original baselines with their w/ our data variants demonstrates the dataset's contribution, while EGO's further gains over both matched-data baselines demonstrate the method's contribution.
| Model | HACS mIoU | HACS [email protected] | HACS [email protected] | HACS RejF1 | InternVid mIoU | InternVid [email protected] | InternVid [email protected] | InternVid RejF1 | QV-M2 mIoU | QV-M2 [email protected] | QV-M2 [email protected] |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Time-R1-7B | 21.70 | 26.70 | 15.30 | 18.40 | 34.10 | 37.70 | 21.30 | 32.10 | 25.50 | 32.20 | 22.10 |
| Time-R1-7B w/ our data | 34.60 | 40.80 | 22.10 | 61.90 | 44.60 | 43.30 | 26.90 | 78.80 | 31.90 | 37.80 | 28.60 |
| MUSEG-7B | 33.60 | 39.50 | 23.20 | 0.00 | 40.20 | 44.30 | 24.90 | 0.30 | 31.00 | 36.20 | 26.60 |
| MUSEG-7B w/ our data | 35.30 | 41.90 | 25.70 | 24.80 | 43.50 | 47.50 | 30.70 | 55.20 | 33.40 | 38.60 | 29.30 |
| EGO-7B (Ours) | 41.30 | 47.90 | 31.50 | 75.40 | 50.40 | 57.60 | 42.40 | 87.10 | 38.00 | 46.00 | 32.90 |
Finally, the OMTG experiment in Table 3 also supports the effectiveness of our method beyond AnyMoment training data. Using the same OMTG-46K dataset and Qwen3-VL-4B backbone, our EGO-4B outperforms OMTG-4B on all reported metrics.
| Method | C-Acc | [email protected] | [email protected] | [email protected] | mIoU |
|---|---|---|---|---|---|
| OMTG-4B | 55.63 | 73.46 | 65.40 | 48.96 | 61.24 |
| EGO-4B (Ours) | 59.06 | 75.72 | 67.03 | 50.86 | 62.43 |
W4: Human-verification scope, disagreement resolution, query/temporal errors, and overall versus subgroup accuracy
The scope and sample size of human verification and disagreement resolution. All 6,200 test samples are human-verified. Annotators receive machine labels as a reference and inspect the video and annotations. They remove event annotations that do not match the query, adjust inaccurate temporal boundaries, and add missing matching events with their timestamps. Human judgments always take precedence when they disagree with MLLM annotations. A second annotator double-checks the resulting annotations. Any problems identified lead to reannotation, with the reannotated results used as the final labels. In practice, 99.8% of the non-zero-event test samples have their automatic event boundaries adjusted by humans.
Query and temporal annotation errors. Under our task definition, there is no separate query-error category. If a query does not match the video, annotators leave the query unchanged and correct the temporal annotation to an empty set. Such corrected cases account for 3.5% of samples.
Relationship between overall and subgroup accuracy. The 87% overall automatic-annotation accuracy and the target-type statistics refer to AnyMoment-InternVid. The reported rates are 96.5%, 82.6%, and 80.6% for zero-, single-, and multi-event samples. Weighting these rounded rates by InternVid's test composition gives approximately 87%, consistent with the overall estimate. HACS's source-specific accuracy is a separate statistic.
Q1: Domain, action, and scene distributions, semantic coverage, and bias toward repetitive actions
We supplement the source-wise domain and scene distributions below, alongside a summary of HACS action coverage. We acknowledge a bias toward human activities and repeatable actions. Our collection prioritizes repeated occurrences, which are readily available in human-activity videos such as sports and everyday routines.
Domain distribution
| Category | HACS | InternVid |
|---|---|---|
| Human activities | 99.16% | 72.40% |
| Animal activities | 0.12% | 8.10% |
| Transportation and machinery | 0.50% | 7.60% |
| Objects and static displays | 0.22% | 10.47% |
| Other | 0.00% | 1.43% |
Scene distribution
| Category | HACS | InternVid |
|---|---|---|
| Indoor | 20.67% | 33.80% |
| Urban and streets | 3.80% | 10.10% |
| Rural and farmland | 0.44% | 4.63% |
| Natural | 10.89% | 15.23% |
| Sports | 42.74% | 15.44% |
| Water environments | 13.26% | 13.83% |
| Other | 8.20% | 6.97% |
HACS action distribution. Our data provide broad coverage of HACS's original 200 action labels. We will include the detailed action-frequency distribution figure in the revised paper.
Q2: Public release of AnyMoment and the EGO method
Upon acceptance, we will publicly release the AnyMoment dataset, EGO models, and training/evaluation code.
A Brief Follow-up
Dear Reviewer Xh6w,
Thank you for your thoughtful review and constructive feedback. As the discussion period is approaching its end, we would greatly appreciate any feedback on our rebuttal. Please let us know if you have any remaining concerns, and we would be glad to discuss them further.
Best regards,
The Authors
We thank the reviewer for recognizing the practical zero-, single-, and multi-event setting, the value of recurring events and challenging negative samples in AnyMoment, and the motivation for event-level credit assignment. We also appreciate your positive assessment of the evaluation across multiple backbones and benchmarks, especially the gains for samples with more occurrences.
W1: Recent Gemini Flash/Pro, GPT, and Seed baselines on AnyMoment and EGO's relative performance
1 Thank you for requesting these comparisons. We have added AnyMoment results for Seed-2.0-Lite, GPT-6.1-sol, Gemini-3.1-pro, and Gemini-3.8-flash. We also compare with MUSEG-7B, the second-best baseline that specialized multi-event grounding method. Our EGO-8B leads mIoU and [email protected] on both datasets and [email protected] on HACS. Gemini-3.1-pro has the highest [email protected] on InternVid, while GPT-6.1-sol has the highest RejF1 on both datasets. This comparison highlights EGO's localization strength alongside the strong rejection performance of frontier models.
2 We further compare the performance on multi-event samples. Multi-event grounding remains challenging: across the four strong frontier models, mIoU drops a lot on multi-event samples. The advantage of our EGO over the these frontier models is clearest in multi-event mIoU and [email protected]: it exceeds their strongest result in mIoU by 4.33 points on HACS and 2.58 on InternVid, and in [email protected] by 1.46 and 4.86 points, respectively. These results show that AnyMoment remains challenging even for frontier models, while EGO-8B is more effective at precisely localizing repeated event occurrences.
W2: Baseline video inputs, decoding parameters, output parsing, and model-specific rejection instructions
1 Video inputs and decoding parameters. All the baselines and our models samples videos at 2 FPS, caps the input at 768 frames, and dynamically adjusts resolution to a budget of approximately 3,584 visual tokens, for a fair comparasion. Generation uses temperature=0, top-p = 1.0, repetition penalty = 1.05, and a maximum of 1,024 output tokens.
2 Output parsing. We preserve each grounding baseline's native training-time output format and parse its predicted temporal intervals using format-specific regular expressions. The parsing is robust across these native formats: no evaluated sample from any model was assigned a zero score due to parsing failure.
(3) Rejection instructions. Different methods have different default prompts and output formats. We preserve each method's native prompt structure, try alternative rejection instructions, and retain the stronger-performing version for that method. Consequently, the wording differs, while the task remains to return an empty prediction when the event is absent and all matching intervals when it is present. The selected native templates are described in Appendix E.4.