acceptodds
Under review as a conference paper at ICLR 2027

TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

Abstract

Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs. We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task: long-video labels often rely on brittle one-pass annotation, while reinforcement-learning rewards either fail to distinguish non-overlapping predictions or require fragile segment matching. TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. TimeLens2-93K constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. Our temporal Wasserstein reward computes exact one-dimensional between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation; temporal IoU complements it with precise-overlap feedback. Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal large language models, Video temporal grounding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_M2ySOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TimeLens2 that can predict single or multiple start and end timestamps across diverse video lengths, domains, query forms, and viewpoints. It proposes a pipeline that produces reliable temporal grounding supervision for multi-duration videos, and constructs TimeLens2-93K dataset. A temporal Wasserstein reward is proposed to complement tIoU by providing distance-aware feedback for non-overlapping predictions.

Critical strengths
  1. This paper proposes a pipeline for constructing annotations for the temporal grounding task and demonstrates the effectiveness of each stage of the pipeline through the experiments presented in Table 3.

  2. This paper propose a new temporal grounding reward and demonstrates the effectiveness across various models and benchmarks.

Critical weaknesses
  1. Dataset construction is a key contribution of this paper, but the annotation pipeline relies on the initial coarse proposals and may miss relevant intervals in multi-span tasks.
Questions for authors
  1. Why do the authors use Qwen3-VL-30B-A3B for local grounding rather than Qwen3-VL-235B-A22B, which is already used for caption generation and boundary refinement?

  2. The temporal Wasserstein reward favors predictions closer to the ground truth even when they have zero overlap. While Table 10 and Table 11 support its overall usefulness, could this preference be misleading when nearby intervals contain unrelated actions and similar events occur farther away? Could the authors analyze such cases and compare with alternative reward designs, such as the timestamp-aware IoU reward in Time-R1?

  3. Since both grounders operate on the selected coarse proposals, they may agree on the retrieved intervals while missing relevant events outside those proposals. Subsequent verification and refinement focus on the selected evidence. Is there a full-video verification step to detect omitted occurrences, particularly for multi-span queries?

3 comments

W1 & Q3: The annotation pipeline may miss relevant intervals in multi-span tasks

Thank you for your comment. Our pipeline aims to recover all relevant occurrences through full-video caption indexing and global query–proposal generation. Below, we explain these efforts to improve full-video recall, their remaining limitations, and the human audit quantifying omissions.

1. How full-video recall is handled. To minimize missed intervals, our global-to-local pipeline enforces full-video coverage before local grounding. Every video is first partitioned into contiguous, non-overlapping scene clips spanning 100% of the timeline, with Qwen3-VL-235B-A22B generating a detailed dense caption for every clip. Rather than writing queries independently and searching post hoc, Kimi-K2.5 reads the complete video caption hierarchy to jointly synthesize each query and select its supporting candidate clips, under explicit instructions to collect all occurrences across the video; within each selected clip, both local visual grounders can further output multiple intervals whenever an event recurs.

Nevertheless, we admit these mechanisms do not guarantee exhaustive recall: an occurrence may still be omitted by the captioner or missed by a local grounder. Eliminating such omissions via brute-force verification offers a poor cost–quality trade-off, e.g., running both visual grounders on every clip across hour-long videos, would dramatically inflate the already 2M+ grounding calls while introducing false positives from unrelated distractors. We therefore conduct a full-video human inspection to directly evaluate and quantify the remaining omissions.

2. Quantifying omitted occurrences via human audit. We conducted a human audit on a subset of 1,000 queries (500 single-span + 500 multi-span) sampled from videos across all five duration bins (<2 min to ≥20 min). For each query, human annotators watched the entire unclipped video from start to finish to verify query validity, semantic precision, boundary accuracy, and any omitted occurrences outside the proposed clips:

Metric (query-level)Single-spanMulti-spanAll
Valid query rate98%98%98%
Semantic precision (all intervals contain the event)100%96%98%
Boundary precision (exact match / mIoU / MAE)90.0% / 0.97 / 0.17s80.0% / 0.95 / 0.21s85.0% / 0.96 / 0.19s
Completeness: span recall100%94%97%
Completeness: queries with a 100% complete interval set100%90%95%

Across the 500 multi-span queries, full-video inspection yields 94% query-level span recall and a 90% complete-set rate. Even in the longest duration bin (≥20 min), where proposal omissions are most challenging, recall remains 97.4% with an 89.5% complete-set rate. These results quantify the residual omissions in the audited subset and support the practical effectiveness of full-video indexing followed by local grounding, while keeping annotation tractable at scale.

Q1: Why do the authors use Qwen3-VL-30B-A3B for local grounding rather than Qwen3-VL-235B-A22B

The main reason is compute cost under our limited budget:

  1. Too many queries to annotate with the 235B model. Kimi-K2.5 generates a large number of candidate queries, which yield over 1M query–clip pairs to be grounded. Running Qwen3-VL-235B-A22B on every pair would cost roughly 7×7\times more compute for this stage, which exceeds our compute budget. Qwen3-VL-30B-A3B activates only 3B parameters (vs. 22B), so it can annotate all candidates at a much lower cost.
  2. 30B-A3B is nearly as accurate. As shown in Table 1, Qwen3-VL-30B-A3B performs close to Qwen3-VL-235B-A22B on short-video grounding, which matches our setting since local grounding runs on short 20–60s clips. It therefore offers a much better accuracy–cost trade-off.

Q2: Could RTWR_{\mathrm{TW}}'s preference for closer predictions be misleading when nearby intervals contain unrelated actions and similar events occur farther away? Comparison with Time-R1 [1].

Thank you for raising this question. Below, we explain how RTWR_{\mathrm{TW}}'s preference for closer predictions can help boundary refinement despite this risk, and provide a controlled comparison against Time-R1 [1].

1. Why preferring closer predictions can help boundary refinement. As a geometric signal, RTWR_{\mathrm{TW}} itself does not assess semantic relevance, so a nearby unrelated interval can in principle score higher than a distant one. Nevertheless, the following mechanisms and evidence support its practical benefit:

  • Sharp distance decay and unnormalized GRPO advantages: For a predicted and target interval of length LL separated by gap gg, W1=L+gW_1=L+g and RTW≈exp⁡[−(1+g/L)]R_{\mathrm{TW}}\approx\exp[-(1+g/L)], which is sizable only near the target boundary (0.37–0.14 for g≤Lg\le L) and vanishes quickly beyond it (0.05 at 2L2L, 0.018 at 3L3L, <0.003<0.003 beyond 5L5L). Crucially, our GRPO advantages are mean-centered without dividing by group standard deviation (Table 12 in the revised manuscript; Table 11 in the original submission), so tiny reward differences between non-adjacent unrelated intervals and distant distractors are not amplified.
  • Query-conditioned rollouts support boundary refinement: Rollouts come from a visually grounded MLLM conditioned on the query, which provides semantic guidance alongside the geometric reward. Nearby zero-overlap rollouts (≤1L\le 1L) can land on the target event's immediate setup or aftermath, whereas distant rollouts may select similar false-positive distractors. In such all-zero-tIoU groups, favoring the boundary-adjacent candidate can pull near-miss boundaries into the target.
  • Empirical evidence (Table 11 in the revised manuscript; Table 10 in the original submission): Across all 4,332 zero-overlap predictions from tIoU-only RL, adding RTWR_{\mathrm{TW}} recovers 21.9% of near misses (≤1×\le 1\times target length) and 15.8% of mid-distance misses (1–5×1\text{–}5\times) into positive overlap with the ground truth (7.5% and 4.0% mIoU), vs. only 5.7% for far misses (>5×>5\times, 66.5% of cases), supporting the benefit of this distance signal for recovering near-miss predictions.

2. Controlled comparison with Time-R1's timestamp-aware IoU [1]. To compare reward designs in a complementary direct-RL setting, we ran GRPO on Qwen3.5-4B (no SFT):

RewardCharadesActivityNetQVHighlightsVUE-TRVUE-TR-V2Avg.
No RL47.953.566.935.427.746.3
tIoU55.157.768.451.045.055.4
NGIoU (MUSEG [2])50.857.667.652.645.054.7
Timestamp-aware IoU (Time-R1 [1])54.355.964.248.946.353.9
tIoU + Temporal Wasserstein (ours)55.659.170.153.247.557.1

NGIoU already includes tIoU. Time-R1 [1] multiplies single-interval tIoU by a boundary-offset penalty; because it is multiplicative in tIoU, it is still zero for all disjoint predictions and even underperforms plain tIoU. In contrast, our reward achieves the best result on all five benchmarks, with the largest gains on multi-interval VUE-TR (+2.2) and VUE-TR-V2 (+2.5). We have added this comparison to Appendix A.4.


References

[1] Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding. NeurIPS 2025.

[2] MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding. ACL 2026.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewer M2yS,

Thank you again for the time and effort you put into reviewing our work. Your questions have helped us improve the paper’s analysis of annotation completeness and reward behavior.

As the discussion period ends in less than 48 hours, we would be grateful for your thoughts on our responses and additional results. Please feel free to raise any remaining concerns; we would be happy to provide further clarification.

All the best,
The TimeLens2 Authors

Reviewer_BA7ROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents TimeLens2, a generalist video temporal grounding framework based on multimodal large language models, aiming to localize a variable number of temporal segments across diverse video lengths, domains, query forms, and viewpoints. The main contributions are twofold. First, the authors construct TimeLens2-93K, a temporal grounding dataset using a multi-stage automated pipeline. Second, the authors propose a temporal Wasserstein reward that measures the distance between distributions over merged temporal interval supports, providing feedback for non-overlapping predictions without requiring explicit segment matching. Experiments on seven temporal grounding benchmarks demonstrate strong performance across different model sizes.

Critical strengths
  1. Important and practically relevant research problem. The paper addresses an important problem: developing a unified temporal grounding model capable of handling videos with diverse durations, domains, viewpoints, and query types, including queries corresponding to multiple temporal segments. Such generalization is essential for real-world video understanding applications, where existing single-segment grounding assumptions can be restrictive. The attempt to improve both temporal supervision and reinforcement learning objectives makes the work practically relevant.

  2. Strong empirical performance and extensive experimental analysis. The proposed models demonstrate consistently strong performance across seven benchmarks.

  3. The paper also provides extensive ablations covering dataset construction, long-context training, instruction diversity, and reward design. These results demonstrate the effectiveness of the overall training framework.

Critical weaknesses
  1. The paper overlooks several closely related studies. First, this is not the first work to investigate multi-segment video temporal grounding. Relevant methods and datasets have been introduced in [1–4], while the paper only discusses the NGIoU reward proposed in [4]. Second, this is not the first work to explore generalized video temporal grounding across different video lengths, domains, query forms, and viewpoints. In particular, [5] has already investigated a unified model for grounding videos with diverse viewpoints, genres, and durations, supporting various query types, including captions, procedural steps, and questions. The authors should provide a more comprehensive discussion of these studies and clearly distinguish the contributions of the proposed work from existing approaches.

[1] Towards One-to-Many Temporal Grounding. ICML 2026.

[2] When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions. NeurIPS 2025.

[3] Generalized Video Moment Retrieval. ICLR 2025.

[4] MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding. ACL 2026.

[5] Universal Video Temporal Grounding with Generative Multi-modal Large Language Models. NeurIPS 2025.

  1. The main contribution of the TimeLens2-93K dataset is unclear. If the primary contribution is the inclusion of long and diverse videos, it is unclear why the authors construct a new dataset rather than aggregate existing datasets, as explored in [5]. If the primary contribution is multi-segment supervision, the authors should clarify how TimeLens2-93K differs from and improves upon the multi-segment datasets introduced in [1–4]. A systematic comparison of dataset characteristics, including video duration, domain diversity, query types, and the number of target segments, would help establish the unique contributions of TimeLens2-93K.

  2. The quality of the automatically annotated dataset lacks human validation. TimeLens2-93K is constructed entirely through automated annotation. No independent human evaluation is provided to verify the accuracy of the generated annotations. It is suggested to manually annotate a representative subset of the dataset and quantitatively evaluate annotation quality, including semantic correctness, temporal boundary accuracy, and the completeness of multi-segment annotations.

  3. The novelty of the proposed temporal Wasserstein reward is limited. The paper emphasizes that the proposed reward can provide informative feedback about how non-overlapping predictions should move toward the ground truth and be invariant to equivalent fragmentation. However, providing distance-aware feedback is not entirely new. Previous studies have explored timestamp-aware reward design [6], and NGIoU [4] can already provide graded feedback for non-overlapping predictions. The improvement over NGIoU in Table 9 is relatively limited (47.7 vs. 47.1 average mIoU), raising questions about the practical significance of this design.

[6] Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding. NeurIPS 2025.

  1. The motivation for fragmentation invariance is questionable in multi-segment temporal grounding. I do not think that multi-segment grounding should necessarily be fragmentation-invariant. In this task, distinct occurrences of an event should be localized separately, while a single continuous event should not be unnecessarily fragmented. This principle is consistent with video action segmentation/detection and existing multi-segment temporal grounding studies [1–4], as reflected in their reward designs and evaluation protocols. However, the proposed Wasserstein reward considers predictions with identical merged temporal support equivalent, regardless of their segmentation. For example, a single ground-truth event [10, 20] and two predicted segments [10, 15] and [15, 20] receive the same reward. Such invariance ignores the distinction between individual event instances and may encourage incorrect event segmentation. The authors should provide a stronger justification for this design choice.

  2. The proposed reward and evaluation metrics are insufficient for assessing multi-segment grounding. Following the concern above, both the proposed reward and the reported evaluation metrics primarily focus on temporal coverage rather than the correctness of individual event instances. Consequently, they cannot adequately penalize incorrect fragmentation or evaluate whether distinct event occurrences are correctly separated. For instance, splitting a single event into multiple adjacent segments does not affect the reward, while merging two neighboring events may still receive a high score. The current evaluation relies on mIoU and recall-based metrics, which cannot fully capture these errors. In contrast, existing multi-segment grounding studies explicitly consider event-level localization quality. In particular, [1] adopts [email protected]/0.5/0.7 to evaluate multi-segment predictions. The authors should report these metrics to demonstrate whether TimeLens2 can accurately identify individual event occurrences rather than merely recover their combined temporal coverage.

  3. Given that OMTG [1] is a closely related work, I suggest that the authors train TimeLens2 on the OMTG dataset and conduct controlled comparisons. Specifically, comparing TimeLens2 with OMTG under the same training data would help clarify the contribution of the proposed method, while comparing TimeLens2 trained on OMTG and TimeLens2-93K under the same training settings would help isolate the contribution of the proposed dataset. These comparisons would provide more convincing evidence of the respective contributions of the dataset and method over this closely related work.

  4. Minor. How does the proposed model behave when the queried event never occurs in the video? The ability to handle absent events is an important aspect of generalized video temporal grounding.

Questions for authors

Q1. What are the specific contributions of TimeLens2-93K compared with existing multi-segment [1-4] and generalist temporal grounding datasets [5]?

Q2. Can the authors provide human validation of the automatically generated annotations?

Q3. Why is fragmentation invariance desirable for multi-segment grounding, and how does the proposed method handle event merging and splitting errors?

Q4. What is the performance when reporting the event-level localization metrics, such as [email protected]/0.5/0.7 in OMTG [1]?

Q5. Can the authors provide controlled comparisons with OMTG [1] to better distinguish the contributions of the proposed dataset and method?

Q6. How does TimeLens2 handle queries that have no corresponding events in a video?

12 comments

W1: Overlooking closely related studies

Thank you for pointing out these references. We have updated Sections 1 and 2 in the revised manuscript (in red) to discuss and compare against all five studies (note that MUSEG [4] was already compared in our original submission). We do not claim to be the first to formulate multi-segment or universal temporal grounding. Rather, TimeLens2 addresses two key bottlenecks in training a unified MLLM: curating verified supervision for hour-long, multi-segment videos and optimizing multi-segment RL without one-to-one interval matching.

Compared to prior multi-segment grounding works, FlashMMR [2] and GVMR [3] design specialized localization modules for short videos, whereas TimeLens2 trains a single generative MLLM across diverse queries and durations. While OMTG [1] and MUSEG [4] also apply GRPO to MLLMs, OMTG-56K [1] targets repetitive events in shorter videos (mean 182s, up to 20 min), whereas TimeLens2-93K provides verified annotations up to 100 minutes. For RL, both MUSEG [4] and OMTG [1] require one-to-one interval matching, whereas our RTWR_{\mathrm{TW}} reward computes exact 1D optimal transport over merged timelines without interval assignment.

Compared to universal grounding in UniTime [5], which focuses on input timestamp/frame representation via SFT, TimeLens2 differs in four key aspects: (1) Video length: UniTime's training videos reach at most 34 min (only 2.4% exceed 10 min), whereas TimeLens2-93K reaches 100 min (37.4% exceed 10 min). (2) Multi-segment supervision: Among the 881K training samples in UniTime's released data that supervise final interval outputs, only 15 contain multiple target intervals (all from QVHighlights; its other multi-window labels are used only for coarse timestamp-point retrieval), whereas we provide 12.1K verified multi-segment instances. (3) Data curation: UniTime aggregates existing training datasets, whereas we purposefully curate videos across 15 diverse domains. (4) Optimization: UniTime relies solely on SFT, while TimeLens2 introduces RTWR_{\mathrm{TW}}-guided GRPO.

Comparison with UniTime under the same backbone. To directly compare with this prior generalist model, we applied our full data and training recipe to Qwen2-VL-7B, the same backbone used by UniTime [5]. We evaluated both models under identical test settings:

Model / RecipeTraining data (videos / queries)Charades-STAActivityNetQVHighlightsVUE-TRVUE-TR-V2MomentSeekerAvg.
UniTime [5]57.3K / 1.34M41.3035.5758.0029.3125.4621.9735.27
TimeLens2 data + training recipe44.5K / 203.7K53.9151.0264.5443.9939.2024.6546.22

Using about 15% of UniTime's training queries, our data and training recipe outperforms UniTime on all six benchmarks, with an average gain of 10.95 mIoU points, demonstrating the superior quality of our training data.

W2 & Q1: Unique contribution of TimeLens2-93K vs. dataset aggregation [5] and prior multi-segment datasets [1–4]

TimeLens2-93K contributes verified single- and multi-span supervision on long videos, built by a pipeline that uses only open-weight models. Its quality is confirmed by human audit (98.0% semantic precision) and by SFT alone: +11.1 avg. mIoU on Qwen3-VL-4B (34.7 → 45.8), vs. +4.7 for the Gemini-2.5-Pro-annotated TimeLens-100K (Table 2). We have added a systematic dataset comparison to Appendix A.7 (Table 16):

DatasetScale (Videos / Queries)Video Duration (Avg. / Max)Total HoursMulti-SpanOpen-Source
QV-M$^2$ [2]1.3K / 2.2K150s / 150s56YesYes
NExT-VMR [3]10.0K / 153.2Kn/r / n/rn/rYesNo
MUSEG [4]10.1K / 12.6K105s / 4.3 min~294YesYes
OMTG-56K [1]44.9K / 56.2K182s / 20 min2,272YesYes
UniTime Mix [5]57.3K / 1.34M180s / 34 min~2,870NoYes
TimeLens-100K [7]19.5K / 96.6K106s / 8.3 min575NoYes
TimeLens2-93K23.8K / 93.2K615s / ~100 min4,064YesYes (will be fully released)

Coverage. TimeLens2-93K has the most video hours and is the only corpus with a substantial long-video subset: 37.4% of its videos exceed 10 min and 9.5% exceed 30 min, whereas no video in [1–5] exceeds ~30 min (OMTG-56K: 7.0% over 10 min). It contains 5.7K multi-span queries in videos over 10 min and 1.3K in videos over 30 min. Evidence is also sparser: the median query covers 7.4% of its video, vs. 36.8% in OMTG-56K and 48.2% in MUSEG. Videos are stratified across 15 domains (Fig. 2); queries are free-form descriptions of events, objects, and scenes; multi-span queries contain 2–11 segments.

Quality. We regard the label quality as the dataset's main contribution. Four independent lines of evidence support it:

EvidenceResult
Human audit (1,000 stratified queries, see W3)98% semantic precision, 97% span recall, 0.96 boundary mIoU (MAE 0.19s)
Verification cascade vs. raw labels (Table 3)Keeps 93.2K of 735.4K proposals (12.7%); +3.8 avg. mIoU
vs. aggregated data, same SFT recipe (Table 2)+6.4 avg. mIoU over TimeLens-100K (45.8 vs. 39.4); a 5% subset alone already exceeds it (42.8)
Full data recipe vs. OMTG data recipe, same TimeLens2 method (see W7)+2.56 avg. mIoU (60.72 vs. 58.16) across Charades-STA, ActivityNet, and QVHighlights

Why not aggregate existing datasets [5]? Existing datasets lack long-video multi-span supervision: among the 881K samples in UniTime's released data that supervise interval outputs, only 15 contain more than one interval; its other multi-window labels serve only as coarse timestamp-point targets. Moreover, even compared with TimeLens-100K [7]—which aggregates existing datasets and re-annotates them with Gemini 2.5 Pro—training on TimeLens2-93K under the same SFT recipe still yields large gains on long-video benchmarks (+11.3 on VUE-TR, +12.2 on VUE-TR-V2). In W1, our data and training recipe also outperforms UniTime by 10.95 average mIoU points using about 15% of its training queries under the same backbone and test settings.

What improves over prior multi-segment datasets [1–4]? Beyond video length, the key difference is how boundaries are obtained. MUSEG [4] reuses action-class labels from short-video datasets. OMTG-56K [1] takes boundaries from a single proprietary grounder (Gemini 2.5 Pro) and checks them only for semantics and recall; 3.2% of its queries have segments that end beyond the video. In TimeLens2-93K, a label is kept only when two independent open-weight grounders agree (set IoU > 0.9); it then passes semantic verification and boundary refinement. No proprietary API is used anywhere in the pipeline.

W3 & Q2: The quality of the automatically annotated dataset lacks human validation.

Following your suggestion, we manually audited a stratified subset: 500 single-span + 500 multi-span queries from videos across five duration bins (<2m to ≥20m). Annotators watched each video in full to check query validity and verify whether each segment contained the described event; to establish human reference labels for evaluating boundary accuracy and segment completeness, they also marked exact start/end timestamps and logged any missed occurrences. The results are summarized below:

Metric (query-level)Single-span (500)Multi-span (500)All (1,000)
Valid query rate98%98%98%
Semantic precision100%96%98%
Exact boundary match rate90%80%85%
Boundary accuracy: mIoU / MAE0.97 / 0.17s0.95 / 0.21s0.96 / 0.19s
Multi-segment completeness: span recall100%94%97%
Multi-segment completeness: queries with a complete segment set100%90%95%

Errors occur mainly in multi-span queries, yet 90% of multi-span queries still retain a complete event set. The 180 adjusted boundaries shifted by 4.3s on average without start or end bias, yielding 0.96 mIoU and 0.19s MAE overall. Quality also holds on long videos of 20 minutes or longer (97.4% precision, 97.4% recall, 89.5% complete-set rate). We have added the audit protocol and results to Appendix A.9 (Table 18).

Reviewer_pGS1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper introduces TimeLens2, a generalist video temporal grounding model capable of localizing single or multiple evidence intervals across diverse video durations, domains, query types, and viewpoints.
  • The core contributions include a curated 93K-sample grounding dataset and a temporal Wasserstein reward that provides dense supervision without explicit interval matching.
  • Evaluations across seven benchmarks show consistent gains over the base Qwen3-VL models and competitive performance against substantially larger counterparts.
Critical strengths
  • Valuable dataset contribution with clear empirical gains. TimeLens2-93K provides 93,232 grounding samples across 23,793 videos (including 12,091 multi-interval examples) curated via localization consensus, semantic verification, and boundary refinement. Controlled SFT ablations confirm its quality: under an identical training recipe, it improves average mIoU from 39.4 (TimeLens-100K) to 45.8 at a comparable data scale (Table 2).

  • Comprehensive evaluation and well-motivated reward design. The method is evaluated across diverse model scales and benchmarks (covering long videos, multi-interval queries, QA-style prompts, and egocentric footage), alongside thorough data and training ablations. The proposed temporal Wasserstein reward effectively addresses zero-overlap optimization issues, with diagnostic analyses on zero-overlap recovery and group-relative signals supporting its formulation.

Critical weaknesses
  • modest technical novelty and empirical gains. While the temporal Wasserstein reward provides useful matching-free supervision for non-overlapping predictions, it primarily adapts standard Wasserstein distance to merged temporal intervals within an existing RL pipeline. Its empirical gains over tIoU-only RL (47.0 to 47.7, Table 6) are relatively small, with the curated dataset and SFT recipe driving most of the overall improvement.

  • Unverified annotation fidelity. While downstream ablations (Table 3) validate the automated curation pipeline, the paper lacks an independent human audit to directly quantify semantic accuracy, boundary precision, and multi-interval completeness. Relying solely on model consensus and downstream performance leaves intrinsic label noise and missed evidence intervals unmeasured in this core dataset contribution.

  • unclear scaling economics.

    • Appendix A.7 notes that financial and computational constraints motivated the use of predominantly open-weight annotation models, but does not quantify total or per-sample construction costs. Without such cost accounting, the cost–benefit of full-scale data expansion remains unclear.
    • Although the paper specifies SFT and GRPO hyperparameters, it does not report GPU hardware, training time, or GPU-hours, making the computational cost of reproducing the models difficult to assess.
Questions for authors

None

4 comments

W1: Modest technical novelty and empirical gains

Thank you for this comment.

Novelty. Our novelty lies in how set-valued temporal reward is formulated, not in the Wasserstein distance itself. Prior temporal rewards fall into two paradigms: overlap-based rewards [1], which vanish completely once predictions and targets are disjoint; and pairwise-matched rewards [2], which provide non-overlap signal only through fragile interval-to-interval assignments. To our knowledge, RTWR_{\mathrm{TW}} is the first RL reward to cast set-valued grounding as distribution transport over time, removing both failure modes at once. Furthermore, applying W1W_1 is non-trivial: our ablation on how interval sets are lifted to distributions (Tab. 6) shows that point or Gaussian abstractions lose interior occupancy or duration and degrade on individual tasks, whereas uniform support over merged intervals preserves both location and duration—making it the only design that improves all seven benchmarks.

Gains. In the full pipeline, the +0.7+0.7 gain over tIoU-only RL is measured after SFT on the full corpus. Our RL samples are drawn from data already used during SFT, so this comparison measures the incremental benefit of reward-based optimization. Even there, adding RTWR_{\mathrm{TW}} more than doubles the RL gain over SFT (+0.6+0.6 with tIoU; +1.3+1.3 with RtIoU+RTWR_{\mathrm{tIoU}} + R_{\mathrm{TW}}). To test reward effectiveness in a complementary setting, we also ran GRPO directly on Qwen3.5-4B (no SFT):

RewardCharadesActivityNetQVHighlightsVUE-TRVUE-TR-V2Avg.
No RL47.953.566.935.427.746.3
tIoU55.157.768.451.045.055.4
NGIoU [2]50.857.667.652.645.054.7
Timestamp-aware IoU [1]54.355.964.248.946.353.9
tIoU + RTWR_{\mathrm{TW}} (ours)55.659.170.153.247.557.1

NGIoU already includes tIoU, while Time-R1 multiplies tIoU by a timestamp penalty. Both alternative rewards underperform plain tIoU on average in this setting. In contrast, our reward achieves the best result on all five benchmarks, with the largest gains over tIoU on the multi-interval VUE-TR (+2.2+2.2) and VUE-TR-V2 (+2.5+2.5) benchmarks for which RTWR_{\mathrm{TW}} was designed. We have added this experiment to Appendix A.4 (Table 10) of the revised manuscript (highlighted in red).


W2: Unverified annotation fidelity.

Thank you for this comment. To verify annotation quality, we conducted a human audit on 1,000 queries (500 single-span and 500 multi-span) sampled from videos across five duration bins (from under 2 min to at least 20 min). Annotators watched each video in full to check query validity and verify whether each segment contained the described event; to establish human reference labels for evaluating boundary accuracy and segment completeness, they also marked exact start/end timestamps and logged any missed occurrences. The results are summarized below:

Metric (Query-level)Single-spanMulti-spanAll
Valid query rate98%98%98%
Semantic precision (all intervals contain the event)100%96%98%
Exact boundary match rate90%80%85%
Boundary precision (mIoU / MAE)0.97 / 0.17s0.95 / 0.21s0.96 / 0.19s
Completeness (span recall)100%94%97%
Completeness (queries with a complete interval set)100%90%95%

As shown above, single-span annotations achieve 100% semantic precision and recall, while errors are rare and mainly appear in the more complex multi-span queries. Temporal boundaries are also highly accurate (0.96 mIoU overall) with no systematic start or end bias (mean offset: +0.03s/−0.12s+0.03\text{s} / -0.12\text{s}), and annotation quality remains robust even on long videos of at least 20 minutes (97% precision and recall). We have added the audit protocol and results to Appendix A.9 (Table 18).

W3: Unclear scaling economics.

Thank you for pointing this out. To clarify both data construction and model reproduction costs, the table below reports the exact computational breakdown standardized to NVIDIA H200 GPU-hours (including frame loading, preprocessing, and batched model inference):

Component / StageModel / SettingH200 GPU-hours
TimeLens2-93K Data Construction
1. Captioning (global + all segments)Qwen3-VL-235B-A22B (no think)~612
2. Query & proposal synthesisKimi-K2.5 (thinking)~558
3. Dual-agent local groundingQwen3-VL-30B-A3B + TimeLens-8B [3]~337
4. Semantic verificationQwen3-VL-Embedding~160
5. Boundary refinementQwen3-VL-235B-A22B~499
Data Construction Total/~2,166
TimeLens2-4B Training & Rollout
1. Supervised fine-tuningQwen3-VL-4B~230
2. Offline RL sample generationTimeLens2-4B (SFT)~22
3. GRPO reinforcement learningTimeLens2-4B~31
Model Training Total/~283

Data construction averages 84 GPU-seconds per retained QA pair; routing the 2M+ local grounding calls to smaller models (30B-A3B and 8B) keeps grounding to just 16% of the budget. For data scaling (Table 2), a 5% subset (108 GPU-hours) already raises Qwen3-VL-4B from 34.7 to 42.8 avg. mIoU, outperforming 100% of TimeLens-100K [3] (39.4); a 20% subset (433 GPU-hours) captures 95% of the full gain (45.3 vs. 45.8), while the rest further improves harder long-video and egocentric benchmarks. Since we will release TimeLens2-93K, reproducing TimeLens2-4B takes only ~283 H200 GPU-hours (1.5 days on one 8×H200 node). We have added these details to Appendix A.8 (Table 17).


References

[1] Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding. NeurIPS 2025.

[2] MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding. ACL 2026.

[3] TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs. CVPR 2026.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewer pGS1,

Thank you again for your time and thoughtful review. Your suggestions have helped us strengthen the paper, particularly the annotation validation and computational cost analysis.

With less than 48 hours remaining in the discussion period, we would appreciate your feedback on our rebuttal when you have a chance. Please let us know if any concerns remain unresolved; we would be happy to discuss them further.

All the best,
The TimeLens2 Authors

Thanks for authors response. These results address several of my initial concerns and provide a clearer understanding. I maintain the current socre.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.