acceptodds
Under review as a conference paper at ICLR 2027

Learning to Ground Without Forgetting: Reinforcement Learning for Long-Video Temporal Understanding in Multimodal LLMs

Abstract

Long-video temporal grounding introduces complexities not found in standard short-video settings, as the extended temporal horizon amplifies localization error and increases event sparsity. A central challenge in adopting reinforcement learning (RL) for this task is preserving pre-trained knowledge while actively exploring novel strategies for generating the precise, structured timestamps needed to resolve event sparsity and ambiguity. To address this, we propose Token-aware KL Regularization, which selectively relaxes the KL-divergence on timestamp-related tokens to encourage exploration during early training. To further mitigate the sparsity of key events, we introduce a dense reward called Center Distance Reward (CenDist). Combined with our automatically constructed SceneTG dataset that facilitates effective RL training, our model, LongVTG-R1, achieves the highest mIoU among comparable-scale methods across three long-video temporal grounding benchmarks. Beyond grounding, our model generalizes to long-video question answering (QA), serving either as an explicit grounding module or, more importantly, as a unified system that jointly performs grounding and QA. Finally, despite being trained only on single-segment data, LongVTG-R1 exhibits an emergent capability for recursive multi-segment grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Model, Temporal Grounding, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ESLyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the knowledge forgetting in long-video temporal grounding. It proposes Token-aware KL Regularization and Center Distance Reward (CenDist), and builds the SceneTG dataset to improve training data quality. Experiments show superior temporal grounding performance while largely preserving general video QA ability.

Critical strengths
  1. The method achieves superior results on several long-video temporal grounding benchmarks.

  2. This paper is well-written and easy to understand.

Critical weaknesses
  1. RL has already been widely used for video temporal grounding, such as in VideoChat-R1 and TimeLens2. This paper is built on existing RL framework and modifies the loss and reward terms, its main technical contribution is not clear.

  2. The design of token-aware KL is ad hoc and does not verified on more related tasks. It mainly adjusts the KL regularization weights for different token types. Although the paper provides some mathematical analysis, it lacks a clear explanation for the token grouping and weight choices. It also lacks evidence that this design helps reduce knowledge forgetting.

  3. Table 6 shows marginal gains from the proposed components, while the improvements in Table 1 are much larger. I suggest adding a full-scale GRPO baseline without Token-aware KL and CenDist, using exactly the same training data, training steps, and settings. This would help clarify the actual contribution of the proposed methods.

Questions for authors

In Weakness.

2 comments

We thank the reviewer for the constructive comments, and for recognizing the strong results on long-video temporal grounding benchmarks and the clarity of the writing. Below we address all the concerns.

W1 Main technical contribution beyond existing RL frameworks (VideoChat-R1, TimeLens2)

Our work differs from prior RL-based temporal grounding methods in three connected aspects.

1) Problem scope: temporally sparse grounding in longer videos. Our CG-Bench evaluation includes videos up to 6,325 seconds (105.4 minutes). This is substantially longer than the predominantly short-video setting in the original VideoChat-R1 paper, whose grounding evaluation centers on Charades-STA and ActivityNet, with average video durations of roughly 30 seconds and 2 minutes, respectively. Although TimeLens2 also studies long videos, sparsity, and ambiguity through VUE-TR and VUE-TR-V2, our particular focus is the joint grounding-QA trade-off: learning sparse-event localization over long timelines while retaining the model's general question-answering capability.

2) Three technical changes motivated by this setting. When a target occupies only a small fraction of a long video, many sampled predictions receive zero IoU; ambiguous boundaries further introduce noisy rewards.

  • Token-aware KL separates exploration-critical timestamp tokens from semantic tokens, relaxing the former while regularizing the latter.
  • CenDist complements IoU with a video-duration-normalized center-distance reward that provides informative feedback for non-overlapping predictions.
  • SceneTG provides stable grounding targets for RL training through an automatic data-construction pipeline: it proposes scene boundaries before generating queries and filters queries that also match neighboring scenes.

Thus, alongside the loss and reward contributions, our third contribution supplies cleaner, less ambiguous long-video grounding supervision.

3) Extending the benefits of grounding to long-video QA and multi-segment localization. We evaluate direct QA retention and show that selecting evidence with the learned grounding policy improves long-video QA in both ground-then-answer and unified settings. We also reuse the single-segment policy recursively to retrieve multiple relevant intervals, without multi-segment fine-tuning. The VideoChat-R1 paper does not investigate this multi-segment setting. TimeLens2 learns interval-set prediction with explicit multi-span supervision; our recursive approach instead transfers a policy learned from single-segment supervision to multi-segment grounding.

W2 Token-aware KL: rationale for token grouping and weights, and evidence for reducing forgetting

Rationale for the design. Token-aware KL is a structured relaxation of uniform regularization. Timestamp tokens encode the temporal boundaries directly evaluated by the grounding reward, motivating greater freedom to explore alternative boundary predictions. We therefore relax their KL constraint while retaining regularization on language and answer generation.

Figure 2 examines this behavior through separate entropy curves for the token groups, showing how relaxing timestamp regularization affects exploration during training. Entropy curves are also used to analyze exploration and training dynamics in prior RL work, such as DAPO [R1]. Our entropy analysis provides evidence about exploration behavior, while the grounding ablations evaluate whether this design improves task performance.

Choice of KL weights. Appendix E of the original submission already includes a controlled sensitivity analysis: Table 11 (now Table 12) fixes β_text at 0.04 and varies β_time over {0.04, 0.02, 0}. As β_time decreases, CG-Bench [email protected] increases from 9.27 to 9.43 to 9.84. Thus, removing the timestamp-token penalty gives the best result among the tested settings while retaining language-token regularization. These results provide empirical support for the selected allocation of KL regularization. NExT-GQA also shows a consistent trend. These results support the choice of the weights.

Evidence regarding forgetting. Following the reviewers’ suggestions, we additionally evaluated direct-QA performance under uniform KL and token-aware KL on four long-video QA benchmarks:

LongVideoBenchMLVUVideoMME (Long)LVBench
Qwen2.5-VL (w/o finetuning)60.568.653.441.3
Uniform KL 0.0460.969.053.140.3
Token-aware KL 0.0458.968.953.840.7

Both variants broadly retain the original model’s QA performance, but their differences are mixed across benchmarks. These results do not show a consistent QA-retention advantage from token-aware KL over uniform KL. Together with the grounding improvements in Table 6 (now Table 7) , they support token-aware KL as improving grounding while broadly retaining direct-QA performance.

Clarifying the role of token-aware KL. These controlled experiments support the grounding benefit of token-aware KL while showing that direct-QA performance is broadly retained. We have revised the corresponding statements in the introduction and method description. In particular, L112–114 now reads: “We propose token-aware KL regularization that encourages timestamp exploration while retaining regularization on language tokens, improving temporal grounding in controlled ablations.” We have revised L092–095 consistently.

[R1] Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., ... & Wang, M. DAPO: An open-source LLM reinforcement learning system at scale. In NeurIPS, 2025.

W3 Marginal gains in Table 6 (now Table 7) vs. larger gains in Table 1; full-scale GRPO baseline without Token-aware KL and CenDist

Thank you for suggesting this controlled comparison. The requested full-scale GRPO baseline is already included as “VideoChat-R1 (fair comparison)” in Table 1. This baseline uses the same training data, training steps, and settings as LongVTG-R1, but retains conventional KL regularization and the IoU reward, without token-aware KL or CenDist. Its comparison with LongVTG-R1 therefore evaluates the combined algorithmic contribution at full scale, with the training data and budget held fixed.

The contributions of data and algorithmic changes differ across benchmarks. On CG-Bench, SceneTG accounts for much of the improvement, raising the 16K VideoChat-R1 baseline from 0.8 to 9.2 mIoU; LongVTG-R1 further improves this to 10.3 using the same data mixture. On TaCoS and Ego4D-NLQ, the algorithmic changes make a larger contribution. On Ego4D-NLQ, even in the reduced setting, adding token-aware KL alone raises mIoU to 4.9, already exceeding the full-scale GRPO baseline with SceneTG at 4.5, while our full method further improves it to 5.7.

The training scale also matters when interpreting the gains. Table 6 isolates individual components in the reduced setting, whereas Table 1 evaluates the complete method and its matched baseline at full scale. Figure 5 in Appendix E further shows that the gap between VideoChat-R1 and LongVTG-R1 widens as more training data is used. This supports the observation that the combined algorithmic advantage becomes more pronounced at larger training scales.

We hope these clarifications and additional results have addressed your concerns. We have incorporated the new experiments and discussions into the revised paper, and we are happy to answer any further questions.

Reviewer_2CKuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper presents LongVTG-R1, a reinforcement learning (RL) framework for long-video temporal grounding that encourages timestamp exploration while preserving general video understanding capabilities.
  • The method integrates token-aware KL regularization, a dense center-distance reward, and SceneTG—an automatically curated dataset designed to reduce temporal annotation ambiguity.
  • Experiments show consistent gains across three long-video grounding benchmarks under matched data scales, alongside improved grounded video QA and zero-shot transfer to multi-segment grounding via recursive inference.
Critical strengths
  • Well-motivated token-aware regularization for timestamp exploration. Relaxing KL penalties on timestamp tokens while retaining regularization elsewhere directly addresses the trade-off between timestamp exploration and preserving pretrained language priors, supported by training-dynamics and coefficient-sensitivity analyses.

  • Effective data curation validated by human agreement. SceneTG leverages scene proposals and uniqueness filtering to reduce temporal ambiguity. Applying the pipeline to public ActivityNet videos boosts human–ground-truth IoU from 58.8 to 76.5, confirming improved label consistency.

Critical weaknesses
  • Incremental novelty of the CenDist reward over Time-R1. Combining IoU with a center-distance penalty was already introduced in Time-R1 (Appendix D, Eq. 9), with CenDist differing only in video-duration normalization, non-negative clipping, and mixture weighting. Direct comparisons against Time-R1's formulation under identical training settings are needed to justify these minor modifications.

  • Confounded attribution and limited mechanistic analysis of QA retention. The current evaluations do not disentangle whether QA retention stems from the proposed RL objective or from the curated data mixture and grounded-QA supervision.

  • Unestablished benefit of token-selective KL over uniformly weaker regularization. In Tables 6 and 11, fixing the text-token KL coefficient at 0.04 while lowering it on numeric and structural tokens simultaneously reduces average regularization strength, confounding selective placement with weaker overall KL. Under identical data, rewards, and budgets, comparing against (i) a tuned smaller uniform KL coefficient across all tokens and (ii) a random mask exempting the same token proportion reporting both grounding and direct QA is needed to verify whether targeting numeric and structural tokens outperforms simply weakening KL or masking arbitrary positions.

Questions for authors

None

4 comments

We thank the reviewer for the constructive comments, and for recognizing the well-motivated token-aware regularization and the effective data curation validated by human agreement. Below we address all the concerns.

W1 Incremental novelty of the CenDist reward over Time-R1

Thank you for pointing us to Time-R1's alternative reward. We have clarified the relationship precisely in the revised Appendix K, and mentioned that Time-R1 “explores a center-distance reward” in L257–258. Time-R1 (NeurIPS 2025 version, Appendix D, Eq. 9) defines IoU + 1 − |c_pred − c_GT| / L_GT. Our center-distance term instead normalizes by video duration, clips at zero, and is combined with IoU using the configured mixture weights. This simple modification improves learning under sparse temporal supervision, a key challenge in long-video grounding.

Following your suggestion, we systematically evaluate how these reward formulations behave as temporal sparsity increases. To control sparsity, we concatenate each training video with other videos randomly selected from the training set three times, which makes the training data three times sparser (denoted 3×). In the same way, we test 1×, 3×, 9×, and 27× with IoU, CenDist, and the exact Eq. (9) reward. Due to time constraints, each run starts from Qwen2.5-VL and uses 500 updates (~0.21 epoch). In the 1× setting, all reward variants perform equally well. However, the gap between CenDist and IoU grows as the training data becomes sparser. These results demonstrate that the benefit of the center-distance term becomes more pronounced as temporal sparsity increases.

We appreciate the idea behind Time-R1's Eq. (9), but it performs poorly in this setting. A possible explanation is that normalizing center errors by the ground-truth segment duration can produce large negative rewards when the target segment is short relative to the video, potentially making optimization less stable. Our video-duration normalization and zero clipping bound the center-distance term. These results highlight the importance of carefully normalizing, clipping, and combining reward signals in sparse settings.

W2 Confounded attribution and limited mechanistic analysis of QA retention

Thank you for highlighting the attribution of QA retention. During the rebuttal period, we conducted two complementary groups of ablations: (1) comparing KL configurations to examine the grounding-QA trade-off, and (2) removing grounded-QA supervision to examine whether retention persists without it. Unlike Table 4, all new training runs below exclude SceneTG, allowing us to examine retention without our introduced dataset.

(1) Effect of token-aware KL on grounding and QA retention. Table 6 (now Table 7) validates the grounding benefit of token-aware KL. To complement this analysis, we additionally evaluated direct-QA performance for the three KL variants (uniform KL, token-aware KL, and no KL) on four long-video QA benchmarks, as reported below.

LongVideoBenchMLVUVideoMME (Long)LVBench
Qwen2.5-VL (w/o finetuning)60.568.653.441.3
Uniform KL 0.0460.969.053.140.3
Token-aware KL 0.0458.968.953.840.7
No KL59.068.852.740.3
Token-aware KL 0.04, w/o QA data59.268.353.040.2

Token-aware KL and uniform KL exhibit mixed differences across the four QA benchmarks, with neither consistently outperforming the other. Substantial QA retention also persists without KL: relative to the backbone (the original Qwen2.5-VL without fine-tuning), the changes are −1.5, +0.2, −0.7, and −1.0 points on LongVideoBench, MLVU, VideoMME (Long), and LVBench, respectively. Thus, these results do not show a consistent QA-retention advantage from token-aware KL. Taken together with the grounding improvements in Table 6 (now Table 7), they support token-aware KL as improving grounding while broadly retaining direct-QA performance.

(2) Retention without grounded-QA supervision or SceneTG. We additionally trained the token-aware KL + IoU variant after removing QA-labeled training examples, with SceneTG still excluded. Relative to the corresponding model trained with QA examples, the scores change by +0.3, −0.6, −0.8, and −0.5 points across the four benchmarks. The resulting scores remain close to the backbone’s performance. This shows that substantial QA retention persists even when neither grounded-QA examples nor SceneTG is included in training.

The matched-data SFT results in Table 4 provide complementary evidence: SFT exhibits forgetting despite including grounded-QA supervision, showing that such supervision alone is insufficient to ensure retention.

Clarifying the role of token-aware KL. These controlled experiments support the grounding benefit of token-aware KL while showing that direct-QA performance is broadly retained. We have revised the corresponding statements in the introduction and method description. In particular, L112–114 now reads: “We propose token-aware KL regularization that encourages timestamp exploration while retaining regularization on language tokens, improving temporal grounding in controlled ablations.” We have revised L092–095 consistently.

W3 Unestablished benefit of token-selective KL over uniformly weaker regularization

We agree that tuned uniform KL and random-mask KL are important controls.

In our submission, the original ablation (Tables 6 and 11, now Tables 7 and 12) varies timestamp and non-timestamp regularization separately to examine their roles more clearly. We report two findings for these coefficients alone: 1) adjusting timestamp regularization alone improves temporal grounding learning and its learning dynamics, as shown by the token entropy curves during training in Figure 2; and 2) keeping non-timestamp regularization alone maintains similar QA retention.

Per your request, we conducted two additional experiments: uniform KL with β = 0.02, and a randomly masked KL variant. Due to time constraints, the training is still ongoing, but we have included the training curves in Appendix L, which show a clear trend: 1) random-mask KL exhibits similar IoU to conventional KL, 2) token-aware KL elevates the training IoU against conventional KL, and 3) a uniformly reduced KL is in between the vanilla KL and our token-aware KL.

We hope these clarifications and additional results have addressed your concerns. We have incorporated the new experiments and discussions into the revised paper, and we are happy to answer any further questions.

I thank the authors for their candid response. Based on the updated findings, I re-examined the paper's core contributions and conclusions. While the proposed token-aware KL effectively improves grounding performance (which is the actual contribution), the new results confirm that this is not directly linked to the paper's central motivation of preventing forgetting. As a result, the abstract, introduction, and main text are built around this mismatched motivation and the manuscript requires a major rewrite beyond what can be accommodated in this review cycle. Therefore, I will keep my score at 2.

Thank you for re-examining the paper and for recognizing the grounding benefit of token-aware KL. We appreciate your concern about consistency between the paper’s motivation and the new findings. We would like to distinguish the framework-level objective and empirical findings from the role of token-aware KL, and clarify the scope of the revisions already incorporated into the manuscript.

The framework-level objective and findings remain unchanged. Our objective is to improve long-video temporal grounding while retaining direct-QA capability. QA retention motivates our use of RL over SFT, and Table 4 evaluates this behavior through the comparison with SFT and other RL methods. The original submission already reported substantial QA retention in the evaluated RL baselines. Thus, the framework-level finding is improved grounding alongside broadly retained QA performance; it does not depend on token-aware KL providing a unique retention advantage.

The new control experiment clarifies the component’s contribution. Token-aware KL relaxes the reference-policy constraint on timestamp tokens while retaining regularization on language tokens. Its grounding benefit is supported by the controlled comparisons in Table 7, the coefficient sensitivity analysis in Appendix E, and the exploration analysis in Figure 2 in the original submission. The new direct-QA experiment clarifies component attribution without reversing the original grounding or QA results.

We have already made the corresponding distinction explicit in the manuscript. We recognize that the original introductory wording could suggest a stronger connection between language-token KL and QA preservation than the controlled evidence supports. We have corrected that wording. The scope of the current revision is as follows:

  • Abstract, L18: clarified the role of token-aware KL in encouraging timestamp exploration. The overall framework-level objective and empirical claims remain supported.
  • Introduction, L092–095 and L112–114: revised to describe the grounding role of token-aware KL precisely and distinguish it from QA retention.
  • Method and training objective: unchanged.
  • Experiments: supplemented with the controlled QA comparisons among uniform KL, token-aware KL, and no KL, together with the ablation removing grounded-QA supervision.

Specifically, L112–114 now reads: “We propose token-aware KL regularization that encourages timestamp exploration while retaining regularization on language tokens, improving temporal grounding in controlled ablations.” L092–095 has been revised consistently. These changes align the component-level interpretation with the new evidence while preserving the paper’s central objective, method, and supported empirical findings.

We therefore respectfully view the current revision as a clarification of attribution supported by completed controlled experiments, with the paper’s central contribution remaining unchanged. The revised text and additional evidence are already available for assessment in the current discussion. We would appreciate your reconsideration of whether a major rewrite beyond this review cycle is required in light of these concrete revisions.

If you believe a substantive mismatch remains, could you please identify the remaining passages or claims that require broader revision and clarify what aspect is not addressed by the revised text and controlled evidence? We would be happy to address those points directly.

Thank you again for your time and careful consideration of our responses and revisions.

Reviewer_qsnrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LongVTG-R1 for long-video temporal grounding while retaining general video question-answering ability. The approach combines token-aware KL regularization, which relaxes constraints on numeric and structural tokens, a center-distance reward that supplements IoU, and SceneTG data constructed from scene boundaries with uniqueness filtering. Experiments evaluate grounding, direct and grounding-assisted question answering, and recursive multi-segment retrieval, including comparisons with baselines trained on the same data mixture.

Critical strengths
  1. Meaningful improvements under matched-data comparisons. With the same approximately 40K training mixture, LongVTG-R1 improves mIoU over VideoChat-R1 by 9.3, 3.2, and 1.6 points on TaCoS, Ego4D-NLQ, and CG-Bench, respectively. These comparisons provide credible evidence that the complete training recipe contributes beyond the additional data alone.

  2. Evaluation of practical downstream value. The experiments examine both direct question answering and using grounding to select relevant evidence. The improvements in grounding-assisted QA demonstrate utility beyond localization metrics. The restricted-decoding SFT comparison also helps distinguish genuine QA degradation from output-format failures.

Critical weaknesses
  1. The role of token-aware KL in preventing forgetting is insufficiently validated. Table 4 shows that the complete model largely retains direct-QA performance, but does not isolate the proposed regularizer. Existing RL baselines also preserve much of the backbone’s QA ability, while approximately one quarter of the proposed training mixture contains grounded-QA supervision and the objective includes an answer-accuracy reward. These are plausible alternative contributors to retention. The KL ablations report grounding performance rather than the joint grounding–QA trade-off. Consequently, the evidence supports retention under the overall training recipe, but does not establish that selectively relaxing KL provides the claimed balance between timestamp exploration and knowledge preservation.

  2. Attribution of the full-scale improvement to individual algorithmic components remains incomplete. Table 3 usefully separates data curation from the complete algorithmic recipe: on CG-Bench, adding SceneTG raises the 16K VideoChat-R1 baseline from 0.8 to 9.2 mIoU, with LongVTG-R1 reaching 10.3 on the same mixture. However, the component ablations use only 30% of the data and one-third of the input resolution. Some incremental effects are small in this setting, including token-aware KL on CG-Bench and CenDist relative to GIoU. The matched-data results establish an overall algorithmic benefit, but do not determine how reliably the individual components account for that benefit at the scale of the main experiments.

Questions for authors
  1. Retention mechanism: Can the authors report direct-QA performance for the uniform-KL, token-aware-KL, and no-KL variants under the same data, rewards, and training budget? Joint grounding and QA results for these controlled variants would clarify whether selective regularization provides a distinct retention benefit.

  2. Component attribution: What evidence supports transferring the component-level conclusions from the reduced-scale ablations to the full-scale setting? Please clarify whether available experiments isolate token-aware KL and CenDist at that scale, and whether their relative contributions change with the larger data and input budgets.

  3. Related work and scope: Section 4.3 would benefit from a brief comparison with Towards One-to-Many Temporal Grounding (OMTG, ICML 2026). OMTG directly trains a model to generate the set of matching intervals, with explicit supervision for event count and retrieval completeness. LongVTG-R1 instead retrieves multiple intervals through recursive single-segment prediction, interval removal, and a learned no-answer stopping signal, without multi-segment fine-tuning. Discussing this distinction would clarify the contribution of the recursive approach and its implications for completeness and inference cost. This is a contextual clarification, not an independent reason for rejection or a request for additional experiments.

2 comments

We thank the reviewer for the thoughtful and constructive comments, and for recognizing the meaningful improvements under matched-data comparisons and our evaluation of practical downstream value. Below we address all the concerns.

W1 & Q1 The role of token-aware KL in grounding and QA retention

Thank you for requesting this controlled comparison. Following your suggestion, we conducted additional experiments comparing uniform KL, token-aware KL, and no KL under the same training data, rewards, and budget. We also conducted a separate experiment removing grounded-QA supervision. These experiments clarify the contribution of token-aware KL and the relationship between grounding performance and QA retention.

(1) Whether token-aware KL improves the grounding-QA trade-off. Table 6 (now Table 7) evaluates the grounding benefit of token-aware KL. To complement this analysis, we additionally evaluated direct-QA performance for the three KL variants on four long-video QA benchmarks, as reported below.

LongVideoBenchMLVUVideoMME (Long)LVBench
Qwen2.5-VL (w/o finetuning)60.568.653.441.3
Uniform KL 0.0460.969.053.140.3
Token-aware KL 0.0458.968.953.840.7
No KL59.068.852.740.3
Token-aware KL 0.04, w/o QA data59.268.353.040.2

Token-aware KL and uniform KL exhibit mixed differences across the four QA benchmarks, with neither consistently outperforming the other. Substantial QA retention also persists without KL: relative to the backbone (the original Qwen2.5-VL without fine-tuning), the changes are −1.5, +0.2, −0.7, and −1.0 points on LongVideoBench, MLVU, VideoMME (Long), and LVBench, respectively. Thus, these results do not show a consistent QA-retention advantage from token-aware KL. Taken together with the grounding improvements in Table 6 (now Table 7), they support token-aware KL as improving grounding while broadly retaining direct-QA performance.

(2) Whether retention persists after removing grounded-QA supervision. Removing grounded-QA examples changes the token-aware model’s QA scores from 58.9 to 59.2 on LongVideoBench, 68.9 to 68.3 on MLVU, 53.8 to 53.0 on VideoMME (Long), and 40.7 to 40.2 on LVBench. The resulting scores remain close to the backbone’s performance. This shows that substantial QA retention persists without grounded-QA supervision in the evaluated setting.

The matched-data SFT results in Table 4 provide complementary evidence: SFT exhibits forgetting despite including grounded-QA supervision, showing that such supervision alone is insufficient to ensure retention.

Clarifying the role of token-aware KL. These controlled experiments support the grounding benefit of token-aware KL while showing that direct-QA performance is broadly retained. We have revised the corresponding statements in the introduction and method description. In particular, L112–114 now reads: “We propose token-aware KL regularization that encourages timestamp exploration while retaining regularization on language tokens, improving temporal grounding in controlled ablations.” We have revised L092–095 consistently.

W2 & Q2 Attribution of the full-scale improvement to individual algorithmic components

We appreciate your acknowledgment that our experiments isolate data curation from the algorithm. We would like to clarify a possible misunderstanding regarding the benefits of the different components. According to the ablations, data curation (SceneTG) and the algorithm (Token-aware KL and CenDist) contribute differently across benchmarks: SceneTG accounts for most of the CG-Bench improvement (0.8 to 9.2 mIoU), whereas the algorithm contributes more on TaCoS and Ego4D-NLQ. In Table 6 (now Table 7), the effects of Token-aware KL and CenDist on TaCoS and Ego4D-NLQ are not small: in the reduced setting, adding token-aware KL alone raises the score to 4.9, already exceeding VideoChat-R1’s full-scale result of 4.5, while our full method further improves it to 5.7.

For CG-Bench, Figure 5 in Appendix E provides scaling evidence for the overall recipe: the gap widens as the training scale increases. This indicates that the algorithmic advantage of LongVTG-R1 becomes more pronounced at larger scales.

We greatly appreciate the suggestion to conduct full-scale ablations for each component. Each full-scale run takes 90 hours on 8 NVIDIA B200 GPUs (L345). Due to time and resource limits, we could not finish them within the rebuttal period. However, we believe the clarifications and additional evidence presented above support our component-level design choices.

Q3 Related work and scope: comparison with OMTG

Thank you for pointing us to OMTG; we have added a discussion of it to the revision in L478–504. As you mentioned, OMTG explicitly targets event count and completeness through carefully curated multi-segment supervision and dense video captions. LongVTG-R1 instead retrieves multiple intervals through recursive single-segment prediction without multi-segment fine-tuning. As further demonstrated by the results below, LongVTG-R1 generalizes from single-segment training to multi-segment grounding, achieving good per-segment accuracy and offering a general way to transfer single-segment capability to multi-segment grounding. As the number of segments increases, however, the inference cost grows linearly, and completeness may suffer from error accumulation. We believe OMTG and LongVTG-R1 complement each other and may inspire future designs for multi-segment temporal grounding.

In addition, we evaluated LongVTG-R1 on OMTG-Bench in a zero-shot manner, without multi-segment fine-tuning. LongVTG-R1 achieves EtF1 16.68, C-Acc 27.19, and tIoU 43.99, with tP/tR/tF1 of 58.14/55.09/52.83 at IoU 0.3, 41.89/41.33/39.13 at 0.5, and 25.19/26.79/24.89 at 0.7. Inference takes on average 3.96 recursive calls and 165.37 seconds per query on our evaluation setup. This indicates that LongVTG-R1 with recursion captures the one-to-many complexity and largely outperforms standard open-source video temporal grounding models (e.g., Qwen2.5-VL-72B, VideoChat-R1-7B, and UniTime, which all obtain 0% C-Acc). Although OMTG achieves higher scores with dedicated multi-segment supervision, LongVTG-R1 achieves these results by extending its single-segment capability through recursive inference, without multi-segment fine-tuning.

We hope these clarifications and additional results have addressed your concerns. We have incorporated the new experiments and discussions into the revised paper, and we are happy to answer any further questions.

Comments on the paper

1 comment

Dear SAC, AC, and Reviewers,

Thank you for your time and consideration of our submission. We have provided point-by-point responses to all three reviewers and revised the manuscript. Below, we summarize the additional experiments, corresponding revisions, and current discussion status.

  • 1. QA retention and the role of token-aware KL. We added direct-QA evaluations on four benchmarks comparing uniform KL, token-aware KL, and no KL. These results show QA retention also persists in the no-KL and no-grounded-QA ablations. Taken together with the grounding improvements in Table 7, these results support token-aware KL as improving grounding while broadly retaining direct-QA performance.

    • The framework-level objective and findings remain unchanged: improving long-video temporal grounding while retaining direct-QA capability. QA retention motivates our use of RL over SFT, while timestamp exploration motivates token-aware KL. The new experiments clarify component attribution without reversing the original grounding or QA results. We have revised the abstract (L18) and introduction (L092–095 and L112–114) to distinguish the grounding role of token-aware KL from QA retention.
  • 2. Algorithmic contribution at scale.
    We clarified that the requested full-scale comparison is already included as “VideoChat-R1 (fair comparison)” in Table 1. This baseline uses the same training data and budget as LongVTG-R1, but retains conventional KL regularization and the IoU reward, thereby evaluating the combined algorithmic contribution. Figure 5 in Appendix E further shows that the gap between VideoChat-R1 and LongVTG-R1 widens as the training scale increases.

  • 3. Additional reward and KL comparisons.
    Appendix K adds comparisons of CenDist, IoU, and Time-R1’s Eq. (9) reward under increasing temporal sparsity. The advantage of CenDist over IoU grows as the training data becomes sparser. We also clarified the differences in reward normalization, clipping, and combination compared with Time-R1. Appendix L adds training curves for uniformly reduced KL and random-mask KL. The curves show higher training IoU for token-aware KL than for either variant.

  • 4. Multi-segment grounding and related work.
    We added a discussion of OMTG and a zero-shot evaluation on OMTG-Bench without multi-segment fine-tuning (Section 4.3, Table 6). LongVTG-R1 achieves 16.68 EtF1 and 27.19 C-Acc, demonstrating transfer from single-segment training to multi-segment grounding through recursive inference. The revision also reports inference cost and discusses limitations in completeness, including potential error accumulation as the number of segments increases.

Scope of the revision. The follow-up discussion with Reviewer 2CKu has focused on revision scope; the changes clarify the grounding contribution of token-aware KL, while the method, training objective, and framework-level findings remain unchanged. The targeted revisions and supporting experiments are already incorporated into the manuscript and available for assessment in the current review cycle.

Thank you again for your time and constructive suggestions.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.