- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the knowledge forgetting in long-video temporal grounding. It proposes Token-aware KL Regularization and Center Distance Reward (CenDist), and builds the SceneTG dataset to improve training data quality. Experiments show superior temporal grounding performance while largely preserving general video QA ability.
Critical strengths
-
The method achieves superior results on several long-video temporal grounding benchmarks.
-
This paper is well-written and easy to understand.
Critical weaknesses
-
RL has already been widely used for video temporal grounding, such as in VideoChat-R1 and TimeLens2. This paper is built on existing RL framework and modifies the loss and reward terms, its main technical contribution is not clear.
-
The design of token-aware KL is ad hoc and does not verified on more related tasks. It mainly adjusts the KL regularization weights for different token types. Although the paper provides some mathematical analysis, it lacks a clear explanation for the token grouping and weight choices. It also lacks evidence that this design helps reduce knowledge forgetting.
-
Table 6 shows marginal gains from the proposed components, while the improvements in Table 1 are much larger. I suggest adding a full-scale GRPO baseline without Token-aware KL and CenDist, using exactly the same training data, training steps, and settings. This would help clarify the actual contribution of the proposed methods.
Questions for authors
In Weakness.
2 comments
W2 Token-aware KL: rationale for token grouping and weights, and evidence for reducing forgetting
Rationale for the design. Token-aware KL is a structured relaxation of uniform regularization. Timestamp tokens encode the temporal boundaries directly evaluated by the grounding reward, motivating greater freedom to explore alternative boundary predictions. We therefore relax their KL constraint while retaining regularization on language and answer generation.
Figure 2 examines this behavior through separate entropy curves for the token groups, showing how relaxing timestamp regularization affects exploration during training. Entropy curves are also used to analyze exploration and training dynamics in prior RL work, such as DAPO [R1]. Our entropy analysis provides evidence about exploration behavior, while the grounding ablations evaluate whether this design improves task performance.
Choice of KL weights. Appendix E of the original submission already includes a controlled sensitivity analysis: Table 11 (now Table 12) fixes β_text at 0.04 and varies β_time over {0.04, 0.02, 0}. As β_time decreases, CG-Bench [email protected] increases from 9.27 to 9.43 to 9.84. Thus, removing the timestamp-token penalty gives the best result among the tested settings while retaining language-token regularization. These results provide empirical support for the selected allocation of KL regularization. NExT-GQA also shows a consistent trend. These results support the choice of the weights.
Evidence regarding forgetting. Following the reviewers’ suggestions, we additionally evaluated direct-QA performance under uniform KL and token-aware KL on four long-video QA benchmarks:
| LongVideoBench | MLVU | VideoMME (Long) | LVBench | |
|---|---|---|---|---|
| Qwen2.5-VL (w/o finetuning) | 60.5 | 68.6 | 53.4 | 41.3 |
| Uniform KL 0.04 | 60.9 | 69.0 | 53.1 | 40.3 |
| Token-aware KL 0.04 | 58.9 | 68.9 | 53.8 | 40.7 |
Both variants broadly retain the original model’s QA performance, but their differences are mixed across benchmarks. These results do not show a consistent QA-retention advantage from token-aware KL over uniform KL. Together with the grounding improvements in Table 6 (now Table 7) , they support token-aware KL as improving grounding while broadly retaining direct-QA performance.
Clarifying the role of token-aware KL. These controlled experiments support the grounding benefit of token-aware KL while showing that direct-QA performance is broadly retained. We have revised the corresponding statements in the introduction and method description. In particular, L112–114 now reads: “We propose token-aware KL regularization that encourages timestamp exploration while retaining regularization on language tokens, improving temporal grounding in controlled ablations.” We have revised L092–095 consistently.
[R1] Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., ... & Wang, M. DAPO: An open-source LLM reinforcement learning system at scale. In NeurIPS, 2025.
W3 Marginal gains in Table 6 (now Table 7) vs. larger gains in Table 1; full-scale GRPO baseline without Token-aware KL and CenDist
Thank you for suggesting this controlled comparison. The requested full-scale GRPO baseline is already included as “VideoChat-R1 (fair comparison)” in Table 1. This baseline uses the same training data, training steps, and settings as LongVTG-R1, but retains conventional KL regularization and the IoU reward, without token-aware KL or CenDist. Its comparison with LongVTG-R1 therefore evaluates the combined algorithmic contribution at full scale, with the training data and budget held fixed.
The contributions of data and algorithmic changes differ across benchmarks. On CG-Bench, SceneTG accounts for much of the improvement, raising the 16K VideoChat-R1 baseline from 0.8 to 9.2 mIoU; LongVTG-R1 further improves this to 10.3 using the same data mixture. On TaCoS and Ego4D-NLQ, the algorithmic changes make a larger contribution. On Ego4D-NLQ, even in the reduced setting, adding token-aware KL alone raises mIoU to 4.9, already exceeding the full-scale GRPO baseline with SceneTG at 4.5, while our full method further improves it to 5.7.
The training scale also matters when interpreting the gains. Table 6 isolates individual components in the reduced setting, whereas Table 1 evaluates the complete method and its matched baseline at full scale. Figure 5 in Appendix E further shows that the gap between VideoChat-R1 and LongVTG-R1 widens as more training data is used. This supports the observation that the combined algorithmic advantage becomes more pronounced at larger training scales.
We hope these clarifications and additional results have addressed your concerns. We have incorporated the new experiments and discussions into the revised paper, and we are happy to answer any further questions.
We thank the reviewer for the constructive comments, and for recognizing the strong results on long-video temporal grounding benchmarks and the clarity of the writing. Below we address all the concerns.
Our work differs from prior RL-based temporal grounding methods in three connected aspects.
1) Problem scope: temporally sparse grounding in longer videos. Our CG-Bench evaluation includes videos up to 6,325 seconds (105.4 minutes). This is substantially longer than the predominantly short-video setting in the original VideoChat-R1 paper, whose grounding evaluation centers on Charades-STA and ActivityNet, with average video durations of roughly 30 seconds and 2 minutes, respectively. Although TimeLens2 also studies long videos, sparsity, and ambiguity through VUE-TR and VUE-TR-V2, our particular focus is the joint grounding-QA trade-off: learning sparse-event localization over long timelines while retaining the model's general question-answering capability.
2) Three technical changes motivated by this setting. When a target occupies only a small fraction of a long video, many sampled predictions receive zero IoU; ambiguous boundaries further introduce noisy rewards.
Thus, alongside the loss and reward contributions, our third contribution supplies cleaner, less ambiguous long-video grounding supervision.
3) Extending the benefits of grounding to long-video QA and multi-segment localization. We evaluate direct QA retention and show that selecting evidence with the learned grounding policy improves long-video QA in both ground-then-answer and unified settings. We also reuse the single-segment policy recursively to retrieve multiple relevant intervals, without multi-segment fine-tuning. The VideoChat-R1 paper does not investigate this multi-segment setting. TimeLens2 learns interval-set prediction with explicit multi-span supervision; our recursive approach instead transfers a policy learned from single-segment supervision to multi-segment grounding.