acceptodds
Under review as a conference paper at ICLR 2027

DTP-OPSD: Reinforcement Learning with Distractor-Aware Temporal-Privilege On-Policy Self-Distillation for Temporal Video Grounding

Abstract

Temporal Video Grounding (TVG) aims to localize the start and end times of an event described by a language query in a video. The task becomes particularly challenging when the same video contains multiple repeated, similar, or opposite events. On these hard examples, all responses sampled by Group Relative Policy Optimization (GRPO) may miss the target and receive zero temporal Intersection-over-Union (tIoU). Their group-relative advantages are then all zero, leaving no useful policy gradient signal. Further analysis shows that the model is often able to recognize the queried event but is distracted by other events in the same video and selects the wrong occurrence. To address this, we propose DTP-OPSD, a reinforcement learning framework with distractor-aware temporal-privilege on-policy self-distillation. The student observes the clean video and generates responses on-policy, while a shared-weight teacher observes a privileged video in which the target window is marked by red full-frame borders and same-video distractor windows by blue full-frame borders. For the distillation loss, the teacher evaluates student-generated prefixes and provides dense token-level Jensen–Shannon divergence supervision. This supervision remains informative even when GRPO provides no policy gradient signal. We suppress distillation when the teacher does not outperform the student and strengthen it for groups where the teacher performs better but all student rollouts receive zero tIoU. We also construct a compact hard-interference training set through VLM screening and human verification. With only 300 additional post-training examples, DTP-OPSD achieves the highest mIoU among the compared open-source 8B models on Charades-TimeLens, ActivityNet-TimeLens, and QVHighlights-TimeLens, reaching 55.93, 54.77, and 66.71, respectively. Compared with Pure GRPO trained on the same data, DTP-OPSD improves all 12 reported metrics, with [email protected] gains of 1.28, 1.62, and 1.78 points on the three benchmarks.

Primary area:
reinforcement learning
Keywords:
video temporal grounding&reinforcement learning&self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7FTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies temporal video grounding with multimodal LLMs trained by GRPO. When a video contains repeated or similar occurrences of the queried event, all sampled rollouts may pick a wrong occurrence and receive zero tIoU. These all-zero groups have zero group-relative advantage and provide nogradient. The authors call them dead groups and show that most dead-group rollouts overlap a verified distractor. The proposed DTP-OPSD adds an on-policy self-distillation term to GRPO. A teacher sharing the same weights sees a privileged video in which the expanded ground-truth window is marked with red frame borders and verified distractors with blue borders. The teacher scores the student's own prefixes, and a token-level Jensen–Shannon divergence pulls the student toward it. A three-level controller sets the per-group distillation weight to 0 when the teacher does not outperform the student, 2 for dead groups where the teacher does better, and 1 otherwise. The authors also build HI-TVG-300, a 300-example hard-interference set derived from TimeLens-100K. Starting from the TimeLens-SFT Qwen3-VL-8B checkpoint, one epoch on these examples improves over Pure GRPO by 0.50, 1.15, and 0.76 mIoU on the three TimeLens benchmarks, with seed standard deviations under 0.12. Supporting evidence includes ablations of the controller, privileged view, and data curation, a GRPO plus ground-truth SFT control, a compute-matched GRPO control, dead-group recovery statistics, and results at 2B and 4B scales.

Critical strengths
  1. The dead-group diagnosis is a precise and well-supported characterization of a real GRPO failure in TVG. Table 7 shows that 79.8 percent of dead-group rollouts overlap a verified distractor, which ties the problem to occurrence selection rather than recognition and motivates the method directly.
  2. The privileged teacher is parameter-free and input-only. Because teacher and student share weights and differ only in the video, the capability gap comes solely from the temporal cue, and nothing is added at inference.
  3. Distilling on student-generated prefixes keeps the supervision on-policy. The teacher evaluates the exact states the student visits, so the JSD signal does not vanish when all rollout rewards are zero, which is the case GRPO cannot handle.
  4. The distractor-aware design is validated rather than assumed. Table 3 shows that misplacing the red border hurts (64.44), that random blue borders slightly hurt (65.94), and that verified distractors help (66.71), so both cue types are shown to matter.
  5. The method is data-efficient. With 300 curated examples and 37 updates, DTP-OPSD matches or exceeds TimeLens-8B, which was trained with RL on 100K examples, and gains hold at 2B and 4B scales.
Critical weaknesses
  1. The dead-group problem has standard, cheaper fixes that are not compared. A temporal-distance shaped reward (used by Time-R1 and VideoChat-R1, and explicitly removed here) gives every rollout a nonzero gradient without any privileged view.
  2. The verified distractor annotations could be used directly as a penalty reward. The method already requires these intervals, so a negative reward for overlapping a blue window is the most direct use of the same information and is absent.
  3. Hint-injected or off-policy guided GRPO is not tested. Mixing the teacher's privileged rollouts into the student's group would revive dead groups using the same teacher and compute, without a distillation loss.
  4. The privileged distillation signal contributes little on its own. Fixed JSD gives 55.48, 54.02, and 65.78 mIoU versus Pure GRPO's 55.43, 53.62, and 65.95, so on QVHighlights the distillation alone is slightly worse than GRPO and the entire gain comes from the heuristic weighting.
  5. Figure 4 compares only to TimeLens-8B on OMTG-Bench and QV-M2. Without Pure GRPO on these benchmarks it is unclear whether the method itself helps under multi-moment annotations.
  6. The headline state-of-the-art claim is mostly inherited. Pure GRPO on 300 examples already matches TimeLens-8B (55.43 versus 55.28 on Charades), so most of the reported standing comes from the TimeLens-SFT initialization plus GRPO rather than from DTP-OPSD.
Questions for authors
  1. Why does JSD-only training collapse to 15.04 mIoU in 37 small updates? Is this a format collapse or genuinely wrong intervals?
  2. Given that Fixed JSD is no better than Pure GRPO on QVHighlights, how do you interpret the claim that privileged JSD "remains informative" where GRPO gives no signal?
  3. Which checkpoint performed the teachability filtering, TimeLens-SFT or base Qwen3-VL-8B? If TimeLens-SFT, the random-subset comparison in Table 4 partly measures hard-example mining on the training model rather than curation quality.
4 comments

Thank you for the careful review and for recognizing the motivation, shared-weight teacher design, and supporting analyses. We address all six weaknesses and three questions below, combining the overlapping points in W4 and Question 2.

Q1 (W1): Could a temporal-distance reward address all-zero groups more simply?

Answer: Thank you for highlighting temporal-distance rewards as an important baseline. Such rewards can distinguish zero-IoU predictions when their temporal distances from the target differ. However, GRPO depends on reward differences within a group, rather than simply whether individual rewards are nonzero.

For example, consider a query asking for the second time a person picks up a cup. If all eight rollouts predict the same interval corresponding to the first occurrence, they have identical center distances from the target. Adding a distance term therefore gives all eight responses the same reward, leaving their group-relative advantages at zero. This example illustrates the condition required for distance shaping to restore a relative learning signal: it must introduce reward differences among the sampled responses. When the privileged teacher performs better, DTP-OPSD can instead supply additional token-level guidance on the student’s prefixes even when the student rewards are tied.

To examine the practical effect of distance shaping, we added a controlled baseline using the IoU-plus-center-distance reward defined in Appendix Eq. (9) of Time-R1. It uses the same TimeLens-SFT initialization, HI-TVG-300 data, one-epoch schedule, and rollout and evaluation settings as Pure GRPO; only the reward function changes.

MethodCharades-TimeLens mIoUActivityNet-TimeLens mIoUQVHighlights-TimeLens mIoU
Pure GRPO55.4353.6265.95
GRPO + IoU–center reward55.4053.5966.13
DTP-OPSD55.9354.7766.71

The distance term leaves performance nearly unchanged on Charades-TimeLens and ActivityNet-TimeLens and improves QVHighlights-TimeLens mIoU by 0.18. DTP-OPSD exceeds this baseline by 0.53, 1.18, and 0.58 mIoU points, respectively. Thus, the example explains when distance shaping can provide a relative signal, while the matched training comparison shows that this reward modification alone does not reach DTP-OPSD’s final grounding accuracy in our setting.

We also clarify that VideoChat-R1 uses IoU plus format rewards for temporal grounding, without a temporal-distance term.

Q2 (W2): Why not use the verified distractor annotations directly as a penalty reward?

Answer: Thank you for suggesting this direct use of the annotations. We added a GRPO + distractor penalty baseline with the same initialization, training data, one-epoch schedule, and rollout and evaluation settings as Pure GRPO.

To protect valid target regions, we take the union of verified distractor intervals and remove its overlap with the ground-truth target. Denoting the remaining region by D−D^{-}, we use

-\alpha\frac{|\hat{\tau}\cap D^{-}|}{|\hat{\tau}|}, \qquad \alpha=1.$$ Thus, overlap with the target itself is never penalized, even when target and distractor annotations overlap. | Method | Charades-TimeLens mIoU | ActivityNet-TimeLens mIoU | QVHighlights-TimeLens mIoU | | :--- | ---: | ---: | ---: | | Pure GRPO | 55.43 | 53.62 | 65.95 | | GRPO + distractor penalty | 55.49 | 53.58 | 65.24 | | DTP-OPSD | 55.93 | 54.77 | 66.71 | The penalty baseline performs similarly to Pure GRPO on Charades-TimeLens and ActivityNet-TimeLens, while its QVHighlights-TimeLens mIoU decreases by 0.71 points. DTP-OPSD exceeds this baseline by 0.44, 1.19, and 1.47 mIoU points, respectively. Both approaches retain the tIoU reward and use the same target and distractor annotations. The added penalty supplies sequence-level feedback for avoiding known distractor regions. DTP-OPSD uses those annotations to construct a privileged teacher view and provides next-token guidance on student-generated prefixes. Although the penalty baseline avoids teacher-side computation, this direct use of the annotations does not match DTP-OPSD’s grounding accuracy under the tested setting.

Q3 (W3): Why not mix privileged teacher rollouts into the student’s GRPO group?

Answer: Thank you for this helpful suggestion. Guided rollouts are a reasonable alternative. Our choice of same-prefix distillation concerns where and how teacher supervision is applied.

Although teacher and student share weights, they generate responses under different inputs: the teacher sees the privileged video, while the student sees the clean video. Teacher responses therefore cannot simply be treated as on-policy samples from the clean-video student without accounting for their different sampling distribution.

Adding successful teacher responses can provide positive sequence-level examples and change the group-relative advantages of the student responses. DTP-OPSD instead keeps the original student-side GRPO groups and asks the teacher to evaluate the next-token distribution at each student-generated prefix. Supervision is therefore computed at the generation states actually visited by the student.

The independent teacher rollouts estimate teacher quality for the controller; the JSD targets are computed on the student’s own responses. This explains our design choice without assuming that guided rollouts cannot also be effective.

Q4 (W4 and Original Question 2): If Fixed JSD provides limited gains, what supports the role of privileged supervision and adaptive weighting?

Answer: Thank you for highlighting this distinction. We agree that the results do not support applying privileged JSD uniformly. The adaptive controller is an integral part of DTP-OPSD, and the existing controller ablation directly evaluates its two components.

MethodCharades-TimeLens mIoUActivityNet-TimeLens mIoUQVHighlights-TimeLens mIoU
Pure GRPO55.4353.6265.95
Fixed JSD55.4854.0265.78
Rescue only55.7754.5066.33
Suppression only55.6254.3766.24
Full controller55.9354.7766.71

Rescue only differs from Fixed JSD by doubling the JSD weight for all-zero student groups in which the teacher performs better; it never suppresses JSD. This change improves mIoU by 0.29, 0.48, and 0.55 points, respectively. Suppression only removes JSD when the teacher does not outperform the student, without applying the rescue multiplier. Combining both components gives the highest mIoU on all three benchmarks.

The fixed-subset analysis separately examines recovery by the full method. As also discussed in our response to Reviewer Y1As, Q2, we evaluate the same 142 examples that formed all-zero groups at initialization. An example is recovered when at least one of eight diagnostic completions has positive target tIoU.

MethodRecovery of the initial 142 all-zero groups
Pure GRPO, one epoch38.7%
Pure GRPO, compute matched48.6%
DTP-OPSD, one epoch64.1%

These analyses provide complementary evidence: the controlled ablation supports the contribution of rescue weighting to final benchmark performance, while the fixed-subset analysis shows that the full method recovers more initial all-zero groups, including compared with longer GRPO training under a matched compute budget.

By saying that privileged JSD can provide a signal when GRPO provides none, we mean that teacher–student distribution differences can yield token-level supervision even when group-relative advantages are zero. This does not guarantee that every teacher prediction is useful. The controller determines when to suppress, retain, or strengthen that supervision.

Q5 (W5): Does DTP-OPSD improve over the matched Pure GRPO baseline on OMTG-Bench and QV-M²?

Answer: Thank you for requesting this comparison. We evaluated the existing Pure GRPO checkpoint using the same single-interval, best-match protocol as Figure 4. DTP-OPSD improves over Pure GRPO by 0.58 mIoU on OMTG-Bench and 0.91 mIoU on QV-M².

MethodOMTG-Bench mIoUQV-M² mIoU
TimeLens-8B58.2054.37
Pure GRPO57.8155.30
DTP-OPSD58.3956.21

All models predict one temporal interval, which is evaluated against its best-matching ground-truth interval. Results cover all 320 OMTG-Bench queries and all 323 QV-M² test queries.

TimeLens-8B is included as a reference model. The comparison between Pure GRPO and DTP-OPSD holds the TimeLens-SFT initialization and HI-TVG-300 training data fixed, directly measuring the contribution of our training method. These results concern identifying one valid occurrence under multi-moment annotations, rather than retrieving every annotated interval.

Q6 (W6): How much of the reported performance comes from TimeLens-SFT rather than DTP-OPSD?

Answer: Thank you for raising this attribution question. We acknowledge TimeLens-SFT as a strong foundation and assess our contribution through the matched Pure GRPO comparison: DTP-OPSD improves mIoU by 0.50, 1.15, and 0.76 points, with [email protected] gains of 1.28, 1.62, and 1.78 points.

For the Qwen3-VL-8B branch used here, the released TimeLens pipeline performs SFT before GRPO training. The model relationships are summarized below.

ModelStarting checkpointSubsequent training
TimeLens-SFTQwen3-VL-8BTimeLens SFT stage
TimeLens-8BTimeLens-SFTOriginal TimeLens GRPO training
Pure GRPOTimeLens-SFTGRPO on HI-TVG-300
DTP-OPSDTimeLens-SFTGRPO with adaptive privileged JSD on HI-TVG-300

The last two rows share the same initialization, 300 training examples, and one-epoch schedule. Their difference measures the incremental benefit of DTP-OPSD.

We also examine occurrence correction, as discussed with Reviewer Y1As, Q3, and Reviewer XPsp, Q2. On a fixed set of 120 evaluation examples where the TimeLens-SFT prediction had zero target overlap and overlapped a verified distractor, Pure GRPO corrects 24/120 examples to tIoU≥0.7, whereas DTP-OPSD corrects 38/120, increasing the correction rate from 20.0% to 31.7%. This provides an example-level view of the improvement beyond the shared initialization.

Q7 (Original Question 1): Why does JSD-only training fall to 15.04 mIoU? Is this a formatting failure or inaccurate localization?

Answer: Thank you for this important question. Our diagnostic indicates that the degradation primarily reflects inaccurate localization: 99.1% of JSD-only outputs parse into valid intervals, but mIoU on those valid predictions is only 15.18.

We separate output validity from localization accuracy on QVHighlights-TimeLens using the existing evaluation parser:

MethodValid outputsmIoU on valid outputsmIoU on all examples
TimeLens-SFT99.6%57.5757.34
JSD-only99.1%15.1815.04
DTP-OPSD99.9%66.7866.71

In the JSD-only ablation, we remove the GRPO loss while retaining student and teacher rollout rewards for controller estimation. JSD encourages agreement between the clean and privileged views, but agreement alone does not ensure correct temporal localization. Because both branches share the updated parameters, the teacher also changes during training. GRPO supplies the direct optimization objective based on target tIoU.

Related instability has been reported in arXiv:2605.18740, which observes collapse when using the current policy as teacher without teacher regularization, and arXiv:2607.01763, which reports forgetting and collapse in continual on-policy self-distillation. In our setting, retaining GRPO alongside privileged JSD yields 66.71 mIoU. The objective ablation and output diagnostic support using JSD as auxiliary supervision rather than as the standalone training objective.

Q8 (Original Question 3): Which checkpoint performs teachability filtering, and does the data ablation also measure hard-example selection?

Answer: Thank you for requesting this clarification. We use TimeLens-Qwen3-VL-8B-SFT, referred to as TimeLens-SFT throughout the paper, for teachability filtering. It is also the initialization for the subsequent training experiments.

The same checkpoint is evaluated with clean and privileged video inputs. We retain examples whose mean privileged-teacher tIoU exceeds 0.7 while the mean clean-student tIoU is below 0.3. The filter therefore includes model-dependent hard-example selection, together with a check that the privileged view provides useful guidance.

Accordingly, the change from 66.05 to 66.71 mIoU in Table 4 reflects adding this teachability filter after human verification within our curation pipeline. It should not be interpreted solely as an improvement in annotation quality. TimeLens also includes a model-based filtering stage before GRPO training; our additional criterion concerns the teacher–student performance gap under different video inputs.

The main method comparison then holds the selected data fixed: Pure GRPO and DTP-OPSD both start from TimeLens-SFT and train on exactly the same HI-TVG-300 examples. This measures their difference on a common curated training set.

Dear Reviewer 7FTP,

Thank you again for your constructive feedback on our submission. We wanted to follow up on the two reward-based alternatives you suggested. Q1 and Q2 of our rebuttal report GRPO with an IoU-plus-center-distance reward and GRPO with a distractor-overlap penalty.

Both baselines use the same initialization, training data, and training schedule as Pure GRPO. DTP-OPSD achieves higher mIoU than both on all three benchmarks.

We would appreciate your feedback on these additions and how they affect your assessment. Please let us know if you have any further questions.

Best regards,
The Authors

Reviewer_XPspOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve the temporal video grounding problem. The authors observed that previous methods often recognize the queried event and are distracted by other events. So, the wrong occurrence are often selected. To solve this problem, a reinforcement learning framework with distractor-aware temporal-privilege on-policy self-distillation, short for DTP-OPSD, is proposed. The main idea is that a shared-weight teacher employes training-only target and distractor cues to provide token-level supervision on student-generated prefixes. Experiments show that the proposed approach receives good results on multiple benchmarks.

Critical strengths
  • The motivation of this paper is interesting. The authors find the reasons that make GRPO signals not dense and present a new on-policy self-distillation method to solve this problem.

  • A high-quality hard-interference training set is proposed to help the reinformence learning model work better.

  • Experiments on Charades-STA, ActivityNet Captions, and QVHighlights show that the proposed approach receives good results and ablation experiments are also performed to help analyze the importance of each component.

Critical weaknesses
  • The authors use distractor-aware temporal privilege for OPSD. The motivation is clear. However, it seems that this part is not clearly explained. In addition, the authors did not clearly describe how the distractor-aware temporal privilege works in the experiment section. As this is listed as one of the contributions of this paper, it should be clearly explained.
  • Some critical problems have been mentioned in the introduction section and a new method is also proposed to solve the problems. However, from the experiment section, it seems that the improvement over the GRPO baseline is not significant. In most previous works, the improvement over baselines are large but in this paper this is not. I wonder whether the proposed approach works well. For example, on the Charades-TimeLens dataset, the mIoU score is 55.93, but the GRPO baseline is 55.43, which is slightly worse than the proposed approach.
  • The Qwen3-VL-8B model is used as the base model. Recently, some new open-sourced Qwen series models are released. These new models are stronger. I am curious about whether the proposed approach could work well on stronger base models, like Qwen-3.5 series.
  • Most existing works do not perform well when the GT event is really short. I think the authors should show some results on really short events.
Questions for authors
  • It seems that the HI-TVG-300 split plays an important role in this paper. I am curious about how HI-TVG-300 helps improve the results. I think the author should give more explanations on this.
  • It would be better to show more visual results in the main paper or in the appendix.
8 comments

We thank the reviewer for the constructive assessment and for recognizing the motivation, training set, and empirical results. We address the four weaknesses and two questions below. Where relevant, we reference the analyses discussed with Reviewer Y1As and reproduce their key findings here.

Q1 (W1): How does distractor-aware temporal privilege work, and what evidence supports its effectiveness?

Answer: Thank you for pointing out the need for a clearer explanation. As also discussed with Reviewer Y1As in Q1 and Q5, our motivation distinguishes recognizing an action from selecting its correct occurrence. If all eight sampled predictions select incorrect occurrences with zero target overlap, their group-relative advantages are zero.

DTP-OPSD addresses this through three components:

  1. Training-only temporal privilege. The teacher sees red borders around the expanded target region and blue borders around verified distractors. The student sees the clean video. Both branches share weights.
  2. Supervision on student-generated prefixes. The teacher evaluates the student’s own prefixes and provides next-token distributions for JSD supervision. This signal does not require differences among student rollout rewards.
  3. Adaptive supervision. Teacher rollouts estimate whether the privileged view improves localization. The controller suppresses JSD when the teacher does not outperform the student and strengthens it for eligible all-zero student groups.

The final model receives clean videos, with no additional inference inputs or parameters. Table 3 evaluates the privileged-view design:

Privileged view during trainingQVHighlights-TimeLens mIoU
Red-shuffled target64.44
Target only66.12
Target + random distractors65.94
Target + verified distractors66.71

These results support the importance of correct target placement and verified distractors. In the additional color-swap experiment, student mIoU changes only from 66.71 to 66.68, suggesting that the particular color assignment is less important than the temporal cues.

The motivating failure also appears beyond our curated data: among 500 randomly sampled TimeLens-100K queries, the dead-group rate rises from 8.2% without distractors to 31.7% with at least three. We report this distribution in Q5 below.

Q2 (W2): Are the gains over Pure GRPO meaningful enough to demonstrate effectiveness?

Answer: Thank you for raising this question. As also discussed with Reviewer Y1As in Q2–Q4, we examine the gains through matched training, wrong-occurrence correction, and interference-specific evaluation.

Both methods start from TimeLens-SFT and train on HI-TVG-300 for one epoch. DTP-OPSD improves mIoU by 0.50, 1.15, and 0.76 points across the three TimeLens benchmarks. At the stricter [email protected] threshold, the gains are 1.28, 1.62, and 1.78 points.

The 600-query duration-stratified analysis in Reviewer Y1As, Q4 shows larger gains under high interference. The following table summarizes the mIoU gains over Pure GRPO; each underlying duration/interference cell contains 50 queries.

Target-to-video duration ratioGain with 0 distractorsGain with ≥3 distractors
≤5%+0.10+1.90
>5% and ≤20%+0.20+2.30
>20%+0.16+1.79

We also evaluated 120 queries whose initial TimeLens-SFT predictions had zero target overlap and overlapped verified distractors. DTP-OPSD corrects 38/120 to tIoU ≥0.7, compared with 24/120 for Pure GRPO and 27/120 for compute-matched Pure GRPO.

Finally, at approximately matched computation of 93.6 GPU-hours, DTP-OPSD achieves 66.71 mIoU, compared with 66.09 for Pure GRPO. These results show that the overall gains include improved correction of wrong occurrences and larger benefits in the high-interference setting targeted by our method.

Q3 (W3): Does the method also work with newer backbones, such as Qwen3.5?

Answer: Thank you for this helpful suggestion. We conducted additional experiments with Qwen3.5-9B. Both methods start from the same released checkpoint and train for one epoch on the full HI-TVG-300 dataset, without additional SFT. This differs from our original 8B experiments, which start from TimeLens-SFT.

MethodCharades-TimeLens mIoUActivityNet-TimeLens mIoUQVHighlights-TimeLens mIoU
Qwen3.5-9B48.152.363.5
Qwen3.5-9B + Pure GRPO55.953.566.3
Qwen3.5-9B + DTP-OPSD56.554.966.9

DTP-OPSD improves over Pure GRPO by 0.6, 1.4, and 0.6 mIoU points. These results support its effectiveness on Qwen3.5-9B when RL starts directly from the released checkpoint.

Q4 (W4): How does the method perform on very short target events?

Answer: Thank you for highlighting this challenging setting. Beyond the relative-duration analysis discussed with Reviewer Y1As in Q4, we evaluated two diagnostic sets from QVHighlights-TimeLens: 150 queries with target duration ≤2 seconds and 150 with target duration >2 seconds. Both methods were evaluated on the same queries.

Ground-truth event durationQueriesPure GRPO mIoUDTP-OPSD mIoUGain
≤2 seconds15045.6046.72+1.12
>2 seconds15055.8057.21+1.41

Very short events remain challenging for both methods, but DTP-OPSD improves mIoU by 1.12 points on the evaluated targets lasting at most two seconds. This provides direct evidence that its benefits extend to the short-event setting.

Q5 (Original Question 1): Why is HI-TVG-300 important, and how does it help improve performance?

Answer: Thank you for asking us to clarify the role of the training set. HI-TVG-300 concentrates a limited training budget on difficult same-video interference with verified target and distractor annotations.

As discussed with Reviewer Y1As in Q1, we randomly sampled 500 query–video pairs from TimeLens-100K before curation and verified distractors for each query. Using eight clean-video rollouts from TimeLens-SFT, we obtained:

Verified distractor countQueriesDead-group rate
02208.2%
113013.8%
29022.2%
≥36031.7%
Total50015.0%

Only 30.0% of this random sample contains at least two verified distractors, whereas every HI-TVG-300 example meets this condition. Multiple annotations in a video are insufficient by themselves: distractors must be verified against the current query. We also verify temporal boundaries and apply teachability filtering to identify examples where the privileged view provides useful guidance.

Table 4 evaluates the complete curation pipeline while holding DTP-OPSD and the 300-example training budget fixed:

Training subsetQVHighlights-TimeLens mIoU
Random subset65.53
Human-verified subset66.05
HI-TVG-30066.71

The improvement supports the value of selecting verified, difficult, and teachable examples. This ablation measures the complete curation process. In the main method comparison, Pure GRPO and DTP-OPSD use exactly the same HI-TVG-300 examples.

Q6 (Original Question 2): Can additional visual examples make the improvements clearer?

Answer: Thank you for this helpful suggestion. We have expanded Appendix C, retaining the original Figure 6 and adding three examples in Figures 7–9. The added text and captions are marked in blue.

  • Figure 7: Similar placement actions. The query asks for placing a glass on the table rather than near the sink. The target lasts one second; Pure GRPO selects an earlier action, while DTP-OPSD achieves 0.83 tIoU.
  • Figure 8: Opposite actions. The video contains taking a camera out of a box and putting it back. DTP-OPSD selects the queried “putting back” event and achieves 0.83 tIoU.
  • Figure 9: Related actions involving the same object. The person picks up a cup, drinks, and puts it down. DTP-OPSD identifies the final placement event and achieves 0.87 tIoU.

Pure GRPO has zero target overlap in all three examples. Each figure shows the query, clean-video frames, ground truth, and both predictions. The accompanying Pure GRPO video description is generated with a separate description prompt.

Together with the repeated-event examples in the main paper, these additions illustrate occurrence correction, opposite-action discrimination, and localization of one- or two-second targets. They complement the quantitative correction analysis discussed with Reviewer Y1As in Q3 and summarized in Q2 above.

I appreciate the responses from the authors. For the experiments on the verified distractors, it seems that the improvement is not large. In additioin, when applied to stronger Qwen3.5 model, the performance gain also reduces compared to Qwen3.0.

For the results on very short target events, it seems that the improvement is not larger than those long events. Did the authors analyze why this happens?

Reviewer_Y1AsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses same-video distractor interference and the all-zero reward problem in GRPO-based Temporal Video Grounding (TVG). The authors propose DTP-OPSD, a privileged self-distillation framework in which a shared-weight teacher observes videos with highlighted target and distractor intervals. An adaptive controller adjusts the distillation weights based on teacher performance and dead-group status. Post-training on 300 curated hard examples from TimeLens-100K yields consistent improvements over Pure GRPO across multiple TVG benchmarks.

Critical strengths
  1. Intuitive and reasonable methodology The proposed method addresses the lack of learning signals in GRPO when all sampled responses receive zero rewards. The adaptive controller, which adjusts distillation based on teacher competence and dead-group status, is a reasonable design choice. The teacher and student share the same model parameters, and no additional inputs or computation are required during inference.

  2. Consistent empirical improvements with useful analyses The method consistently improves over Pure GRPO across three major TVG benchmarks, with stable results across three independent training runs. Figure 1 shows that the performance gains become larger as the number of distractors increases. Appendix Table 9 also reports an improvement in the recovery rate of initially dead groups from 38.7% to 64.1%. The ablation studies and compute-matched GRPO comparison further support the effectiveness of the proposed approach.

Critical weaknesses
  1. Insufficient evidence connecting distractor-related failures to the dead-group optimization bottleneck The paper argues that same-video distractors can lead to incorrect occurrence selection, causing all GRPO rollouts to receive zero rewards and provide no learning signal. However, HI-TVG-300 is deliberately curated to contain at least two distractors and examples that are challenging for the student. Therefore, the current analysis does not sufficiently establish how strongly the presence of distractors is associated with dead-group formation. Furthermore, the dead-group recovery results in Appendix Table 9 are measured on the same examples used for post-training. It remains unclear whether the proposed method generalizes to unseen distractor-related failures or mainly improves fitting to the curated training examples. In particular, a more direct analysis is needed to determine whether the final performance improvements actually arise from correcting wrong-occurrence predictions.

  2. Insufficient justification for the proposed method given its additional supervision and training cost The proposed approach requires GT-derived temporal cues and human-verified distractor annotations, along with additional teacher computation. Although the improvements are consistent, the mIoU gains over Pure GRPO are 0.50–1.15 points. On QVHighlights, training cost increases from 58.4 to 93.6 GPU-hours (approximately 60%), while the gain over compute-matched GRPO is 0.62 mIoU. The comparisons with GT-SFT and compute-matched GRPO partially address these concerns. However, the paper does not sufficiently analyze how dead-group recovery evolves with continued conventional GRPO training. Consequently, it remains unclear whether the proposed method primarily addresses failures that remain difficult for conventional GRPO or provides a more efficient way to learn such examples.

Questions for authors

Q1. Relationship between distractors and dead-group formation What proportion of the original TimeLens-100K training set contains same-video distractors? Could the authors compare dead-group frequency across examples with different numbers of distractors, including those without distractors? This would provide more direct evidence connecting distractor interference to dead-group formation.

Q2. Dead-group recovery with continued training Could the authors report the recovery rate of the same 142 initially dead groups in Appendix Table 9 for compute-matched GRPO? Additionally, does dead-group frequency or recovery continue to improve with further training, or does it eventually plateau? A comparison with hard-example sampling or curriculum learning would also be informative.

Q3. Actual improvement in distractor-related failures On a held-out evaluation set, could the authors measure how often examples initially predicted as incorrect distractor occurrences are corrected to the target occurrence after DTP-OPSD training? Comparing this with Pure GRPO and compute-matched GRPO would help distinguish wrong-occurrence correction from boundary refinement.

Q4. Confounding factors in distractor-count analysis Does the performance degradation in Figure 1 persist after controlling for video duration, absolute target duration, and relative target duration? Could the authors also report sample counts and performance improvements separately for examples with 0, 1, 2, and 3 or more distractors?

Q5. Interpretation of colored borders The teacher prompt does not explicitly explain the meanings of the red and blue borders. How does teacher performance change when the colors are swapped or their meanings are explicitly described in the prompt? This would help distinguish general visual highlighting effects from target/distractor-specific guidance.

3 comments

We thank the reviewer for the careful assessment and constructive suggestions. We address all five questions below. Q1, Q3, and Q4 address W1 through analyses of the broader training pool, wrong-occurrence correction, and duration-related effects. Q2 addresses W2 through compute-matched and longer-training comparisons.

Q1 (W1): How common are same-video distractors in TimeLens-100K, and how are they related to dead groups?

Answer: Thank you for suggesting an analysis beyond the curated training set. We randomly sampled 500 query–video pairs from TimeLens-100K before applying our curation filters and manually verified distractors for each query. Multiple annotations in the same video were not automatically counted as distractors.

Using TimeLens-SFT, the common initialization of both methods, we generated eight clean-video completions per query. A group was classified as dead when all eight completions had zero target tIoU.

Verified distractor countQueriesSample proportionDead groupsDead-group rate
022044.0%188.2%
113026.0%1813.8%
29018.0%2022.2%
≥36012.0%1931.7%
Total500100.0%7515.0%

In this sample, 56.0% of queries contain at least one verified distractor. The dead-group rate rises from 8.2% without distractors to 31.7% with at least three, supporting an association between interference and dead groups beyond HI-TVG-300.

The overall rate of 15.0% is lower than the 142/300 rate in HI-TVG-300, consistent with our selection of difficult training examples. Appendix Table 7 provides complementary evidence: within that curated set, 79.8% of the 1,136 initial dead-group rollouts overlap a verified distractor.

Q2 (W2): Do initially dead groups recover with compute-matched or longer Pure GRPO training?

Answer: Thank you for this important question. We tracked the same 142 initially dead training examples used in Appendix Table 9. An example is recovered when at least one of eight diagnostic completions has positive target tIoU.

MethodTraining budgetRecovery of the initial 142 groupsQVHighlights-TimeLens mIoU
Pure GRPO1 epoch; 58.4 GPU-hours38.7%65.95
Pure GRPO, compute matched93.6 GPU-hours48.6%66.09
Pure GRPO2 epochs52.8%65.88
Pure GRPO3 epochs54.9%65.42
DTP-OPSD1 epoch; 93.6 GPU-hours64.1%66.71

Longer Pure GRPO training increases recovery on the training examples, while evaluation mIoU decreases at two and three epochs. With only 300 training examples, this is consistent with stronger fitting to the training set without a corresponding improvement in generalization.

This does not contradict the dead-group mechanism: an all-zero group provides no reward-driven gradient in the current update, but updates from other groups can change its later predictions. DTP-OPSD adds token-level supervision when the privileged teacher performs better.

At approximately matched computation, DTP-OPSD achieves 15.5 percentage points higher recovery and 0.62 higher evaluation mIoU. Q3 and Q4 further examine whether these benefits extend to evaluation failures and high-interference examples. These comparisons hold the training data fixed; they do not evaluate adaptive sampling or curriculum learning.

Q3 (W1): Do the gains include correcting wrong-occurrence predictions on evaluation examples?

Answer: Thank you for recommending a direct evaluation of this failure mode. We selected a fixed set of 120 QVHighlights-TimeLens evaluation queries, excluded from post-training, for which the initial TimeLens-SFT prediction had zero target tIoU and overlapped a verified distractor. Selection used only the initialization’s predictions and annotations, independently of the post-trained methods’ results.

We then measured correction to the target under the standard single-interval evaluation protocol.

MethodCorrected to target tIoU ≥0.5Corrected to target tIoU ≥0.7
Pure GRPO37/120 (30.8%)24/120 (20.0%)
Compute-matched Pure GRPO40/120 (33.3%)27/120 (22.5%)
DTP-OPSD43/120 (35.8%)38/120 (31.7%)

At tIoU ≥0.5, 11 queries are corrected only by DTP-OPSD, while five are corrected only by Pure GRPO. At the stricter threshold of 0.7, DTP-OPSD corrects 38 queries, compared with 24 for Pure GRPO and 27 for compute-matched Pure GRPO.

These results show that the improvements include recovering the target occurrence from initially non-overlapping predictions, extending the training-set recovery analysis to evaluation examples. We will add this analysis, together with the sample selection protocol and correction results, to the revised manuscript.

Q4 (W1): Does the distractor-count trend persist after accounting for video and target duration?

Answer: Thank you for identifying these potential confounding factors. Video durations in QVHighlights-TimeLens are nearly uniform: 97.6% of query–video pairs have a duration of 150 seconds. We therefore examined the trend within target-duration groups.

We jointly grouped queries by target-to-video duration ratio and verified distractor count, then randomly sampled 50 queries from each of the 12 combinations, yielding 600 queries. Each cell below contains 50 queries and reports Pure GRPO → DTP-OPSD mIoU on the same examples.

Target-to-video duration ratioTotal queries0 distractors1 distractor2 distractors≥3 distractors
≤5%20063.80 → 63.9062.40 → 62.7060.60 → 61.2557.10 → 59.00
>5% and ≤20%20068.60 → 68.8067.20 → 67.4565.30 → 66.1562.00 → 64.30
>20%20075.00 → 75.1673.50 → 73.9071.70 → 72.3568.30 → 70.09

Within every target-duration group, both models’ mIoU decreases as distractor count increases. DTP-OPSD achieves its largest gains in the ≥3-distractor groups: 1.90, 2.30, and 1.79 points, compared with 0.10, 0.20, and 0.16 points without distractors.

Thus, the interference-related trend and larger gains under high interference remain observable within the target-duration groups, complementing Figure 1.

Q5: How does performance change when border colors are swapped or their roles are explicitly described?

Answer: Thank you for this useful suggestion. We tested four configurations that vary the target/distractor color assignment and whether the teacher prompt explains their roles. We evaluated the teacher using a fixed TimeLens-SFT checkpoint and separately measured the final clean-video student after training with each configuration.

Target / distractor colorsExplicit role descriptionFixed-teacher mIoUFinal student mIoU
Red / blueNo62.1066.71
Blue / redNo62.0666.68
Red / blueYes62.1866.65
Blue / redYes, with the swapped roles62.1566.67

Only small changes are observed across these configurations. Swapping colors changes final student mIoU from 66.71 to 66.68, and explicit role descriptions provide no improvement in these experiments.

The results suggest limited sensitivity to the particular red/blue assignment. Our interpretation is that the borders distinguish temporal regions for a teacher conditioned on the query and video, without requiring a predefined meaning for each color.

Temporal placement and distractor selection remain important: Table 3 reports 64.44 mIoU with an incorrectly placed target cue, 66.12 with an aligned target-only cue, and 66.71 with verified distractors, compared with 65.94 for random distractors.

Dear Reviewer Y1As,

Thank you again for your careful review and constructive feedback. As the discussion period is closing soon, we wanted to gently follow up on our rebuttal.

In Q3, we followed your suggestion to evaluate wrong-occurrence correction on 120 held-out QVHighlights-TimeLens queries where the initial predictions missed the target and overlapped a verified distractor. At a target tIoU threshold of 0.7, DTP-OPSD corrected 38 predictions, compared with 27 for compute-matched Pure GRPO.

If you have had a chance to review these results, we would appreciate your thoughts on whether they address your questions.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.