acceptodds
Under review as a conference paper at ICLR 2027

Guide the Tokens, Rank the Moments: Adaptive On-Policy Distillation for Temporal Video Grounding

Abstract

Temporal video grounding requires identifying a queried event and recovering its temporal extent: an answer can contain the right action yet include irrelevant context or omit part of the event. On-policy distillation supplies dense teacher guidance during timestamp generation, but matching token distributions does not explicitly require the student’s preferences over complete answers to reflect their localization quality. Our central idea is to use teacher proposals both as answers to imitate and as alternatives whose relative quality can be evaluated through temporal overlap. We propose Geo-OPD, a geometry-aware on-policy distillation framework that combines pointwise imitation of a teacher answer with listwise supervision, aligning the student’s relative sequence scores with an annotation-IoU-derived preference distribution over teacher proposals. Adaptive token guidance complements these sequence objectives by balancing soft and sharp views of the same frozen teacher according to temporal discrepancy, while reliability filtering selects teacher supervision. Experiments on Charades-TimeLens, ActivityNet-TimeLens, and QVHighlights-TimeLens show improvements over the reported distillation baselines, with component ablations indicating that the additional benefit of listwise supervision varies across datasets. These results support combining guidance on how to generate temporal answers with explicit supervision on which candidate intervals better localize the queried event.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Temporal Video Grounding, On-Policy Distillation, Video-Language Models, Knowledge Distillation, Geometry-Aware Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TwvqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies teacher-guided post-training for generative temporal video grounding. Its motivation is that token-level distillation does not explicitly align the student’s preferences over complete answers with their temporal localization quality. Geo-OPD combines three objectives: adaptive token-level distillation using two temperature views of the same frozen teacher on student-generated prefixes, imitation of a designated primary teacher answer, and listwise preference learning based on the IoU between teacher-proposed intervals and annotations. It also uses teacher reliability filtering and an offline teacher–student temporal discrepancy score to select training samples. The authors report improvements on three TimeLens-corrected benchmarks and use component ablations and candidate-ranking diagnostics to examine the contributions of the proposed objectives.

Critical strengths

The overall approach is clear, and using interval localization quality to guide the student’s answer preferences is reasonable. The teacher’s primary answer may not be the most accurate, so imitating it alone can leave useful information in other candidates unused. Learning from these quality differences is a useful aspect of the method.

Critical weaknesses

None

Questions for authors
  1. Prior work, including TimeLens (2026), Uncertainties in Temporal Boundary (2022), and Embracing Uncertainty (2021), has discussed errors and subjectivity in temporal boundary annotations. In Geo-OPD, small changes to the annotated boundaries may alter the IoU ranking of teacher candidates. Have the authors examined how stable these rankings are under plausible boundary perturbations?
  2. The authors explain that pointwise imitation anchors the primary answer’s absolute score, while the listwise objective constrains relative preferences among candidates. This explains the role of the pointwise term, but not why the teacher’s primary answer is chosen as its target. Table 5 shows that the primary answer is often not the highest-IoU candidate in the held-out proposal sets. How would grounding performance change if the pointwise term instead used the highest-IoU teacher candidate against the annotation for each training example, while keeping the candidate sets, other losses, and training budget unchanged?
Reviewer_yPUFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper post-trains VLMs for temporal video grounding by distilling from a frozen teacher. The loss has three parts. A forward KL on student rollouts mixes two teacher temperatures with a weight set by an offline discrepancy score, an SFT term imitates the teacher’s primary answer, and a listwise KL matches the student’s sequence scores to a softmax over the IoU of several teacher proposals. Training samples are first filtered by teacher IoU against the annotation. Results are reported on the three TimeLens benchmarks.

Critical strengths

The gap between the primary and the best teacher proposal in Table 5 is a clear motivation for using more than one teacher answer. The ablation covers all eight combinations of the three terms, and the ranking test in Table 4 reports bootstrap intervals and states plainly that listwise does not help on Charades. The paper is easy to read.

Critical weaknesses

The major weaknesses are as follows.

  • In Table 2 each term alone already gives almost the full gain. A token KL, an SFT loss and a shift-invariant ranking loss should not all produce the same jump unless something shared by all rows does the work, and the paper never says what the token loss is when Adaptive is off. The listwise term, which is the main idea, adds little over Adaptive plus Pointwise and nothing on Charades.

  • Reliability filtering keeps only samples where the teacher is already close to the annotation, and it is not stated whether GT-SFT, OP-FKD and Video-OPD train on the same pool. In Table 3 the filter alone beats the full selector on two datasets, and random sampling beats it on ActivityNet. The ground-truth filter looks like a main driver, and the discrepancy stratification is not supported.

  • Ground truth is available during training and the best proposal is well above the primary one in Table 5. Imitating the best-IoU proposal pointwise is the obvious baseline for the listwise term and is not reported. The large gap to GT-SFT also needs an explanation, since GT-SFT sees the same annotations directly.

  • G_phi in Eq. 3 is not defined, and no value is given for the reliability threshold, the number of proposals, the temperatures or the loss weights. Training data, number of updates and whether one model is trained per benchmark are also missing. Eq. 9 applies the same temperature to IoU and to mean log probabilities, which live on a different scale.

  • The reproduced Video-OPD on ActivityNet gives the same [email protected] and [email protected] as the untrained base model, and on QVHighlights it is below OP-FKD. The main on-policy baseline looks under-tuned. Where the other post-training rows in Table 1 come from is not stated.

  • Figure 2 packs most of the equations of Section 3 into small boxes and repeats the text rather than showing how data flows between the three parts. Figure 1 puts a toy example, two loss sketches and the numbers from Table 5 in one panel under a one-line caption. In Figure 3 the intervals are in small print and the frames do not show where the event starts or ends. All three figures are hard to read, and the captions give no help.

Questions for authors
  • What exactly is the first row of Table 2, and what loss remains when Adaptive is off? A run with only a single-temperature token KL on the same filtered data would show how much of the gain comes from the three proposed terms at all.

  • Do GT-SFT, OP-FKD, OP-RKD and Video-OPD train on the same reliability-filtered pool with the same number of updates? If not, please report them on that pool.

  • Which model is the teacher, how large is it, and how are the proposals sampled? Please add its own scores on the three test sets to Table 1 and confirm whether the distillation baselines use the same teacher.

  • What happens if the pointwise target is the best-IoU proposal instead of the primary one? This is the direct test of whether the listwise term is needed.

  • Please give the definition of G_phi and all hyperparameter values, along with the training data, the number of updates and whether one model is trained per benchmark.

  • Why does the reproduced Video-OPD match the untrained base model on ActivityNet, and where do the other post-training rows in Table 1 come from?

Reviewer_qtgFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the gap between token-level teacher guidance and interval-level localization quality in video temporal grounding. Geo-OPD combines adaptive token guidance with pointwise answer imitation and IoU-guided listwise supervision. Experiments report improved grounding performance across three benchmarks, with component ablations and candidate-ranking analyses examining the effects of the supervision objectives.

Critical strengths
  1. The paper clearly identifies a limitation of applying on-policy distillation to video temporal grounding: token-level teacher guidance does not explicitly account for differences in localization quality among complete responses. The distinction between learning to generate answers and learning to prefer better-localized intervals is conceptually clear. Combining pointwise imitation with IoU-guided listwise supervision is a sensible way to address this gap.

  2. The experiments examine both quality differences among teacher candidates and the student’s candidate-ranking accuracy, beyond overall grounding performance. Comparisons on identical candidate sets show that listwise supervision improves score–quality alignments. These analyses provide candidate-level evidence for the effects of the proposed quality supervision.

Critical weaknesses
  1. Geo-OPD combines on-policy distillation with answer imitation and IoU-guided preference learning. Its methodological novelty appears incremental, and the experiments do not establish its advantage over existing learning-to-rank and preference optimization objectives under matched training conditions. The additional value of the proposed combination therefore remains insufficiently demonstrated.

  2. The choice of the primary answer as the imitation target is not sufficiently justified. Since the method already uses annotation IoU, imitating the highest-IoU candidate appears to be a natural alternative. Without evaluating this alternative with and without listwise supervision, it remains unclear whether the proposed combination provides meaningful grounding gains over simply selecting a better imitation target.

Questions for authors
  1. What additional benefit does Geo-OPD provide over existing learning-to-rank and preference optimization objectives under the same OPD setting? A controlled comparison would clarify the contribution of the proposed combination.

  2. How does grounding performance change when the student imitates the highest-IoU candidate instead of the primary answer?

  3. Does listwise supervision still provide additional gains when the student imitates the highest-IoU candidate?

  4. What explicit formulation of (Gϕ)(\mathcal{G}_{\phi}) in Eq. (3) was used in the experiments? Please specify how the three components are computed and combined, along with the configuration (ϕ)(\phi).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for your careful reviews and constructive discussion. The reviewers highlighted the gap between token-level distillation and the localization quality of complete answers, the motivation for learning from multiple teacher proposals, and the evidence from component ablations and fixed-candidate ranking analyses. Geo-OPD addresses this gap by combining adaptive token guidance on student-generated prefixes with answer imitation and IoU-guided supervision over complete teacher proposals.

Key findings from the rebuttal

  • Learning from multiple proposals adds value beyond selecting the best imitation target. On QVHighlights-TimeLens, adding listwise supervision improved student-generated mIoU from 59.51 to 60.54 even when the student already imitated the highest-IoU proposal. With the same target, the effect was small and mixed on Charades and slightly negative on ActivityNet. Under matched adaptive guidance and primary-answer imitation, the IoU-listwise objective achieved the highest mIoU among the tested ranking and preference losses on all three benchmarks, with modest margins of 0.05–0.17 points over SimPO.
  • The training objective contributes beyond data selection. With the same selected training pool and update budget, Geo-OPD outperformed single-temperature token KL on all nine grounding metrics. This held across all five tested data selectors. The matched comparison with Video-OPD showed a more nuanced result: Geo-OPD improved faster early in training, while the later comparison depended on the benchmark and metric.

Discussion outcomes

  • qtgF raised their rating to Weak Accept. The reviewer said the additional experiments and clarifications addressed most concerns and pointed to the highest-IoU comparison as evidence for the added value of listwise supervision. They continued to view the methodological novelty as incremental.
  • yPUF raised their rating to Weak Accept. The reviewer said the controlled comparisons addressed their main concerns about baseline fairness, the contribution beyond data selection, and the choice of imitation target.
  • Twvq maintained Weak Accept. The reviewer said the new experiment addressed their question about the imitation target and clarified the dataset-dependent gains. Candidate-ranking stability under plausible annotation-boundary perturbations remains untested, and we will state this limitation explicitly.

We appreciate the reviewers' suggestions and ask the Area Chair to consider the controlled comparisons and these discussion outcomes when assessing the paper.

Sincerely, The authors of Paper #23628

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.