acceptodds
Withdrawn from ICLR 2027

Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) transfers capabilities from large models to smaller ones through teacher supervision on student-generated trajectories. GKD and MiniLLM develop this paradigm for language models, while Video-OPD extends it to video tasks. Hidden-state distillation moves beyond output distribution matching, but direct alignment with teacher representations does not explicitly use the student's temporal responses to guide supervision. We introduce Temporal-Response Guidance (TRG), which uses the student's responses to local temporal changes to adjust the direction of teacher supervision. TRG adds the projection of the teacher correction onto the student's temporal response subspace back to the original correction without separately rescaling the projection. This preserves complementary teacher information outside the response subspace. Under an orthogonal projection and a detached target, TRG changes the supervision direction while preserving the cosine-loss value and the local gradient norm with respect to the projected student representation. Across four video understanding benchmarks—TempCompass, Video-MME, MVBench, and Video-MMMU—TRG improves average accuracy by 2.86 percentage points over the reported output-only OPD baseline.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
On-policy distillation, Video understanding, Knowledge distillation, Hidden-state distillation, Temporal modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NXJ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Temporal-Response Guidance (TRG) for intermediate-feature distillation from teacher to student models.

By introducing temporal perturbations to the student's video embeddings, TRG identifies temporal-response directions and uses them to refine the teacher's hidden-state correction.

Experiments with 4B and 9B students demonstrate improvements over on-policy distillation (OPD) and direct feature alignment (Plain-AS) across TempCompass, Video-MME, MVBench, and Video-MMMU.

Critical strengths

The paper proposes Temporal-Response Guidance (TRG), a novel approach to intermediate-feature distillation that leverages the student's temporal responses to guide teacher supervision.

The experimental design is solid, with comparisons against multiple baseline methods (OPD and Plain-AS) across four benchmarks and two student scales (4B and 9B), demonstrating the effectiveness of the proposed method.

Critical weaknesses

Clarity of presentation. The paper is difficult to follow, and the figures are overly abstract and do not effectively illustrate the proposed method. For instance, Figure 1 provides limited intuition about the key innovation of TRG. The writing and presentation would benefit from substantial improvement.

Limited novelty. The contribution appears somewhat incremental, as intermediate-feature distillation is a well-established research direction. The proposed method primarily modifies the supervision direction using student temporal responses.

Additional training overhead. TRG requires multiple Jacobian-vector product (JVP) evaluations and SVD to construct the response subspace for each eligible video, potentially introducing substantial computational overhead. However, the paper does not quantify the additional training cost, making it difficult to assess the practical trade-off between computational efficiency and performance gains.

Questions for authors

Training overhead. The paper does not report the additional computational cost introduced by TRG. Could the authors provide the training-time overhead or additional FLOPs compared with OPD and Plain-AS?

Perturbation location. Why are temporal perturbations applied to video embeddings rather than raw video frames? Perturbations in embedding space may not correspond to meaningful temporal transformations, such as frame reversal, changes in motion speed, or event reordering, which motivate the proposed method. Could the authors justify this design choice and clarify how the embedding-space perturbations capture these temporal semantics?

Reviewer_hAPrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Temporal-Response Guidance (TRG) for on-policy distillation of video-language models. TRG estimates the student’s sensitivity to structured temporal perturbations using Jacobian-vector products and uses the resulting response subspace to redirect the teacher’s latent correction, while preserving the local loss and gradient norm. The method is evaluated with 4B and 9B students on four video-understanding benchmarks. TRG consistently improves over output-only OPD and provides an additional 0.87/0.79 point gain over direct latent alignment, with further controls suggesting that the benefit mainly comes from response-guided supervision directions and is stronger on temporal reasoning tasks.

Critical strengths

Strength 1: Clear motivation and reasonable method design

The paper identifies a meaningful limitation of direct latent alignment and proposes using student temporal responses to guide teacher supervision. The method is conceptually coherent and provides a reasonable approach to incorporating temporal information into knowledge distillation.

Strength 2: Relevant control experiments and theoretical analysis

The paper includes meaningful comparisons with direct latent alignment and random correction directions, helping isolate the contribution of response-guided supervision. The local geometric analysis further clarifies how TRG modifies the correction direction while preserving the representation-gradient norm.

Critical weaknesses

Weakness 1: Limited incremental gain relative to added training complexity

Most of the improvement over output-only OPD is already obtained by direct latent alignment. TRG itself improves over Plain-AS by only 0.87 points at 4B and 0.79 points at 9B. At the same time, TRG introduces additional temporal probes, JVP computations, SVD, and projection operations, but the paper does not report wall-clock time, GPU-hours, memory overhead, or an equal-compute comparison. This makes it difficult to assess whether the modest TRG-specific gain justifies the additional training cost.

Weakness 2: The necessity of the specific student-response guidance is not fully established

The random-orientation control shows that response-guided directions are better than arbitrary directions, but it does not establish that student temporal responses are the most appropriate source of guidance. Comparisons with alternatives such as teacher temporal responses, teacher–student Jacobian matching, or other structured response directions are missing. Therefore, while the proposed mechanism is plausible, the current evidence does not fully establish that the specific TRG construction is essential.

Questions for authors

Q1. Does TRG remain beneficial under an equal-compute comparison?

TRG improves over Plain-AS by only 0.87/0.79 percentage points at 4B/9B while introducing additional JVP and subspace computations. Could the authors report the actual training-time and memory overhead, and clarify whether these gains persist when Plain-AS is given an equivalent computational budget?

Q2. Why are student temporal responses the appropriate guidance signal?

The random-orientation control shows that TRG outperforms arbitrary redirection, but does not establish the necessity of using student temporal responses. Could the authors explain or provide evidence why this guidance is preferable to alternatives such as teacher temporal responses or teacher–student Jacobian matching?

Q3. How robust are the reported improvements across different training conditions?

The repeated runs share the same initialization, training subset, and sample order, varying only rollout randomness. Could the authors clarify whether the improvements remain consistent across different training seeds or data orders, particularly given the relatively modest gains over Plain-AS?

Reviewer_mqNvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRG — a method that redirects the teacher's state correction using the subspace of the student's first-order responses to structured temporal probes under a fixed language context. Via a residual rule that reweights the response-space component 2:1 and restores the original tangent norm, TRG gently steers the teacher correction toward the measured response directions, adding an explicit representation target to OPD distillation without changing the inference model. At two student scales, TRG consistently outperforms output-only distillation and direct alignment on four video-understanding benchmarks, with the clearest gains on event-order questions.

Critical strengths

The paper introduces and operationalizes the measurement concept of "the student's first-order response subspace to controlled local temporal changes" (structured DCT temporal probes + fixed language context + JVP + SVD) as information for constructing video-distillation supervision. This concept is then introduced into the training objective safely and verifiably, supported by a bounded, norm-preserving residual redirection rule and accompanying geometric analysis.

Critical weaknesses
  • The incremental benefit of the core novelty (response-guided redirection) is small, and the paper itself acknowledges that its robustness has not been established. Over direct alignment, the macro-average gain is only +0.87 pp (4B) / +0.79 pp (9B); moreover, most of the benchmark-level gains come from direct alignment itself, so TRG's own contribution accounts for only a small share.
  • Key design choices lack justification. The 2:1 reweighting ratio — the signature setting of the method — has no weight-ratio sweep or ablation.
  • The paper's central mechanistic narrative ("the order-category gains stem from the temporal responses") rests on a single benchmark with one evaluation per method, and event order remains the lowest-scoring category after TRG (71.19%), so the absolute shortfall is not resolved.
  • Capabilities beyond the four benchmarks (spatial/appearance-related skills, pure language ability) are not evaluated, so the possibility that response-guided redirection of temporal changes crowds out other capabilities cannot be ruled out.
Questions for authors
  • What is the justification for the 2:1 reweighting? Has a weight-ratio ablation been conducted?
  • Is the gain of response guidance over direct alignment robust across multiple runs?
  • If the extra computation from JVP/SVD is converted into additional OPD update steps or longer training, can Plain-AS/OPD catch up with TRG?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.