- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Temporal-Response Guidance (TRG) for intermediate-feature distillation from teacher to student models.
By introducing temporal perturbations to the student's video embeddings, TRG identifies temporal-response directions and uses them to refine the teacher's hidden-state correction.
Experiments with 4B and 9B students demonstrate improvements over on-policy distillation (OPD) and direct feature alignment (Plain-AS) across TempCompass, Video-MME, MVBench, and Video-MMMU.
Critical strengths
The paper proposes Temporal-Response Guidance (TRG), a novel approach to intermediate-feature distillation that leverages the student's temporal responses to guide teacher supervision.
The experimental design is solid, with comparisons against multiple baseline methods (OPD and Plain-AS) across four benchmarks and two student scales (4B and 9B), demonstrating the effectiveness of the proposed method.
Critical weaknesses
Clarity of presentation. The paper is difficult to follow, and the figures are overly abstract and do not effectively illustrate the proposed method. For instance, Figure 1 provides limited intuition about the key innovation of TRG. The writing and presentation would benefit from substantial improvement.
Limited novelty. The contribution appears somewhat incremental, as intermediate-feature distillation is a well-established research direction. The proposed method primarily modifies the supervision direction using student temporal responses.
Additional training overhead. TRG requires multiple Jacobian-vector product (JVP) evaluations and SVD to construct the response subspace for each eligible video, potentially introducing substantial computational overhead. However, the paper does not quantify the additional training cost, making it difficult to assess the practical trade-off between computational efficiency and performance gains.
Questions for authors
Training overhead. The paper does not report the additional computational cost introduced by TRG. Could the authors provide the training-time overhead or additional FLOPs compared with OPD and Plain-AS?
Perturbation location. Why are temporal perturbations applied to video embeddings rather than raw video frames? Perturbations in embedding space may not correspond to meaningful temporal transformations, such as frame reversal, changes in motion speed, or event reordering, which motivate the proposed method. Could the authors justify this design choice and clarify how the embedding-space perturbations capture these temporal semantics?