- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies a previously unexplored question in knowledge distillation: whether all points should be weighted equally when supervising the discrepancy between the student and teacher models. It argues that weights should instead be assigned according to how much the current prediction error will affect the future generation process. Experiments on datasets including ImageNet and COCO demonstrate the effectiveness of the proposed method compared with other baselines.
Critical strengths
- The question studied in this paper is relatively novel and worth investigating: when supervising the discrepancy between the student and teacher models, greater weight should be assigned to points that will have a larger future impact. The proposed FD loss provides an intuitive answer to this question.
- The paper provides a mathematical interpretation of the FD loss and theoretically discusses the essence of the method: it is equivalent to replacing the Euclidean distance used in standard KD with a local quadratic metric determined by the teacher dynamics.
Critical weaknesses
- Computing the future rollout of the teacher model requires gradient computation, which results in a very large computational graph and substantially slows down training. Moreover, because the teacher is itself a relatively large model, multi-step sampling is already slow. This may limit the scalability of the method.
Questions for authors
See “Critical Weaknesses.”
3 comments
Thank you for the response. Most of my concerns have been addressed, so I will maintain my positive score.
Thank You for Your Feedback
Thank you for your follow-up and positive assessment. We appreciate your careful review and constructive feedback, which helped us clarify the method and strengthen the manuscript.
Author Response
Dear Reviewer 54UZ,
We thank the reviewer for recognizing the value of evaluating current errors through their future effects and the interpretation of FD as a local quadratic metric induced by Teacher dynamics.
Teacher continuation adds training computation. We address it by clarifying its gradient scope, then examining training cost and quality gains across Teacher sizes.
1. Gradients pass through the current Student update and short Teacher continuation. The preceding sampling history is excluded from backpropagation, and the Teacher reference branch provides a fixed target. On the Student branch, the frozen Teacher retains input-state derivatives to transmit the future discrepancy back to the current Student update. Thus, each query spans one Student update and h Teacher updates. Query gradients are computed separately and accumulated before one Student parameter update. With the default h=1, R=3, this requires 1-step continuation at three separately processed positions.
2. Temporal sampling reduces training cost while retaining FD gains. Alongside the short horizon, temporal sampling (TS) reduces supervised positions and also applies to TPD and OPD. Table 15 separates these effects for SiT-B to SiT-B-D6 at the same update budget, using h=1 for FD and stratified sampling with R=3 for TS. Checkmarks denote enabled components; Cost measures a complete iteration in s/iter. Parentheses show changes from each group’s dense local baseline, and bold marks group-wise best results.
At matched sampling settings, FD improves FID and IS over both local baselines, with similar gains under dense supervision and TS. Meanwhile, TS lowers Cost for all four methods with modest quality changes. Combining FD and TS therefore preserves these gains and outperforms the dense local baselines in FID, IS, and per-iteration Cost at the same update budget.
3. Quality gains extend across Teacher sizes. To examine scalability, Figure 5 and Table 16 fix the SiT-S Student and vary the Teacher from SiT-B to SiT-XL. All configurations use 25K updates, the same global rollout batch, and TS with R=3; FD uses h=1 and shared Teacher continuation. Parentheses and bold indicate changes and best results within each local/FD pair. Cost is reported in s/iter for two RTX 4090 GPUs.
FD improves FID and IS at every tested Teacher size, with the largest FID reductions over local baselines for SiT-XL: 1.87 for FD(T) and 4.37 for FD(S). Their measured per-GPU Mem is 17.25 and 17.05 GB, respectively. These results demonstrate training feasibility and quantify the quality–Cost trade-off as Teacher size increases.
4. The additional computation of FD is confined to training. FD-KD’s additional supervision and Teacher continuation leave inference unchanged: the compact Student retains its original architecture and sampling procedure, with no additional parameters or network evaluations.
Revision in paper.Sections 3.5 and 4.2.5; Appendices B.2 and B.4; Appendices D.4 and D.5, Tables 15 and 16 (gradient scope, inference, and quality–Cost comparisons).