- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper focuses on knowledge distillation between heterogeneous architectures. The authors argue that existing logit-based distillation methods are insufficient because they constrain the student to match only a single endpoint, i.e., the observed teacher logits for a given sample. To address this limitation, they extend conventional logit matching to probability trajectory matching. They construct trajectories by applying a uniform diffusion process to the teacher predictive distributions, gradually transforming them toward uniform distributions. A time-conditioned predictor is trained to estimate the corresponding probability states from student features. Based on this, they introduce an additional trajectory loss to supervise the student's estimated reverse transitions toward lower-noise teacher states. While the method also incorporates multi-granularity supervision and weighting based on sample difficulty, I consider trajectory-based distillation to be the paper's central contribution.
Critical strengths
- The overall narrative of the paper is generally clear.
- The proposed diffusion-based formulation is technically elaborate, integrating probability trajectory modelling and reverse-transition supervision.
Critical weaknesses
- The major weakness, as per my assessment, is Insufficient motivation for the proposed complexity. The paper proposes a technically complex method, but the motivation for introducing this complexity is not clear. While the limitation of single-endpoint matching in knowledge distillation is understandable, it remains unclear why the proposed trajectory-based supervision would lead to better student learning. In short, why does transforming the probability distribution toward a uniform distribution help? Isn't this simply an externally imposed process? Does it convey meaningful, model-specific behaviour?
I suggest that the authors clarify the following questions:
-
Why should trajectory-based matching improve student learning? What specific benefits does it offer beyond conventional logit matching?
-
What teacher-specific knowledge is transferred through probability trajectory matching? Which properties of the teacher does this supervision encourage the student to learn?
-
Why is matching local transitions between adjacent trajectory states beneficial? What additional learning signal does it provide beyond matching individual probability states?
-
What does the proposed reverse process actually learn? How does it contribute to knowledge transfer rather than simply reconstructing softened teacher predictions?
I do not necessarily expect formal theoretical guarantees, but clear intuitive explanations addressing these questions would substantially strengthen the motivation and justification of the proposed method.
-
The necessity and effectiveness of reverse-transition modelling. Since the forward diffusion is equivalent to directly interpolating teacher probabilities with a uniform distribution, the necessity of diffusion is unclear. Does the reverse process capture meaningful teacher-specific information beyond simple probability matching? The authors should clarify this mechanism and justify its benefits over simpler alternatives.
-
Related to the above concern, while the ablation results demonstrate accuracy improvements from adding the state and trajectory losses (Eqs. 22–23) to the overall objective (Eq. 24), their distinct learning benefits remain unclear. What specific properties of the student are improved by each loss, and what is the motivation for introducing both beyond their empirical accuracy gains?
-
The figures could be more abstract and clear. The writing could also be more concise and precise. For example, phrases such as “static endpoint-matching paradigm” could be simplified for clarity. Similarly, terms such as “dynamic probability-trajectory learning” and “joint state-and-transition learning” need clearer explanations.
-
The paper's organization could be improved, as several important details are deferred to the supplementary material. One suggestion is to shorten the introduction to retain only essential information. Additionally, Figures 1 and 2 appear to convey overlapping information. However, I leave this to the authors' discretion.
Questions for authors
Please refer to the Weaknesses section for specific questions.
Additional questions for the authors:
-
What meaningful teacher-specific information is captured by the reverse transitions? Since the forward diffusion effectively interpolates teacher probabilities toward a uniform distribution, why is a diffusion-based formulation necessary instead of directly generating and matching these perturbed distributions?
-
Does the proposed categorical probability diffusion formulation introduce any methodological novelty compared to existing diffusion literature, or is its contribution primarily the application of established diffusion techniques to teacher–student knowledge distillation? Please clarify the specific novelty.
-
What are the standard deviations of the reported ablation results across the three runs?
-
Beyond classification accuracy, does trajectory supervision improve other student properties, such as calibration or robustness, to justify the method's added complexity?
5 comments
[2/4] Official Comment by Authors
To W1 (Point 4). Continuing our response to W1, we clarify what the reverse process learns and how it contributes to knowledge transfer.
(4) What the reverse process learns.
Compared with independently matching softened targets obtained by interpolating the teacher distribution with the uniform distribution, MG-EATD additionally constrains recovery between student predictions at different noise levels. The time-conditioned predictor estimates current-state class probabilities from student features, enabling the student to learn how the teacher's class preferences are expressed across noise levels.
The local reverse update then redistributes probability mass to determine the neighboring lower-noise distribution from the current prediction under the specified schedule, without an additional direct prediction at the neighboring time. The same current prediction therefore participates in two coupled objectives: matching the current teacher state and producing a matching lower-noise state after the reverse update. This derived distribution provides additional error feedback for the current prediction. Under ideal trajectory matching, both the noise-conditioned predictions and the local reverse-update results agree with the corresponding teacher states.
The state loss updates the time-conditioned predictor, whereas the trajectory loss backpropagates to student features without updating predictor parameters. Reverse rates are analytically constructed from the current probabilities; learning occurs in the student representations and conditional predictions that support both state matching and consistent reverse recovery. Its contribution to knowledge transfer therefore extends beyond generating softened targets: it jointly constrains student-generated predictions and their probability-redistribution outcomes using the teacher's class preferences.
To W2. Uniform forward diffusion indeed admits a closed-form interpolation between teacher probabilities and the uniform distribution. Our core contribution, however, lies in the coupled training constraints constructed from discrete diffusion. As explained in W1, Points (3)–(4), the neighboring lower-noise student state is obtained through the prescribed local reverse update of the current prediction, rather than an independent prediction at another time. During optimization, the trajectory loss backpropagates to student features without updating predictor parameters. The same current prediction therefore participates in two coupled constraints: matching the current teacher state and producing a matching neighboring teacher state after the reverse update, providing additional error feedback.
To further examine the trajectory-matching design, we compare three simpler controls: shared affine mixing of teacher and student probabilities without an additional predictor; the same predictor supervised only at the clean teacher endpoint; and the same predictor supervised only at one fixed noise level on the teacher trajectory. As shown below, all three controls fall below the full method's 72.48%.
| Control experiment | Acc.(%) |
|---|---|
| No additional predictor: shared affine mixing of teacher/student probabilities | 71.23 |
| Same predictor: teacher endpoint only () | 71.86 |
| Same predictor: one fixed teacher trajectory state () | 71.72 |
Furthermore, the gradient-matched experiment in W1, Point (3), shows that reverse updates outperform dual-time direct prediction by 0.38 percentage points (72.48% vs 72.10%). These results support both the benefit of trajectory supervision over the tested static targets and, more specifically, the added value of local reverse connections over independent dual-time prediction.
[3/4] Official Comment by Authors
To W3. The two losses are intended to impose complementary training constraints on the same student prediction, rather than transfer two independent sources of teacher knowledge. The state loss requires the student to match the teacher's class preferences at the current noise level and updates both the time-conditioned predictor and student features. The trajectory loss requires that prediction, after the prescribed reverse update, to match the neighboring lower-noise teacher state. It backpropagates to student features without updating predictor parameters.
The student therefore receives supervision on both the current state and the result of the reverse update. Although perfect matching between teacher and student distributions would satisfy both objectives in the ideal case, student predictions generally remain imperfect during training. The two losses therefore provide feedback on both current-state matching and consistency after the reverse update, while constraining learning through different parameter-update paths.
Empirically, the full method improves over State-only by 0.42%. Furthermore, the experiments reported in our response to W1, Point (3), show that reverse updates outperform dual-time direct prediction by 0.38%, with both supervision terms and gradient routing matched. These results support the additional training value of explicitly linking neighboring states, beyond simply adding another independent supervision term.
To W4. Thank you for the suggestions on visual presentation and terminology. We will simplify the figure labels and visual elements and revise the text for greater clarity and precision. For example, we will replace “static endpoint-matching paradigm” with “matching the final teacher prediction,” explain trajectory learning as matching teacher probabilities at different noise levels, and describe state-and-transition supervision as supervising the current prediction and the neighboring lower-noise prediction obtained through a reverse update. These changes will be included in the revised manuscript.
To W5. In the camera-ready version, we will shorten the introduction to retain only the central problem, motivation, and main contributions. We will also bring essential details from the appendix into the main text, including explanations of the entropy-adaptive schedule and the predictor's time conditioning, as well as key loss-ablation results.
To Q1. Although the teacher targets at the current and neighboring lower-noise states are both obtained by interpolation, the student's local reverse update is driven by its own current prediction, whose class probabilities and their ratios determine the redistribution of probability mass. By requiring the updated lower-noise distribution to match the corresponding teacher target, reverse supervision encourages the student to learn and recover the teacher's class preferences for the given sample and region. This introduces an additional training constraint for knowledge transfer, rather than an independent source of teacher knowledge.
Compared with independently predicting trajectory states at different times using a time-conditioned predictor, MG-EATD obtains the neighboring lower-noise state by applying the prescribed local reverse update to the current prediction. Matching this derived state to the corresponding teacher state imposes an additional constraint on the current prediction: it must match the current teacher state and, after the reverse update, match the neighboring lower-noise teacher state.
As shown in W1, Point (3), reverse updates achieve 72.48%, versus 72.10% for dual-time direct prediction under identical gradient routing. The same predictor supervised only at a fixed trajectory state achieves 71.72%. These results support the additional value of the neighboring-state reverse constraint over the tested simpler alternatives. Discrete diffusion provides a framework for consistently defining the probability states and their local reverse relationships and its practical benefit is evaluated through these controlled comparisons.
To Q4. To further evaluate MG-EATD under data scarcity, we evaluate MG-EATD on VGG-13 → VGG-8 using 20%, 40%, and 60% of the CIFAR-100, following the CID protocol. MG-EATD outperforms the compared methods at all three ratios, supporting its applicability when training data are limited. These results provide complementary evidence of its practical value under data scarcity.
| Training data | KD | CID | MG-EATD |
|---|---|---|---|
| 20% | 59.14 | 62.13 | 63.46 |
| 40% | 66.89 | 68.64 | 69.13 |
| 60% | 69.90 | 70.85 | 70.98 |
[1/4] Official Comment by Authors
We sincerely thank you for your thoughtful examination of the mechanisms underlying our method. Your comments have helped us recognize the need for a clearer and more thorough explanation of our design motivation and how the method works. Below, we address your questions point by point and provide further clarification.
To W1 (Points 1–3). Thank you for highlighting the need to clarify our motivation. We address the first three points below and discuss Point 4 in Part 2 of our response.
(1) Benefits beyond conventional logit matching. MG-EATD extends teacher–student matching beyond a single prediction endpoint. It supervises both the teacher's relative class preferences at different noise levels and the redistribution of probability mass as these preferences are recovered toward lower-noise states.
For example, given a dog image, the teacher may assign higher probabilities to cats and wolves than to cars and airplanes, alongside its dominant prediction for dogs. These probabilities reflect learned class relationships. Uniform forward diffusion changes the magnitude of these differences while preserving their ordering before complete uniformity. Through the time-conditioned predictor, the student learns the teacher's relative support for different classes across noise levels.
Local reverse transition rates are further determined by class-probability ratios. For a common source class, a more strongly supported destination class has a larger reverse transition rate, reflecting a greater propensity for incoming probability mass. In MG-EATD, these rates are constructed from the current student prediction. Matching the resulting lower-noise state to the corresponding teacher state indirectly constrains the student to recover the teacher's class preferences through this probability redistribution.
Conventional logit matching already contains these class relationships. The additional value of MG-EATD lies in jointly supervising their expression across noise levels and their consistent recovery through local reverse updates, providing more structured training constraints for transferring the teacher's class relationships.
(2) Teacher-specific knowledge transferred through the trajectory. The teacher-specific knowledge consists of the relative support and preferences that the teacher assigns to different classes for a given sample and region. Forward diffusion presents these preferences as probability distributions at different noise levels, which the student matches through the time-conditioned predictor.
Beyond state matching, the trajectory loss Ltraj does not update the predictor parameters. The time-conditioned predictor is updated solely by the current-state loss Lstate. The neighboring lower-noise state is obtained entirely by applying the prescribed local reverse update to the current prediction, without an additional direct prediction at the neighboring time. Matching this updated distribution to the corresponding lower-noise teacher state further constrains the student to recover the teacher's class preferences through probability-mass redistribution. Thus, supervision requires both matching the current-state probabilities and ensuring that the current prediction supports recovery toward a lower-noise distribution consistent with the teacher's preferences.
(3) Benefits of constraining local transitions between neighboring states. The student's neighboring lower-noise distribution is obtained by applying the prescribed local reverse update to its current prediction. The trajectory loss therefore provides additional supervision for that current prediction: it must match the teacher at the current noise level and, after the reverse update, match the neighboring lower-noise teacher distribution. This explicitly connects the two states, so learning the current prediction is constrained by both its current-state error and its error after the reverse update. During training, these two evaluations provide complementary error signals for matching the teacher trajectory.
To examine this effect, we conduct a 2×2 ablation on Swin-T → ResNet18 on ImageNet-1K. We compare dual-time direct prediction with local reverse updates and independently control whether the trajectory loss updates predictor parameters.
Under matched gradient routing, reverse updates outperform direct prediction by 0.08 and 0.38 percentage points, respectively. Gradient isolation improves reverse-update accuracy from 72.24% to 72.48%, but does not improve direct prediction. These results support the additional value of local reverse connections and their complementary role with gradient isolation.