acceptodds
Under review as a conference paper at ICLR 2027

MG-EATD: Multi-Granularity Entropy-Adaptive Trajectory Distillation

Abstract

Most existing KD methods are primarily designed for homogeneous teacher–student architectures (e.g., CNNCNN), leaving the potential and flexibility of knowledge transfer across heterogeneous architectures underexplored. Our work focuses on knowledge distillation across heterogeneous architectures. We observe that most existing logit-based methods follow a static endpoint-matching paradigm: they treat the teacher logits as a single deterministic target and constrain the student only to approximate the teacher's final class distribution. In contrast, we reformulate knowledge distillation as dynamic probability-trajectory learning, enabling the student to learn the teacher's probability states across multiple uncertainty levels as well as the local transitions between adjacent trajectory states. Motivated by this dynamic trajectory perspective, we propose Multi-Granularity Entropy-Adaptive Trajectory Distillation (***MG-EATD***), which reformulates final endpoint matching as joint state-and-transition learning along class-probability trajectories. Specifically, *MG-EATD* decomposes teacher and student features into multi-granularity regions and maps them into a unified class-probability space, constructing hierarchical knowledge from global semantics to local discriminative cues. Building on this, we introduce a class-probability trajectory diffusion mechanism that constructs teacher probability states at varying uncertainty levels through discrete forward perturbations. A time-conditioned trajectory predictor estimates the current student probability state, which is then propagated to an adjacent lower-noise state through a local reverse update. To leverage the teacher–student prediction priors naturally available in KD, we estimate sample difficulty using their entropies and jointly adapt both the diffusion-time range and the reverse evolution step size. Consequently, the student learns the teacher's class knowledge across spatial granularities and uncertainty levels, as well as local reverse transitions between adjacent trajectory states. Extensive experiments on multiple benchmark datasets under both homogeneous and heterogeneous models demonstrate the superiority of *MG-EATD*.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Knowledge Distillation, Heterogeneous Distillation, Discrete Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XxzVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on knowledge distillation between heterogeneous architectures. The authors argue that existing logit-based distillation methods are insufficient because they constrain the student to match only a single endpoint, i.e., the observed teacher logits for a given sample. To address this limitation, they extend conventional logit matching to probability trajectory matching. They construct trajectories by applying a uniform diffusion process to the teacher predictive distributions, gradually transforming them toward uniform distributions. A time-conditioned predictor is trained to estimate the corresponding probability states from student features. Based on this, they introduce an additional trajectory loss to supervise the student's estimated reverse transitions toward lower-noise teacher states. While the method also incorporates multi-granularity supervision and weighting based on sample difficulty, I consider trajectory-based distillation to be the paper's central contribution.

Critical strengths
  1. The overall narrative of the paper is generally clear.
  2. The proposed diffusion-based formulation is technically elaborate, integrating probability trajectory modelling and reverse-transition supervision.
Critical weaknesses
  1. The major weakness, as per my assessment, is Insufficient motivation for the proposed complexity. The paper proposes a technically complex method, but the motivation for introducing this complexity is not clear. While the limitation of single-endpoint matching in knowledge distillation is understandable, it remains unclear why the proposed trajectory-based supervision would lead to better student learning. In short, why does transforming the probability distribution toward a uniform distribution help? Isn't this simply an externally imposed process? Does it convey meaningful, model-specific behaviour?

I suggest that the authors clarify the following questions:

  1. Why should trajectory-based matching improve student learning? What specific benefits does it offer beyond conventional logit matching?

  2. What teacher-specific knowledge is transferred through probability trajectory matching? Which properties of the teacher does this supervision encourage the student to learn?

  3. Why is matching local transitions between adjacent trajectory states beneficial? What additional learning signal does it provide beyond matching individual probability states?

  4. What does the proposed reverse process actually learn? How does it contribute to knowledge transfer rather than simply reconstructing softened teacher predictions?

I do not necessarily expect formal theoretical guarantees, but clear intuitive explanations addressing these questions would substantially strengthen the motivation and justification of the proposed method.

  1. The necessity and effectiveness of reverse-transition modelling. Since the forward diffusion is equivalent to directly interpolating teacher probabilities with a uniform distribution, the necessity of diffusion is unclear. Does the reverse process capture meaningful teacher-specific information beyond simple probability matching? The authors should clarify this mechanism and justify its benefits over simpler alternatives.

  2. Related to the above concern, while the ablation results demonstrate accuracy improvements from adding the state and trajectory losses (Eqs. 22–23) to the overall objective (Eq. 24), their distinct learning benefits remain unclear. What specific properties of the student are improved by each loss, and what is the motivation for introducing both beyond their empirical accuracy gains?

  3. The figures could be more abstract and clear. The writing could also be more concise and precise. For example, phrases such as “static endpoint-matching paradigm” could be simplified for clarity. Similarly, terms such as “dynamic probability-trajectory learning” and “joint state-and-transition learning” need clearer explanations.

  4. The paper's organization could be improved, as several important details are deferred to the supplementary material. One suggestion is to shorten the introduction to retain only essential information. Additionally, Figures 1 and 2 appear to convey overlapping information. However, I leave this to the authors' discretion.

Questions for authors

Please refer to the Weaknesses section for specific questions.

Additional questions for the authors:

  1. What meaningful teacher-specific information is captured by the reverse transitions? Since the forward diffusion effectively interpolates teacher probabilities toward a uniform distribution, why is a diffusion-based formulation necessary instead of directly generating and matching these perturbed distributions?

  2. Does the proposed categorical probability diffusion formulation introduce any methodological novelty compared to existing diffusion literature, or is its contribution primarily the application of established diffusion techniques to teacher–student knowledge distillation? Please clarify the specific novelty.

  3. What are the standard deviations of the reported ablation results across the three runs?

  4. Beyond classification accuracy, does trajectory supervision improve other student properties, such as calibration or robustness, to justify the method's added complexity?

5 comments
AuthorsOpenReview ↗

[1/4] Official Comment by Authors

We sincerely thank you for your thoughtful examination of the mechanisms underlying our method. Your comments have helped us recognize the need for a clearer and more thorough explanation of our design motivation and how the method works. Below, we address your questions point by point and provide further clarification.

To W1 (Points 1–3). Thank you for highlighting the need to clarify our motivation. We address the first three points below and discuss Point 4 in Part 2 of our response.

(1) Benefits beyond conventional logit matching. MG-EATD extends teacher–student matching beyond a single prediction endpoint. It supervises both the teacher's relative class preferences at different noise levels and the redistribution of probability mass as these preferences are recovered toward lower-noise states.

For example, given a dog image, the teacher may assign higher probabilities to cats and wolves than to cars and airplanes, alongside its dominant prediction for dogs. These probabilities reflect learned class relationships. Uniform forward diffusion changes the magnitude of these differences while preserving their ordering before complete uniformity. Through the time-conditioned predictor, the student learns the teacher's relative support for different classes across noise levels.

Local reverse transition rates are further determined by class-probability ratios. For a common source class, a more strongly supported destination class has a larger reverse transition rate, reflecting a greater propensity for incoming probability mass. In MG-EATD, these rates are constructed from the current student prediction. Matching the resulting lower-noise state to the corresponding teacher state indirectly constrains the student to recover the teacher's class preferences through this probability redistribution.

Conventional logit matching already contains these class relationships. The additional value of MG-EATD lies in jointly supervising their expression across noise levels and their consistent recovery through local reverse updates, providing more structured training constraints for transferring the teacher's class relationships.

(2) Teacher-specific knowledge transferred through the trajectory. The teacher-specific knowledge consists of the relative support and preferences that the teacher assigns to different classes for a given sample and region. Forward diffusion presents these preferences as probability distributions at different noise levels, which the student matches through the time-conditioned predictor.

Beyond state matching, the trajectory loss Ltraj\mathcal L_{\mathrm{traj}} does not update the predictor parameters. The time-conditioned predictor is updated solely by the current-state loss Lstate\mathcal L_{\mathrm{state}}. The neighboring lower-noise state is obtained entirely by applying the prescribed local reverse update to the current prediction, without an additional direct prediction at the neighboring time. Matching this updated distribution to the corresponding lower-noise teacher state further constrains the student to recover the teacher's class preferences through probability-mass redistribution. Thus, supervision requires both matching the current-state probabilities and ensuring that the current prediction supports recovery toward a lower-noise distribution consistent with the teacher's preferences.

(3) Benefits of constraining local transitions between neighboring states. The student's neighboring lower-noise distribution is obtained by applying the prescribed local reverse update to its current prediction. The trajectory loss therefore provides additional supervision for that current prediction: it must match the teacher at the current noise level and, after the reverse update, match the neighboring lower-noise teacher distribution. This explicitly connects the two states, so learning the current prediction is constrained by both its current-state error and its error after the reverse update. During training, these two evaluations provide complementary error signals for matching the teacher trajectory.

To examine this effect, we conduct a 2×22\times2 ablation on Swin-T → ResNet18 on ImageNet-1K. We compare dual-time direct prediction with local reverse updates and independently control whether the trajectory loss updates predictor parameters.

Prediction methodTrajectory loss updates predictorAcc.(%)
Dual-time direct prediction✓72.16
Dual-time direct prediction✗72.10
Reverse update✓72.24
Reverse update (our method)✗72.48

Under matched gradient routing, reverse updates outperform direct prediction by 0.08 and 0.38 percentage points, respectively. Gradient isolation improves reverse-update accuracy from 72.24% to 72.48%, but does not improve direct prediction. These results support the additional value of local reverse connections and their complementary role with gradient isolation.

AuthorsOpenReview ↗

[2/4] Official Comment by Authors

To W1 (Point 4). Continuing our response to W1, we clarify what the reverse process learns and how it contributes to knowledge transfer.

(4) What the reverse process learns.

Compared with independently matching softened targets obtained by interpolating the teacher distribution with the uniform distribution, MG-EATD additionally constrains recovery between student predictions at different noise levels. The time-conditioned predictor estimates current-state class probabilities from student features, enabling the student to learn how the teacher's class preferences are expressed across noise levels.

The local reverse update then redistributes probability mass to determine the neighboring lower-noise distribution from the current prediction under the specified schedule, without an additional direct prediction at the neighboring time. The same current prediction therefore participates in two coupled objectives: matching the current teacher state and producing a matching lower-noise state after the reverse update. This derived distribution provides additional error feedback for the current prediction. Under ideal trajectory matching, both the noise-conditioned predictions and the local reverse-update results agree with the corresponding teacher states.

The state loss updates the time-conditioned predictor, whereas the trajectory loss backpropagates to student features without updating predictor parameters. Reverse rates are analytically constructed from the current probabilities; learning occurs in the student representations and conditional predictions that support both state matching and consistent reverse recovery. Its contribution to knowledge transfer therefore extends beyond generating softened targets: it jointly constrains student-generated predictions and their probability-redistribution outcomes using the teacher's class preferences.

To W2. Uniform forward diffusion indeed admits a closed-form interpolation between teacher probabilities and the uniform distribution. Our core contribution, however, lies in the coupled training constraints constructed from discrete diffusion. As explained in W1, Points (3)–(4), the neighboring lower-noise student state is obtained through the prescribed local reverse update of the current prediction, rather than an independent prediction at another time. During optimization, the trajectory loss backpropagates to student features without updating predictor parameters. The same current prediction therefore participates in two coupled constraints: matching the current teacher state and producing a matching neighboring teacher state after the reverse update, providing additional error feedback.

To further examine the trajectory-matching design, we compare three simpler controls: shared affine mixing of teacher and student probabilities without an additional predictor; the same predictor supervised only at the clean teacher endpoint; and the same predictor supervised only at one fixed noise level on the teacher trajectory. As shown below, all three controls fall below the full method's 72.48%.

Control experimentAcc.(%)
No additional predictor: shared affine mixing of teacher/student probabilities71.23
Same predictor: teacher endpoint only (t=0t=0)71.86
Same predictor: one fixed teacher trajectory state (t=t0t=t_0)71.72

Furthermore, the gradient-matched experiment in W1, Point (3), shows that reverse updates outperform dual-time direct prediction by 0.38 percentage points (72.48% vs 72.10%). These results support both the benefit of trajectory supervision over the tested static targets and, more specifically, the added value of local reverse connections over independent dual-time prediction.

AuthorsOpenReview ↗

[3/4] Official Comment by Authors

To W3. The two losses are intended to impose complementary training constraints on the same student prediction, rather than transfer two independent sources of teacher knowledge. The state loss requires the student to match the teacher's class preferences at the current noise level and updates both the time-conditioned predictor and student features. The trajectory loss requires that prediction, after the prescribed reverse update, to match the neighboring lower-noise teacher state. It backpropagates to student features without updating predictor parameters.

The student therefore receives supervision on both the current state and the result of the reverse update. Although perfect matching between teacher and student distributions would satisfy both objectives in the ideal case, student predictions generally remain imperfect during training. The two losses therefore provide feedback on both current-state matching and consistency after the reverse update, while constraining learning through different parameter-update paths.

Empirically, the full method improves over State-only by 0.42%. Furthermore, the experiments reported in our response to W1, Point (3), show that reverse updates outperform dual-time direct prediction by 0.38%, with both supervision terms and gradient routing matched. These results support the additional training value of explicitly linking neighboring states, beyond simply adding another independent supervision term.

To W4. Thank you for the suggestions on visual presentation and terminology. We will simplify the figure labels and visual elements and revise the text for greater clarity and precision. For example, we will replace “static endpoint-matching paradigm” with “matching the final teacher prediction,” explain trajectory learning as matching teacher probabilities at different noise levels, and describe state-and-transition supervision as supervising the current prediction and the neighboring lower-noise prediction obtained through a reverse update. These changes will be included in the revised manuscript.

To W5. In the camera-ready version, we will shorten the introduction to retain only the central problem, motivation, and main contributions. We will also bring essential details from the appendix into the main text, including explanations of the entropy-adaptive schedule and the predictor's time conditioning, as well as key loss-ablation results.

To Q1. Although the teacher targets at the current and neighboring lower-noise states are both obtained by interpolation, the student's local reverse update is driven by its own current prediction, whose class probabilities and their ratios determine the redistribution of probability mass. By requiring the updated lower-noise distribution to match the corresponding teacher target, reverse supervision encourages the student to learn and recover the teacher's class preferences for the given sample and region. This introduces an additional training constraint for knowledge transfer, rather than an independent source of teacher knowledge.

Compared with independently predicting trajectory states at different times using a time-conditioned predictor, MG-EATD obtains the neighboring lower-noise state by applying the prescribed local reverse update to the current prediction. Matching this derived state to the corresponding teacher state imposes an additional constraint on the current prediction: it must match the current teacher state and, after the reverse update, match the neighboring lower-noise teacher state.

As shown in W1, Point (3), reverse updates achieve 72.48%, versus 72.10% for dual-time direct prediction under identical gradient routing. The same predictor supervised only at a fixed trajectory state achieves 71.72%. These results support the additional value of the neighboring-state reverse constraint over the tested simpler alternatives. Discrete diffusion provides a framework for consistently defining the probability states and their local reverse relationships and its practical benefit is evaluated through these controlled comparisons.

To Q4. To further evaluate MG-EATD under data scarcity, we evaluate MG-EATD on VGG-13 → VGG-8 using 20%, 40%, and 60% of the CIFAR-100, following the CID protocol. MG-EATD outperforms the compared methods at all three ratios, supporting its applicability when training data are limited. These results provide complementary evidence of its practical value under data scarcity.

Training dataKDCIDMG-EATD
20%59.1462.1363.46
40%66.8968.6469.13
60%69.9070.8570.98
Reviewer_H8mXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation across heterogeneous visual architectures and proposes Multi-Granularity Entropy-Adaptive Trajectory Distillation (MG-EATD). The authors argue that matching only the teacher’s final predictions may be limited by teacher–student capacity gaps and optimization difficulties. They therefore augment conventional distillation losses with supervision at different noise levels and supervision of lower-noise states obtained through local reverse updates. The method comprises three components: partitioning features into regions at multiple spatial granularities; constructing teacher probability trajectories through uniform discrete diffusion; and using teacher and student prediction entropies to adapt the perturbation range, reverse step size, and supervision weights. Experiments cover various homogeneous and heterogeneous teacher–student pairs on CIFAR-100 and ImageNet. Compared with FBT, MG-EATD improves average accuracy in heterogeneous distillation by approximately 3.12 and 0.11 percentage points on the two datasets, respectively.

Critical strengths
  1. The experiments cover a broad range of model architectures and show substantial gains in some heterogeneous distillation settings. The evaluation includes CNN, Transformer, and MLP architectures. For example, on CIFAR-100, MG-EATD improves accuracy from FBT’s 79.57% to 85.56% for ConvNeXt-T → DeiT-T. On ImageNet, the method outperforms FBT in 11 of 14 heterogeneous teacher–student pairs. These results support the practical value of the overall method.
  2. The ablation studies examine several key design choices and include relevant alternatives. The authors investigate trajectory distillation, entropy adaptation, multi-granularity supervision, loss terms, and predictor architectures, and compare direct prediction at two time points with reverse updates. These experiments support the method’s design, although confounding factors in some comparisons limit the attribution of gains to individual components.
Critical weaknesses
  1. Sensitivity to perturbation strength is insufficiently characterized. Although Appendix A.4 motivates the entropy-adaptive schedule and Table 4 reports an EA ablation, the paper does not characterize sensitivity to the diffusion-time bounds and reverse step sizes. It therefore remains unclear how robust the gains are to these settings and when stronger perturbations become detrimental.
  2. The independent contribution of the reverse-trajectory mechanism has not been sufficiently isolated. The teacher trajectory is a mixture of the original class distribution and the uniform distribution. According to Eqs. (19)–(20), the student’s reverse update can be simplified to affine sharpening around the uniform distribution before clipping and normalization. This does not invalidate the training objective, but further evidence is needed to distinguish the gains attributable to probability smoothing, the auxiliary predictor, additional supervision, and the reverse-trajectory mechanism itself. Moreover, the dual-time direct-prediction baseline allows both losses to update the predictor, whereas the full method blocks predictor-parameter gradients from the trajectory loss. The reported 0.32-point improvement therefore does not isolate the effect of the reverse update from that of gradient routing.
Questions for authors
  1. The paper motivates probability-trajectory supervision through architectural differences. Beyond the shared probability space and teacher-matched predictor, what evidence shows that the trajectory mechanism specifically mitigates cross-architecture mismatch, rather than providing general optimization or regularization benefits?
  2. Beyond the EA ablation in Table 4, could you report sensitivity to the diffusion-time bounds and reverse step sizes, and clarify the non-adaptive settings used in the comparison? How robust are the gains across these settings?
  3. Could you compare the reverse update with dual-time direct prediction under the same gradient-routing settings? The current comparison changes both the update mechanism and whether the trajectory loss updates the predictor, making their individual contributions difficult to distinguish.
3 comments
AuthorsOpenReview ↗

[1/2] Official Comment by Authors

Thank you for recognizing the experimental performance of our method. Your suggestions have also helped us identify gaps in the experimental analysis that call for additional experiments and clarification to better explain the method’s design and effectiveness. We address your questions point by point below.

To W1 and Q2. Thank you for highlighting the lack of a detailed sensitivity analysis of the diffusion-time bounds and reverse step sizes. The original EA ablation evaluates the benefit of adaptation under the default configuration, but does not fully characterize the influence of parameter selection. We have therefore added experiments and report their settings and results below.

For a non-adaptive fixed schedule, an excessively large diffusion-time upper bound may push teacher targets too close to the uniform distribution, weakening class-discriminative information. Conversely, an overly small upper bound restricts the coverage of intermediate states, making supervision resemble endpoint matching and limiting the additional benefits of trajectory supervision.

Under the adaptive schedule, tmax⁡=(1−b)tmax⁡weak+btmax⁡strong,t_{\max}=(1-b)t_{\max}^{\mathrm{weak}}+bt_{\max}^{\mathrm{strong}}, reducing tmax⁡weakt_{\max}^{\mathrm{weak}} mainly narrows the diffusion-time range available to samples with low trajectory reliability. Our experiments show limited gains from conservative settings and a marked performance decrease when tmax⁡weakt_{\max}^{\mathrm{weak}} is excessively small. A plausible explanation is that these samples receive insufficient supervision from intermediate states. Lower sampled times may also constrain the effective reverse step, making neighboring-state supervision more redundant.

The reverse step size requires a similar balance. For the Euler-form update, its magnitude depends on c=Δtσ˙(t)c=\Delta t\dot{\sigma}(t). An overly small step makes the update nearly an identity mapping, weakening the neighboring-state constraint. An overly large step may amplify errors in the current prediction and increase the need for clipping and renormalization. Stronger perturbations or larger reverse steps therefore do not necessarily yield higher accuracy.

We conduct these experiments on Swin-T → ResNet18 on ImageNet-1K, varying the diffusion-time bounds and reverse-step ranges separately while keeping all other training settings unchanged. The default configuration is

(t_{\max}^{\mathrm{weak}},t_{\max}^{\mathrm{strong}})=(0.60,0.98),\quad (\Delta t_{\min},\Delta t_{\max})=(0.01,0.08).$$ |Setting|EA|Upper bound|Reverse step|Acc.(%)| |---|:---:|---|---|---:| |Fixed|✗|$0.98$|$0.01$|72.18| |Lower bounds|✓|$(0.4,0.9)$|$(0.01,0.08)$|72.35| |Default|✓|$(0.6,0.98)$|$(0.01,0.08)$|**72.48**| |Higher bounds|✓|$(0.8,0.995)$|$(0.01,0.08)$|72.44| |Smaller steps|✓|$(0.6,0.98)$|$(0.005,0.04)$|72.41| |Larger steps|✓|$(0.6,0.98)$|$(0.05,0.2)$|72.28| |Very small $t_{\max}^{\mathrm{weak}}$|✓|$(0.1,0.98)$|$(0.01,0.08)$|72.03| **To Q1.** MG-EATD matches teacher and student predictions in the shared class-probability space derived from logits, reducing its dependence on direct feature compatibility between architectures. We focus primarily on heterogeneous distillation because larger differences in inductive biases and representations make knowledge transfer more challenging. To better accommodate this setting, we introduce a teacher-matched predictor whose architecture is selected according to the teacher's model family. MG-EATD also applies to homogeneous distillation. In addition to the original ImageNet-1K experiments, we conduct additional experiments with homogeneous teacher–student pairs on CIFAR-100. As shown below, MG-EATD achieves the highest accuracy among the compared methods on all pairs, supporting its broader applicability. |Teacher|Student|Teacher (scratch)|Student (scratch)|KD|SimKD|ReviewKD|CAT-KD|Ours| |---|---|---:|---:|---:|---:|---:|---:|---:| |ResNet110|ResNet32|74.31|71.14|73.08|73.92|73.89|73.62|**74.72**| |WRN-40-2|WRN-16-2|75.61|73.26|74.92|75.53|76.12|75.60|**76.68**| |ResNet32×4|WRN-40-2|79.42|75.61|77.70|79.29|78.96|78.59|**80.24**| |WRN-40-2|ResNet8×4|75.61|72.50|73.97|75.29|74.34|75.38|**77.13**| Thus, architectural differences motivate our focus on heterogeneous distillation, while the benefits of trajectory supervision extend to homogeneous settings as well. The static-target controls with matched predictor capacity and supervision budgets, together with the gradient-matched comparisons, further assess the added value of the trajectory design itself.
AuthorsOpenReview ↗

[2/2] Official Comment by Authors

To W2 and Q3. According to Eqs. (19)–(20), MG-EATD's local reverse update is indeed equivalent to affine sharpening of the current student probability distribution around the uniform distribution, before clipping and normalization. Our code implementation uses this equivalent vector form to avoid explicitly constructing reverse transition rate matrices, reducing computation and memory usage. The reviewer correctly identifies a confounding factor in our original comparison, which does not sufficiently distinguish the contributions of the reverse-update mechanism and predictor-gradient routing. The dual-time baseline directly predicts both states and updates the predictor with both losses, whereas the full method derives the neighboring state through a reverse update and blocks trajectory-loss gradients to predictor parameters. Thus, the improvement of 0.32 percentage points may reflect the update mechanism, gradient routing, or their interaction, and cannot be attributed solely to the reverse update.

Here, we would like to restate our core contribution and clarify how it differs from augmenting teacher targets through an affine combination of teacher outputs and random noise. Taking TeKAP as an example of teacher-target augmentation through an affine combination of teacher outputs and random noise, we can see from its implementation that random perturbations are assigned a relatively small weight to generate augmented targets around the original teacher outputs.

MG-EATD uses discrete diffusion to define a continuous trajectory evolving from the original teacher probability distribution toward the uniform distribution. Through a time-conditioned predictor, the student learns to match the corresponding teacher states at sampled diffusion times. Compared with fixed-strength label smoothing, we additionally establish an explicit connection between the current state and a neighboring lower-noise state: the latter is obtained entirely through the prescribed reverse update of the current prediction and matched to the corresponding state on the same teacher trajectory.

We exclude the trajectory loss from updates to the time-conditioned predictor's parameters to prevent a prediction “shortcut” in which the predictor is jointly optimized to fit both temporal targets. Under this design, neighboring-state matching explicitly depends on the current prediction and its reverse transition, without an additional predictor call to directly predict the neighboring state. This explains the motivation behind our gradient-routing design.

To separate the contributions of the update mechanism and gradient routing, we conduct a crossed comparison on Swin-T → ResNet18 on ImageNet-1K. All configurations use the same training protocol. We vary only how the neighboring state is obtained and whether its loss updates the predictor parameters.

Prediction methodTrajectory loss updates predictorAcc.(%)
Dual-time direct prediction✓72.16
Dual-time direct prediction✗72.10
Reverse update✓72.24
Reverse update (Our method)✗72.48

Under matched gradient-routing settings, reverse updates outperform dual-time direct prediction by 0.08% when the trajectory loss updates the predictor and by 0.38% when it does not. Gradient isolation improves reverse-update accuracy from 72.24% to 72.48%, but does not benefit direct prediction, whose accuracy decreases from 72.16% to 72.10%. These results support the complementary roles of reverse updates and gradient isolation, showing that the full method's advantage cannot be explained by gradient isolation alone.

To further distinguish the sources of improvement, we first introduce shared probability mixing without a time-conditioned predictor, applying the same affine noise mixture to the teacher and student softmax probabilities. This control achieves 71.23%, only 0.09 percentage points above KD’s 71.14%, indicating that this form of endpoint perturbation alone does not reproduce MG-EATD’s gains.

We then use the same auxiliary predictor and match the auxiliary-supervision budget when supervising clean or fixed-smoothed regional targets. These controls achieve 71.86% and 71.72%, respectively. Static auxiliary supervision is therefore beneficial, but the full method achieves 72.48%, exceeding these controls by 0.62 and 0.76 percentage points. Fixed smoothing also provides no numerical advantage over clean targets in this setting, so simply adding uniform smoothing does not recover the full improvement.

Control experimentAcc.(%)
Shared noise mixing of teacher/student probabilities, without an auxiliary predictor71.23
Same auxiliary predictor matching undiffused teacher regional probabilities71.86
Same auxiliary predictor matching teacher regional probabilities with fixed uniform smoothing71.72

Please let us know if any concerns remain unaddressed; we are happy to discuss them.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer H8mX,

We would appreciate your feedback on whether our additional experiments address your questions and whether our explanations help resolve your concerns about the method. We would be happy to discuss any remaining issues.

Reviewer_9ocmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MG-EATD, a discrete-diffusion-driven knowledge distillation framework for heterogeneous teacher–student pairs (e.g., CNN ↔ ViT ↔ MLP-Mixer). Its central contribution is to move beyond static single-endpoint logit matching toward probability-trajectory learning, supervising both intermediate probability states and local state-to-state transitions along a diffusion trajectory. Technically, this involves multi-granularity regional distributions (M=21M=21), a time-conditioned trajectory predictor, closed-form forward diffusion and reverse transition rates, and entropy-adaptive weighting that combines region-level uncertainty with sample-level difficulty. The validation is on CIFAR-100 and ImageNet-1K across heterogeneous and homogeneous distillation settings, with comparisons to baselines such as OFA/FBT and extensive ablations. As I understand it, the review finds the design well-motivated and the ablations thorough, with clear benefits for heterogeneous pairs and more modest gains in homogeneous settings; the main open concerns are computational overhead, the interplay between region-level and sample-level weighting, low-data robustness, and the limited homogeneous-distillation improvement.

Critical strengths

see Questions For Authors

Critical weaknesses

see Questions For Authors

Questions for authors

This paper presents an interesting discrete-diffusion-driven knowledge distillation framework, MG-EATD, targeting heterogeneous teacher–student distillation where model inductive biases differ substantially (CNN ↔ ViT ↔ MLP-Mixer). The core insight is moving beyond static single-endpoint logit matching to probability-trajectory learning, supervising both intermediate probability states and local state-to-state transitions along the diffusion trajectory. Overall the technical design is well-motivated, with thorough ablations, yet several points can be raised for discussion and potential improvement.

Questions & Suggestions

  1. Computational overhead concern The method maintains multi-granularity regional distributions (M = 21 regions), runs a time-conditioned trajectory predictor, and computes forward diffusion closed-form solutions and reverse transition rates for every region. Could the authors report wall-clock training time and GPU memory compared to baselines like OFA and FBT? If the overhead is non-negligible, are there approximations, such as randomly sampling partial regions instead of using all M regions, to reduce cost with only a minor performance drop?

  2. Interplay between the multi-granularity weight w_t^(ℓ) and entropy adaptation The weighting term w_t^(ℓ) combines a region-level uncertainty factor and a sample-level difficulty factor. The region-level uncertainty factor is H(p_t^(ℓ))/2 multiplied by (1 + H(p̂_t^(ℓ))/H_max). The sample-level difficulty factor is 1 + H(p_T)/(2H_max) multiplied by (1 + H(p_S)/H_max). This combines regional trajectory-state entropy and global sample-level difficulty. Could the authors provide more intuition on potential conflicts? For one sample, a region may have high local entropy while global sample reliability is high. How does this weighting behave under such a mismatch? Is there an ablation on removing either the region-level or the sample-level factor?

  3. Transfer to low-data / small-dataset scenarios Results are reported on standard CIFAR-100 and ImageNet-1K. Would the entropy-adaptive trajectory mechanism still work well under a low-data regime, where teacher entropy estimation itself may be noisy?

  4. Homogeneous distillation improvements are moderate While MG-EATD works well for heterogeneous pairs, gains on homogeneous settings are comparatively smaller. Is there an inherent reason for this? When teacher and student share similar inductive biases, static endpoint supervision may already capture most transferable knowledge, leaving limited room for trajectory-based intermediate states to bring extra benefit.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review of MG-EATD and for recognizing both the motivation for extending distillation beyond a single prediction endpoint and the thoroughness of our ablation studies. We address the points you raised about our method below.

To Q1. In practice, MG-EATD reduces the cost of the reverse update by computing it directly in vector form. We describe this vectorized implementation in detail below and compare the training cost of MG-EATD with that of two open-source baselines. Across the evaluated teacher–student pairs, MG-EATD achieves the highest accuracy while remaining competitive in both training time per epoch and GPU memory usage.

TeacherStudentMethodAccuracy (%) ↑Extra Params (M) ↓GPU Memory (GB) ↓Time per Epoch (s) ↓
Swin-TResNet18FitNet71.1811.028.91312.24
Swin-TResNet18OFA71.764.398.71207.91
Swin-TResNet18Ours72.4813.188.81210.65
DeiT-TMobileNetV2FitNet70.951.0912.11180.33
DeiT-TMobileNetV2OFA71.397.1411.61070.74
DeiT-TMobileNetV2Ours71.811.2411.41047.18
ConvNeXt-TSwin-NFitNet74.8111.0223.541162.92
ConvNeXt-TSwin-NOFA77.8328.1922.551427.62
ConvNeXt-TSwin-NOurs78.1611.9419.871041.69

Following your suggestion, we randomly sample 11 of 21 regions per image at each training step. Compared with using all regions, sampling reduces training cost with only a small drop in accuracy. For Swin-T → ResNet18, we achieved 72.39% accuracy and saved an average of 15 s per epoch. We hypothesize that varying the regions across training steps may help regularize training.

To Q2. We believe that confident global predictions can indeed coexist with uncertain local regions. For example, the teacher may confidently recognize a dog from the whole image, while an ear patch may receive similar probabilities for dogs and cats because of their similar local textures. Forward diffusion further smooths regional class distributions, increasing their uncertainty. Consider an idealized early-training case in which the student's global and regional predictions are approximately uniform, with entropies close to Hmax⁡H_{\max}. The regional and sample factors then simplify to

g\approx 1+\frac{H(p_T)}{H_{\max}}.$$ When the teacher confidently recognizes a dog from the whole image, $H(p_T)$ is low and the sample factor approaches 1. Meanwhile, the entropy-adaptive schedule permits a higher diffusion-time upper bound and a larger reverse step. Ambiguous regions, such as an ear patch, may have high regional entropy due to both local ambiguity and forward perturbation, yielding larger regional weights. The sample factor is shared across all regions of an image and therefore does not change their relative weighting. When the global teacher prediction is confident, this factor approaches 1, while uncertain local regions can still receive larger regional weights. We ablated the regional factor, the sample factor, and both on Swin-T → ResNet18. | Setting | Regional factor | Sample factor | Accuracy (%) | |---|:---:|:---:|---:| |Full MG-EATD|✓|✓|72.48| |w/o regional factor|–|✓|72.03| |w/o sample factor|✓|–|72.37| |w/o both factors|–|–|72.18| **To Q3**.Thank you for considering this scenario. Since MG-EATD extends endpoint supervision to probability trajectories, providing richer supervision per sample, we believe it may benefit distillation under limited data. Following CID’s low-data evaluation protocol,we evaluate MG-EATD on VGG-13 → VGG-8 using 20%, 40%, and 60% of the CIFAR-100. |Training data|KD|CID|MG-EATD| |:---:|---:|---:|---:| |20%|59.14|62.13|**63.46**| |40%|66.89|68.64|**69.13**| |60%|69.90|70.85|**70.98**| **To Q4**. The smaller gains in homogeneous distillation may stem from the similar inductive biases of the teacher and student, which can make endpoint supervision more effective and leave less room for additional gains from trajectory supervision. MG-EATD operates in a shared class-probability space and supports both homogeneous and heterogeneous distillation. It achieves the highest accuracy on all homogeneous ImageNet-1K pairs in Table 3. To further assess its generality, we conducted additional homogeneous distillation experiments on CIFAR-100, where MG-EATD also achieved the highest accuracy across all evaluated pairs. |Teacher|Student|Tea Acc.|Stu Acc.|KD|SimKD|ReviewKD|CAT-KD|Ours| |---|---|---|---|---|---|---|---|---| |ResNet110|ResNet32|74.31|71.14|73.08|73.92|73.89|73.62|**74.72**| |WRN-40-2|WRN-16-2|75.61|73.26|74.92|75.53|76.12|75.60|**76.68**| |ResNet32×4|WRN-40-2|79.42|75.61|77.70|79.29|78.96|78.59|**80.24**| |WRN-40-2|ResNet8×4|75.61|72.50|73.97|75.29|74.34|75.38|**77.13**| Please let us know if any concerns remain unaddressed; we are happy to discuss them.
AuthorsOpenReview ↗

Derivation of an Efficient Vectorized Implementation of the Reverse Update

Following the uniform forward-transition formulation considered in SEDD and DiDiCM, we show that the local reverse update in MG-EATD admits an equivalent vectorized expression. This implementation avoids explicitly constructing pairwise probability-ratio matrices or reverse transition rate matrices, reducing the computational and memory costs of the reverse update. The derivation is provided below.

For a fixed sample, region ℓ\ell, and diffusion time tt, let

R=\widehat R_t^{(\ell)},\qquad u=\frac{\mathbf{1}}{K},$$ where $K$ is the number of classes and $u$ is the uniform distribution. Since $p$ is obtained through softmax, it satisfies $p(a)>0$ and $\sum_{a=1}^{K}p(a)=1$. Following the column-vector convention in the main text, $R(b,a)$ denotes the reverse transition rate from class $a$ to class $b$. Under uniform forward transitions, $Q_{ab}=1/K$ for $a\neq b$, and the off-diagonal entries of the estimated reverse transition rate matrix are $$R(b,a)=\frac{\dot{\sigma}(t)}{K}\frac{p(b)}{p(a)},\qquad b\neq a.$$ Following Appendix A.3, the diagonal entries are therefore $$R(b,b)=-\sum_{a\neq b}R(a,b)=-\frac{\dot{\sigma}(t)}{K}\frac{1-p(b)}{p(b)}.$$ Define $v=Rp$ as the instantaneous rate of change of the current probability vector under the estimated reverse dynamics. For class $b$, the off-diagonal terms account for incoming probability mass, while the diagonal term accounts for outgoing probability mass. Thus, $$\begin{aligned} v_b &= \sum_{a\neq b}R(b,a)p(a)+R(b,b)p(b)\\ &= \frac{\dot{\sigma}(t)}{K}\sum_{a\neq b}\frac{p(b)}{p(a)}p(a)-\frac{\dot{\sigma}(t)}{K}\frac{1-p(b)}{p(b)}p(b)\\ &= \frac{\dot{\sigma}(t)}{K}\left[(K-1)p(b)-(1-p(b))\right]\\ &= \dot{\sigma}(t)\left(p(b)-\frac{1}{K}\right). \end{aligned}$$ Since this identity holds for every class $b$, we obtain $$Rp=\dot{\sigma}(t)(p-u).$$ Substituting this identity into the first-order Euler reverse update with step size $\Delta t>0$, and restoring the time and region indices, gives $$\begin{aligned} \widetilde p_{t-\Delta t}^{(\ell)} &= \left(I+\Delta t\\widehat R_t^{(\ell)}\right)\widehat p_t^{(\ell)}\\ &= \widehat p_t^{(\ell)}+\Delta t\\dot{\sigma}(t)\left(\widehat p_t^{(\ell)}-u\right)\\ &= u+\left(1+\Delta t\\dot{\sigma}(t)\right)\left(\widehat p_t^{(\ell)}-u\right). \end{aligned}$$ The update therefore scales the current deviation from the uniform distribution using only elementwise vector operations. This expression is algebraically identical to the original matrix Euler update and introduces no additional approximation; the approximation to continuous-time reverse evolution still arises from first-order time discretization. Explicitly constructing and applying a dense $K\times K$ reverse transition rate matrix requires $\mathcal{O}(K^2)$ computation and storage per region, whereas the vectorized formulation requires only $\mathcal{O}(K)$. For a sample with $M$ regions, the computational complexity of the reverse update therefore decreases from $\mathcal{O}(MK^2)$ to $\mathcal{O}(MK)$. By avoiding intermediate tensors of quadratic size, this implementation reduces the computational and GPU memory overhead of the reverse update. Together with our use of only the penultimate-layer features for multi-granularity supervision, our measurements show that MG-EATD achieves higher accuracy with competitive training time and GPU memory usage compared with the evaluated open-source baselines.
AuthorsOpenReview ↗

Key Differences Between Trajectory Distillation and Random Perturbation Augmentation

Based on the efficient implementation of the reverse update described above, the reviewer may observe that the local reverse update in MG-EATD is equivalent to affine sharpening of the student's current probability distribution around the uniform distribution. TeKAP demonstrates that perturbing teacher outputs can diversify supervision and regularize student learning. From TeKAP code implementation, we can see that random perturbations are assigned a relatively small weight to generate augmented targets around the original teacher outputs. However, this augmentation does not explicitly characterize the full trajectory evolving from a uniform noise distribution toward the target distribution, nor does it explicitly connect the intermediate states along this trajectory through their corresponding reverse transitions.

The central design of MG-EATD uses discrete diffusion to organize the original teacher class-probability distribution into a trajectory evolving toward the uniform distribution:

pt(ℓ)=e−σ(t)p0(ℓ)+(1−e−σ(t))u.p_t^{(\ell)} = e^{-\sigma(t)}p_0^{(\ell)} + \left(1-e^{-\sigma(t)}\right)u.

During training, we sample diffusion times within an entropy-adaptive range and use a time-conditioned predictor to predict the corresponding teacher probability states from student features. The supervision targets thus share an explicit time index and a common evolution rule.

In addition to state matching, we constrain the relationship between neighboring states. The student's lower-noise state at t−Δtt-\Delta t is obtained by applying the prescribed local reverse update to its current prediction p^t(ℓ)\widehat p_t^{(\ell)}, and is supervised by the corresponding state on the same teacher trajectory. This explicitly connects the current prediction to the neighboring lower-noise target without a separate direct prediction at t−Δtt-\Delta t.

To prevent the student from taking a prediction “shortcut” by using the time-conditioned predictor to directly fit both the current and neighboring states, we do not use the trajectory loss Ltraj\mathcal L_{\mathrm{traj}} to update the predictor's parameters; these parameters are updated only by the state loss Lstate\mathcal L_{\mathrm{state}}. The neighboring lower-noise state is obtained entirely through a reverse transition from the current predicted state and is then matched to the teacher's probability state at the corresponding lower-noise time. Thus, neighboring-state matching explicitly depends on the current state and its reverse transition, without an additional direct prediction of the neighboring state by the predictor.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 9ocm,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.