acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Distillation As Conditional Generation

Abstract

Knowledge distillation (KD) typically treats the teacher representation as a deterministic target, overlooking the inherent uncertainty of knowledge transfer. In this paper, we reformulate KD as a conditional generative problem and propose Conditional Generation for Knowledge Distillation (CondKD), which models the conditional distribution of teacher representations given the student representation. However, directly applying conditional generation to KD faces two challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels. To address these issues, we introduce a Split Tokenization (SplitTok) strategy, achieving stable and effective unsupervised KD. Additionally, we develop the Distribution Contraction technique to integrate label supervision into the conditional reconstruction process. Our theoretical analysis shows that Distribution Contraction provides a stable gradient surrogate for multi-task learning, enabling efficient supervised training without explicit classification loss on multi-step sampling image representations. With such a simplified and unified objective, our training eliminates heavy hyperparameter tuning and effectively scales to strong training schedules. Experiments demonstrate that CondKD improves over the KL baseline by 16.29% under unsupervised KD on CC3M. With label supervision, our method achieves 82.36% top-1 accuracy with ResNet-50 on ImageNet in 600 epochs without extra data, establishing a new state-of-the-art.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Knowledge distillation, Diffusion model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iJMfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CondKD, which reformulates knowledge distillation as conditional generation of teacher features given student representations. It introduces SplitTok to address high-dimensional optimization and Distribution Contraction to incorporate label supervision. Experiments show competitive results across several benchmarks.

Critical strengths
  1. Reformulating KD as conditional generation offers an interesting perspective. The proposed components are reasonably motivated and fit well within the overall framework.
  2. The experiments cover diverse datasets and teacher-student architectures, providing a relatively broad evaluation of the proposed method.
Critical weaknesses
  1. The paper lacks a direct comparison with deterministic feature regression under the same settings. It remains unclear whether the gains come from generative modeling or simply from transferring teacher features.
  2. Theorem 1 establishes a gradient relationship for a single-step feature estimate, but this does not necessarily imply more stable network optimization. The connection between the theorem and the actual training objective also needs clarification.
  3. While SplitTok alleviates high-dimensional optimization difficulties, its token-wise reconstruction objective does not explicitly model dependencies across feature dimensions. It remains unclear whether this affects feature reconstruction and how SplitTok compares with simpler dimensionality-reduction strategies.
Questions for authors
  1. How does CondKD compare with simple MSE-based feature regression, particularly when using the same Distribution Contraction targets?
  2. Can the authors clarify how Theorem 1 supports the claimed optimization stability beyond the single-step gradient relationship?
  3. Could the authors clarify how SplitTok handles dependencies across feature dimensions, and whether it offers advantages over simpler dimensionality-reduction strategies?
5 comments

Q1: Comparison with deterministic feature prediction and the benefits of uncertainty modeling.

(I) Comparison with deterministic feature prediction. We replace CondKD’s diffusion head with a deterministic MLP predictor, keeping all other training settings unchanged. As shown in the Table, CondKD consistently outperforms the deterministic prediction, supporting the effectiveness of diffusion modeling.

MethodTeacher–studentAcc (%)
CondKDR34–R1872.40
Deterministic feature predictionR34–R1871.51
CondKDR50–MV173.76
Deterministic feature predictionR50–MV172.16
Table R11. Comparison with deterministic feature prediction.

(II) Performance on uncertainty groups. We assess local conditional ambiguity on ImageNet using frozen ResNet-34 and ResNet-18 reference models from PyTorch. We extract features immediately before their linear classifiers and align training features by image identity. For each validation image xix_i, we retrieve its k=50k=50 nearest training examples in the ResNet-18 feature space, using Euclidean distance after per-dimension standardization with training-set statistics. We measure the dispersion of the corresponding ResNet-34 features:

\|t_j-\bar{t}_i\|_2^2}{\frac{1}{N} \sum^{N}_{j=1} \| t_j - \mu_t \|_2^2}$$ Here, $t_j$ denotes the teacher feature, $\bar{t}_i$ is the mean teacher feature within the retrieved neighborhood, $\mu_t$ is the training-set teacher-feature mean, and $N$ is the training-set size. A larger score indicates greater teacher-feature variation among nearby student representations, providing a **local proxy for conditional uncertainty**. We partition validation images into five uncertainty quintiles. Both methods use identical group assignments derived from the frozen reference models, independently of their predictions. CondKD achieves larger accuracy gains in the higher-uncertainty groups, supporting its benefit on examples with greater local conditional ambiguity. | Group | Mean $U$ | PyTorch | Deterministic | CondKD | Gain (pp) | |:--|--:|--:|--:|--:|--:| | Bin 1: lowest uncertainty | 0.237 | 84.51 | 84.98 | **85.07** | +0.09 | | Bin 2 | 0.334 | 78.70 | 79.72 | **79.78** | +0.06 | | Bin 3 | 0.418 | 70.18 | 72.68 | **73.28** | +0.60 | | Bin 4 | 0.511 | 61.66 | 63.68 | **65.28** | +1.60 | | Bin 5: highest uncertainty | 0.647 | 53.74 | 56.50 | **58.57** | +2.07 | | Overall | 0.429 | 69.76 | 71.51 | **72.40** | +0.89 | *Table R12. ImageNet accuracy (%) of R18 across uncertainty quintiles, each containing 10,000 images. PyTorch denotes the supervised pretrained ResNet-18 reference model. Gain denotes CondKD’s improvement over deterministic MSE* **(III) Teacher-feature predictability and local residual ambiguity.** We freeze both trained students and fit matched linear and MLP probes to predict the same raw teacher features. To measure local residual dispersion, we compute $r_j=t_j-g(z_j)$. Here, $g$ averages the predictions of three independently initialized MLP probes. For each validation image, we retrieve its $k$ nearest examples in the corresponding student-feature space and calculate: $$V_i = \frac{1}{D(k-1)} \sum_{j \in \mathcal{N}_{k(i)}} \|r_j - \bar{r}_i \|_2^2, \qquad \bar{r}_i = \frac{1}{k} \sum_{j \in \mathcal{N}_{k(i)}} r_j,$$ Here, $D$ is the teacher-feature dimension. We report the average over all 50,000 ImageNet validation images. | Evaluation metric | Deterministic | CondKD | Reduction | |:--|--:|--:|--:| | Linear-probe MSE | 0.28997 | **0.23880** | **17.65%** | | MLP-probe MSE | 0.26104 | **0.20882** | **20.00%** | | Local residual dispersion, k=25 | 0.21256 | **0.16233** | **23.63%** | | Local residual dispersion, k=50 | 0.21791 | **0.16540** | **24.10%** | | Local residual dispersion, k=100 | 0.22602 | **0.16915** | **25.16%** | *Table R13. Teacher-feature predictability and local residual ambiguity. Lower is better. * CondKD improves teacher-feature predictability and reduces local residual dispersion across neighborhood sizes. These results indicate that its student representation preserves more readily recoverable teacher information and is consistent with reduced effective conditional ambiguity.

Q2: Clarification on Theorem 1.

We thank the reviewer for the question and make the following clarification on Theorem 1. Theorem 1 serves two purposes:

(1) Explaining how Distribution Contraction incorporates label supervision. By rewriting the actual noise-prediction objective using the single-step feature estimate, the theorem decomposes its feature gradient into teacher-feature reconstruction and a label-dependent correction anchored to the ground-truth class prototype. This explains how label supervision enters the unified reconstruction objective without an explicit classification loss.

(2) Establishing the connection between CondKD and multi-task learning at the feature-gradient level. The theorem identifies how CondKD replaces the probability-dependent classification component with a teacher-anchored correction, quantifies their gradient discrepancy, and specifies the condition under which they are close.

These results provide a mechanistic justification for Distribution Contraction and its interpretation as a teacher-guided surrogate of multi-task feature gradients. We agree that they do not alone establish more stable network optimization. We will clarify this scope and revise the corresponding stability claims.

Q3: Dependency modeling across feature dimensions.

(I) Dependencies across feature dimensions. CondKD reconstructs teacher-feature tokens separately, while jointly reconstructing the dimensions within each token. Every token is conditioned on the complete student representation, which is learned jointly with the diffusion head. Reconstruction gradients from all tokens encourage this global condition to encode information useful across multiple teacher-feature groups. Consequently, dependencies across tokens can be captured implicitly through information encoded in the shared student representation.

Although CondKD does not explicitly model residual dependencies beyond this learned condition, our reconstruction-error analysis and overlapping-token ablation will assess the practical consequences of token-wise reconstruction. We leave more advanced cross-token dependency modeling to future work.

(II-a) Reconstruction fidelity. We compare SplitTok with deterministic whole-feature regression on ImageNet validation data. Boundary regions include rr dimensions on each side of internal token boundaries; interiors exclude rr dimensions from each token end.

MethodOverall MSEBoundary (r=4)Interior (r=4)Boundary (r=8)Interior (r=8)
CondKD3.252843.240443.252223.250673.25334
Deterministic regression3.870073.857903.869193.869083.87042

Table R14. Reconstruction errors. All values are reported in units of 10⁻³.

CondKD reduces overall MSE by 15.95%, with comparable boundary and interior errors. This suggests no evident boundary-specific reconstruction penalty, although it does not establish explicit complete cross-token dependency modeling.

(II-b) Ablation on overlapping tokens. We compare the original nonoverlapping partition with 50% overlapping tokens. Overlap allows dimensions spanning the original token boundaries to be reconstructed jointly within a token. Repeated dimensions receive inverse-coverage loss weights, and their predictions are averaged when assembling the complete feature. As shown in the Table, accuracy remains nearly unchanged, suggesting limited benefit from overlap in this setting.

SplitTok configurationTop-1 (%)
Without overlap72.40
With 50% overlap72.39

Table R15. Ablation on overlapping tokens for SplitTok.

Q4: Comparison with simpler dimensionality-reduction strategies.

We compare SplitTok with fixed random projections that compress teacher features to 64 or 128 dimensions. SplitTok instead partitions the full teacher representation into 64-dimensional tokens, retaining all feature dimensions. We evaluate downstream distillation accuracy on ImageNet using R34 as the teacher and R18 as the student. As shown in the table, random projection yields much lower accuracy, possibly because compression discards useful teacher information.

MethodTop-1 accuracy (%)
SplitTok (64-dimensional tokens)72.40
Random projection (64 dimensions)70.94
Random projection (128 dimensions)71.20

Table R16. Comparison with fixed random projections on ImageNet using R34 as the teacher and R18 as the student.

Thank you for the additional experiments and clarifications. The new ImageNet comparisons and SplitTok ablations address several of my initial concerns. Three limitations still inform my assessment.

  1. CC3M comparison. The main CC3M result compares CondKD with logit-based KL distillation. A matched deterministic feature-regression baseline is still missing, leaving the specific contribution of diffusion unclear in this unlabeled setting.
  2. Theoretical scope of Distribution Contraction. For a single-step feature estimate and a frozen teacher classifier, Theorem 1 quantifies the difference between the CondKD and multi-task feature gradients and specifies when they are close. As the authors acknowledge, the claim that this surrogate leads to more stable updates to the student network during training goes beyond what the theorem establishes.
  3. SplitTok. All tokens are conditioned on the full student representation. Once that representation is fixed, each token is reconstructed separately, leaving any remaining dependencies between tokens unmodeled. The overlap ablation shows that jointly reconstructing some dimensions across token boundaries has little effect on accuracy in the tested setting.

Given these limitations, I am retaining my original rating of 2.

Reviewer_7qk8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates knowledge distillation as conditional generation of teacher representations given student representations. It proposes SplitTok, which divides teacher features into lower-dimensional tokens to stabilize diffusion training, and Distribution Contraction, which incorporates labels by moving teacher features toward the corresponding frozen teacher-classifier weight vector. At inference, the diffusion head is removed and a linear classifier is used.

Critical strengths

1.The paper explores a potentially useful formulation of feature distillation. Modeling teacher representations conditioned on student representations offers an alternative to direct pointwise feature matching, particularly when teacher and student models differ in architecture or capacity. The paper also identifies the optimization challenges of applying diffusion to high-dimensional features and introduces SplitTok to address them, with ablation results supporting the need for this design.

2.The empirical evaluation covers several relevant settings. The experiments include supervised and label-free distillation, balanced and long-tailed data, and multiple teacher–student pairs. The reported ImageNet results are promising, and removing the diffusion head at inference avoids adding iterative sampling latency to the deployed student.

Critical weaknesses

1.The theoretical justification for Distribution Contraction is not clearly tied to the multi-task objective introduced in Eq. (6). Eq. (6) applies the student classifier CsC_s to the reconstructed feature, whereas Eq. (9), used in Theorem 1, applies the teacher classifier CtC_t. The paper does not clarify why the classifier changes or whether Theorem 1 establishes a surrogate relation to the objective in Eq. (6).

2.The novelty of CondKD over prior generative feature-distillation methods is not sufficiently established. The main comparisons do not include relevant diffusion-based KD baseline, such as DiffKD.Although the paper includes ablations of SplitTok and Distribution Contraction, it does not isolate the contribution of conditioning teacher-feature generation on student features.

Questions for authors

1.Why Eq. (9) uses CtC_t instead of CsC_s, as in Eq. (6), and whether Theorem 1 justifies the multi-task objective in Eq. (6)?

2.The lack of experimental comparisons with state-of-the-art generative distillation methods. DiffKD is discussed and compared in Appendix A.7, but only on three CIFAR-100 configurations.

3.The lack of ablation studies for individual components demonstrating the contribution of conditioning on student features.

4.Prior generative distillation methods are not sufficiently discussed. How does CondKD differ from prior generative and diffusion-based distillation methods, particularly DiffKD, in terms of its objective and mechanism?

5.The BEiT-L-R50(A2) result in Table 6 appears inconsistent with the result in Table 2.

3 comments

Q1: About Eq.(6) and Eq.(9).

Thank you for pointing out this missing clarification. Eq. (9) considers a fixed-classifier specialization of the multi-task formulation in Eq. (6), where Cs=CtC_s=C_t and the classifier parameters are frozen. This choice is motivated by the reconstruction objective, which encourages the generated features to align with the teacher’s feature space and therefore allows the teacher classifier to provide semantic supervision. Under this assumption, Theorem 1 establishes the gradient-level surrogate relation for this specialization. We will explicitly state the classifier assumption in Theorem 1 and clarify the connection between Eqs. (6) and (9).

Q2: Contribution of conditioning on student features.

(I) The effectiveness of diffusion modeling. To assess diffusion modeling, we replace the diffusion head with a deterministic MLP predictor, keeping all other training settings unchanged. CondKD improves ImageNet accuracy by 0.89% and 1.60% for R34–R18 and R50–MV1, respectively.

To assess student conditioning, we shuffle or zero the conditions during training. Shuffling breaks student–target correspondence, while zeroing blocks reconstruction gradients to the student. Both interventions reduce R18 top-1 accuracy to approximately 0.2%, supporting the importance of informative, correctly paired student conditions.

MethodTeacher–studentAcc (%)
CondKDR34–R1872.40
Deterministic feature predictionR34–R1871.51
Shuffled or zeroed conditionsR34–R18≈0.20
CondKDR50–MV173.76
Deterministic feature predictionR50–MV172.16

Table R7. Ablations of diffusion modeling and student-feature conditioning on ImageNet.

(II) Performance on uncertainty groups. We assess local conditional ambiguity on ImageNet using frozen ResNet-34 and ResNet-18 reference models from PyTorch. We extract features immediately before their linear classifiers and align training features by image identity. For each validation image xix_i, we retrieve its k=50k=50 nearest training examples in the ResNet-18 feature space, using Euclidean distance after per-dimension standardization with training-set statistics. We measure the dispersion of the corresponding ResNet-34 features:

\|t_j-\bar{t}_i\|_2^2}{\frac{1}{N} \sum^{N}_{j=1} \| t_j - \mu_t \|_2^2}$$ Here, $t_j$ denotes the teacher feature, $\bar{t}_i$ is the mean teacher feature within the retrieved neighborhood, $\mu_t$ is the training-set teacher-feature mean, and $N$ is the training-set size. A larger score indicates greater teacher-feature variation among nearby student representations, providing a **local proxy for conditional uncertainty**. We partition validation images into five uncertainty quintiles. Both methods use identical group assignments derived from the frozen reference models, independently of their predictions. CondKD achieves larger accuracy gains in the higher-uncertainty groups, supporting its benefit on examples with greater local conditional ambiguity. | Group | Mean $U$ | PyTorch | Deterministic | CondKD | Gain (pp) | |:--|--:|--:|--:|--:|--:| | Bin 1: lowest uncertainty | 0.237 | 84.51 | 84.98 | **85.07** | +0.09 | | Bin 2 | 0.334 | 78.70 | 79.72 | **79.78** | +0.06 | | Bin 3 | 0.418 | 70.18 | 72.68 | **73.28** | +0.60 | | Bin 4 | 0.511 | 61.66 | 63.68 | **65.28** | +1.60 | | Bin 5: highest uncertainty | 0.647 | 53.74 | 56.50 | **58.57** | +2.07 | | Overall | 0.429 | 69.76 | 71.51 | **72.40** | +0.89 | *Table R8. ImageNet accuracy (%) of R18 across uncertainty quintiles, each containing 10,000 images. PyTorch denotes the supervised pretrained ResNet-18 reference model. Gain denotes CondKD’s improvement over deterministic MSE*

Q3: Comparison with diffKD.

(I) Methodological comparison. DiffKD uses a diffusion model trained on teacher features to denoise student features, followed by explicit feature matching. CondKD instead generates teacher features conditioned on student representations and directly supervises the student through conditional reconstruction in a unified loss function, while diffKD applies multiple losses for joint optimization.

AspectdiffKDCondKD
Diffusion usageDenoises student features using a diffusion model trained on teacher features, then explicitly matches the refined features to teacher features.Generates teacher-feature tokens conditioned on student representations, using conditional diffusion to supervise representation learning.
Training objectiveJointly optimizes task loss (CE), unconditional diffusion loss, autoencoder reconstruction loss, and feature-matching KD loss.Uses a unified conditional reconstruction objective, with label supervision incorporated through Distribution Contraction.

Table R9. Conceptual comparison between diffKD and CondKD.

(II) Empirical comparison. Under classical ImageNet training, CondKD achieves 72.40% versus DiffKD’s 72.22% for R34–R18, and 73.76% versus 73.62% for R50–MV1, indicating comparable performance in this setting.

Under their respective strong training recipes, CondKD achieves 81.96% with a ResNet-50 student, compared with diffKD’s 80.50%, using teachers with comparable top-1 accuracy. Since teacher architectures and training recipes differ, we are conducting an additional comparison with a matched teacher and training configuration.

MethodTeacher (top-1 %)EpochsStudent top-1 (%)
DiffKDSwin-L (86.30)45080.50
CondKDBEiT-B (86.12)30081.96

Table R10. Reported strong-setting results on ImageNet with a ResNet-50 student. Teachers achieve similar performance.

Q4: The BEiT-L-R50(A2) result in Table 6 appears inconsistent with the result in Table 2.

Thank you for catching this discrepancy. The column labeled BEiT-L-R50(A2) in Table 6 should be BEiT-L-R50(A1). The reported accuracies correspond to the A1 setting and are consistent with Table 2. We will correct the column heading in the revised manuscript.

Q5: Empirical comparison with diffKD on ImageNet.

We reproduce diffKD using BEiT-B as the teacher and ResNet-50 as the student under the A2 training strategy on ImageNet. diffKD achieves 81.20% top-1 accuracy, which is slightly better than vanilla KD. In contrast, CondKD achieves 81.96%, outperforming diffKD by 0.76% in this setting.

Q6: Contribution of student-feature conditioning.

We evaluate the trained CondKD model with a ResNet-34 teacher and a ResNet-18 student on all 50,000 ImageNet validation images, using resize-to-256 and center-crop-to-224 preprocessing. For each image xix_i, we generate the teacher-feature sample conditioned on its student representation and compute the MSE between the generated sample and the corresponding teacher-feature target. We then replace the student condition with that of another validation image, selected either globally or within the same class, while keeping the original teacher target and sampling noise fixed. Labels are used only to construct targets and select same-class donors; they are not supplied to the generator.

Global and within-class shuffling increase MSE to 10.17×10.17\times and 3.62×3.62\times the correct-condition value, respectively. The degradation under within-class shuffling indicates that CondKD uses image-specific student information beyond class identity to reconstruct teacher features.

ConditionMSE
Correct3.253
Globally shuffled33.084
Within-class shuffled11.783

Table R17. Effect of student-condition shuffling on reconstruction. All MSE values are reported in units of 10−310^{-3}.

Reviewer_MT46OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to address the knowledge distillation problem. Compared with existing strategies that are deterministic, the uncertainty is ignored. To this end, this paper proposes CondKD, to leverage uncertainty modelling with a diffusion-based objective. Experimental results show the effectiveness of the proposed method.

Critical strengths
  1. The idea of uncertainty modeling and Conditional Generation for KD is interesting.

  2. The experiment section is solid, despite some inconsistencies.

  3. The performance of CondKD seems strong compared with existing KD methods like KD and DKD.

Critical weaknesses
  1. The key motivation is not validated. While the paper claims "a given student representation may be compatible with multiple plausible teacher representations. Thus, knowledge transfer need not follow a deterministic mapping.", there is no direct evidence that this uncertainty itself is significant, and the modeling of such uncertainty is beneficial. The benefit can only be interpreted from better downstream performance (accuracy).

  2. Missing important baselines. While the paper claims the proposed CondKD is state-of-the-art (abstract, line 28), recent state-of-the-art methods are missing from the comparison. I suggest comparing with KD methods from recent venues (SKD [a], CVPR 26; SDD [b], CVPR 24; MDS [c], ICLR 25; LS [d], CVPR 24), to ground such a state-of-the-art claim.

  3. This paper claims that CondKD scales more effectively to strong training schedules (line 24). However, this is not sufficiently supported by experiments. As in Table 2, when moving from A2 to A1, all baselines (supervised training of student alone, DKD, and KD) gain significant performance. And all of them, even without distillation(0.58pp), gain more than CondKD (0.57pp). Such an experiment can only prove "CondKD can scale", not "CondKD scales better than baselines".

  4. There are some inconsistencies in the results in the table. In Table 2, (ResNet-34, ResNet-18) is reported to be 72.40, while the same setting in Table 6 is 73.76. This difference is too significant to be due to randomness.

References:

  • [a] Streamlined Knowledge Distillation. CVPR 2026
  • [b] Scaled decoupled distillation. CVPR 2024
  • [c] Medium-Difficulty Samples Constitute Smoothed Decision Boundary for Knowledge Distillation on Pruned Datasets. ICLR 2025
  • [d] Logit Standardization in Knowledge Distillation. CVPR 2024
Questions for authors
  1. Is there any direct proof that modelling the uncertainty is beneficial to KD? Maybe through more direct empirical results.

  2. Is there any proof that CondKD outperforms existing state-of-the-art methods, like SKD, SDD, MDS, and LS mentioned above?

  3. What is the full overhead of CondKD compared with baselines, including end-to-end per-epoch training time and peak memory overhead?

  4. What is the standard deviation of the results reported in the tables?

4 comments

Q1: There is no direct evidence that the uncertainty itself is significant, and the modeling of such uncertainty is beneficial.

We address this concern through three analyses: (I) comparison with deterministic feature prediction; (II) performance across independently defined uncertainty groups; and (III) conditional distribution evaluation.

(I) Comparison with deterministic feature prediction. We replace CondKD’s diffusion head with a deterministic MLP predictor, keeping all other training settings unchanged. As shown in the Table, CondKD consistently outperforms the deterministic prediction, supporting the effectiveness of diffusion modeling.

MethodTeacher–studentAcc (%)
CondKDR34–R1872.40
Deterministic feature predictionR34–R1871.51
CondKDR50–MV173.76
Deterministic feature predictionR50–MV172.16
Table R1. Comparison with deterministic feature prediction.

(II) Performance on uncertainty groups. We assess local conditional ambiguity on ImageNet using frozen ResNet-34 and ResNet-18 reference models from PyTorch. We extract features immediately before their linear classifiers and align training features by image identity. For each validation image xix_i, we retrieve its k=50k=50 nearest training examples in the ResNet-18 feature space, using Euclidean distance after per-dimension standardization with training-set statistics. We measure the dispersion of the corresponding ResNet-34 features:

\|t_j-\bar{t}_i\|_2^2}{\frac{1}{N} \sum^{N}_{j=1} \| t_j - \mu_t \|_2^2}$$ Here, $t_j$ denotes the teacher feature, $\bar{t}_i$ is the mean teacher feature within the retrieved neighborhood, $\mu_t$ is the training-set teacher-feature mean, and $N$ is the training-set size. A larger score indicates greater teacher-feature variation among nearby student representations, providing a **local proxy for conditional uncertainty**. We partition validation images into five uncertainty quintiles. Both methods use identical group assignments derived from the frozen reference models, independently of their predictions. CondKD achieves larger accuracy gains in the higher-uncertainty groups, supporting its benefit on examples with greater local conditional ambiguity. | Group | Mean $U$ | PyTorch | Deterministic | CondKD | Gain (pp) | |:--|--:|--:|--:|--:|--:| | Bin 1: lowest uncertainty | 0.237 | 84.51 | 84.98 | **85.07** | +0.09 | | Bin 2 | 0.334 | 78.70 | 79.72 | **79.78** | +0.06 | | Bin 3 | 0.418 | 70.18 | 72.68 | **73.28** | +0.60 | | Bin 4 | 0.511 | 61.66 | 63.68 | **65.28** | +1.60 | | Bin 5: highest uncertainty | 0.647 | 53.74 | 56.50 | **58.57** | +2.07 | | Overall | 0.429 | 69.76 | 71.51 | **72.40** | +0.89 | *Table R2. ImageNet accuracy (%) of R18 across uncertainty quintiles, each containing 10,000 images. PyTorch denotes the supervised pretrained ResNet-18 reference model. Gain denotes CondKD’s improvement over deterministic MSE* **(III) Teacher-feature predictability and local residual ambiguity.** We freeze both trained students and fit matched linear and MLP probes to predict the same raw teacher features. To measure local residual dispersion, we compute $r_j=t_j-g(z_j)$. Here, $g$ averages the predictions of three independently initialized MLP probes. For each validation image, we retrieve its $k$ nearest examples in the corresponding student-feature space and calculate: $$V_i = \frac{1}{D(k-1)} \sum_{j \in \mathcal{N}_{k(i)}} \|r_j - \bar{r}_i \|_2^2, \qquad \bar{r}_i = \frac{1}{k} \sum_{j \in \mathcal{N}_{k(i)}} r_j,$$ Here, $D$ is the teacher-feature dimension. We report the average over all 50,000 ImageNet validation images. | Evaluation metric | Deterministic | CondKD | Reduction | |:--|--:|--:|--:| | Linear-probe MSE | 0.28997 | **0.23880** | **17.65%** | | MLP-probe MSE | 0.26104 | **0.20882** | **20.00%** | | Local residual dispersion, k=25 | 0.21256 | **0.16233** | **23.63%** | | Local residual dispersion, k=50 | 0.21791 | **0.16540** | **24.10%** | | Local residual dispersion, k=100 | 0.22602 | **0.16915** | **25.16%** | *Table R3. Teacher-feature predictability and local residual ambiguity. Lower is better. * CondKD improves teacher-feature predictability and reduces local residual dispersion across neighborhood sizes. These results indicate that its student representation preserves more readily recoverable teacher information and is consistent with reduced effective conditional ambiguity.

Q2: We have already compared with SD-KD[b], LSKD[d] in Table 2.

We already compare with SD-KD [b] and LSKD [d] in Table 2. MDS [c] studies distillation on pruned training sets; we will discuss its relationship to CondKD in the revision. We additionally compare with SKD [a] below.

MethodTeacher–studentAcc (%)
SKD [a]R34–R1871.13
SKD [a]R50–MV171.53
SD-KD [b]R34–R1871.44
SD-KD [b]R50–MV172.24
SD-KD+DKD [b]R34–R1872.02
SD-KD+DKD [b]R50–MV173.08
LSKD-KD [d]R34–R1871.42
LSKD-KD [d]R50–MV172.18
LSKD+DKD [d]R34–R1871.88
LSKD+DKD [d]R50–MV172.85
CondKD (ours)R34–R1872.40
CondKD (ours)R50–MV173.76

Table R4. Comparison with recent distillation baselines on ImageNet.

Q3: Clarification on misunderstandings of training scalability.

Thank you for raising this point. Comparing absolute accuracy increments from different starting accuracies does not, by itself, establish which method scales more effectively. Higher starting accuracy can leave less room for further improvement. CondKD already achieves 81.79% under A2, substantially exceeding the compared baselines, so its smaller A2-to-A1 increment should be interpreted in this context.

Our scalability claim concerns the ability to sustain effective distillation under strong training recipes. Under A2, DKD and ReviewKD achieve 80.77% and 79.11%, respectively, compared with 80.89% for vanilla KD, while CondKD reaches 81.79%. Under A1, CondKD further reaches 82.36%, exceeding vanilla KD and DKD by 0.68% and 0.53%. Furthermore, CondKD maintains an approximately 2% improvement over training without distillation under both recipes.

Q4: Inconsistent number in Table 6.

Thank you for catching this discrepancy. In Table 6, we mistakenly entered the accuracy of ResNet-50 to MobileNetV1 (73.76%) for the ResNet-34 to ResNet-18 configuration. The correct value is 72.40%, consistent with Table 2. We will correct this entry in the revised manuscript.

Q5: We have already included the training and inference efficiency comparisons in Table 6.

Here, we compare GPU memory usage during training below.

MethodTeacher–studentTraining memory (MB)Acc (%)
KDR34–R185,24871.24
CondKDR34–R186,27272.40
KDBEiT-B–R5013,77880.96
CondKDBEiT-B–R5015,02881.96
KDBEiT-L–R5015,38481.68
CondKDBEiT-L–R5016,69482.36

Table R5. Measured using four RTX 4090 GPUs with a batch size of 512 images.

Q6: Standard deviation of the results.

Teacher–studentAcc (%)
WRN40-2–WRN40-175.98 ± 0.11
R56–R2072.63 ± 0.05
R32×4–R8×477.47 ± 0.08

Table R6. CIFAR-100 results reported as mean ± standard deviation.

Reviewer_MT46OpenReview ↗

Thank you for the rebuttal

I would like to thank the authors for their effort during the rebuttal period.

The rebuttal has addressed my concerns, and I would like to calibrate the score to 3, weak accept.

That said, I believe the authors still need a major effort in revising the manuscript and including the new information. Especially regarding the w3, I highly suggest that the authors be more defensive in their claims. Currently, it is still not very clear or well-grounded that "CondKD scales better than other methods". Without more evidence, maybe it's better to review this description.

AuthorsOpenReview ↗

Thanks for your careful evaluation and constructive suggestions. We're glad that our responses have addressed your concerns.

Thank you for your careful evaluation and constructive suggestions. We are glad that our responses have addressed your concerns and appreciate your updated assessment of ``weak accept". We will thoroughly revise the manuscript to incorporate the additional results and clarifications. In particular, regarding W3, we will qualify the claim that “CondKD scales better than other methods,” limiting our conclusions to the teacher–student configurations and training settings supported by the experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.