acceptodds
Under review as a conference paper at ICLR 2027

Student as Strong as Teacher: Scaling Synthetic Data in Knowledge Distillation of Classifiers

Abstract

We present a pipeline for training a high-performing small classifier through knowledge distillation (KD) augmented with samples generated from a fine-tuned diffusion model. Using this pipeline, we find that student performance scales predictably with the size of the synthetic data. We further improve the pipeline by dropping all synthetic data midway through training, thereby halving the training cost while surprisingly improving performance. Experiments on five classification benchmarks and a semantic segmentation setting consistently show gains from both scaling and dropping synthetic data. To the best of our knowledge, we present the first examples of a ResNet-18 (11M parameters) matching or exceeding a ViT-L teacher (307M parameters) through cross-architecture distillation. Our results suggest that small neural networks can achieve substantially stronger performance when provided with sufficiently rich distillation data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Knowledge Distillation, Synthetic Data, Scaling Law

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZwUmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how large amounts of synthetic images can help small classifiers approach the performance of larger teacher models through knowledge distillation. The authors introduce KD+Syn, which combines diffusion-generated images with real images for distillation, and SynDrop, which removes synthetic images partway through training and continues distillation using only real images. Experiments cover five image classification datasets and one semantic segmentation task. The results show that adding synthetic data substantially improves student performance, while SynDrop further improves accuracy and reduces student training computation when the synthetic dataset is sufficiently large. The ResNet-18 student outperforms its ViT-L teacher on Aircraft and Cars.

Critical strengths
  1. The method improves small models by adjusting the training data and training strategy. On some tasks, the student matches or exceeds the teacher’s accuracy, demonstrating the practical potential of the proposed training approach.
  2. The authors examine different synthetic dataset sizes, the timing of synthetic data removal, and teacher–student configurations. These experiments suggest that retaining synthetic data throughout training may not be optimal and provide useful insights into how synthetic data should be used.
Critical weaknesses
  1. Increasing the synthetic dataset size also substantially increases the amount of training. Although the existing comparisons indicate that teacher supervision is beneficial, they do not adequately establish whether conventional distillation could achieve similar results with sufficiently long training and extensive data augmentation. Consequently, the results do not clearly distinguish the contributions of additional data and improved training strategies to the overall gains. This also complicates the assessment of novelty. Previous work has already investigated synthetic images for distillation, and the lack of direct comparisons with relevant training approaches under matched computational budgets leaves the contribution beyond existing methods insufficiently established.
  2. Removing synthetic images sharply reduces the number of images processed per epoch and substantially increases the proportion of real images used during training. The performance improvement may therefore arise from the combined effects of these changes.
Questions for authors
  1. Does KD+Syn still offer an advantage when conventional distillation uses sufficient training data and extensive data augmentation? Could these comparisons help distinguish the effects of additional data from those of enhanced training strategies?
  2. Could similar gains be achieved by retaining synthetic images while increasing the proportion of real images, or by gradually reducing the proportion of synthetic images?

If the authors satisfactorily address my concerns, I will consider raising my score.

2 comments

Dear Reviewer ZwUm,

Thank you for your time and the constructive review. We answer each point below.

(W1, Q1). Conventional KD with sufficient training and extensive augmentation

KD+Syn beats real-only KD with a long schedule and strong augmentation at matched training steps. KD+Syn trains on the real training set together with a pool of MM synthetic images, for 100 epochs over this combined set. Following Beyer et al. (2022), we distilled on only real data with inception-crop and mixup augmentation, for the same number of steps as KD+Syn with each MM. At the 1M budget, this is 783.9K steps on Aircraft and 785.0K on CUB, i.e., about 30K and 21K epochs over the real training sets. KD+Syn uses no such augmentation and is still better for every pool size, by 3.4–5.1 pt on Aircraft and 1.1–1.9 pt on CUB.

Aircraft

Method \ synthetic pool MM100K200K400K800K1M
Augmented real-only KD78.479.379.780.681.3
KD+Syn (real + MM synthetic)83.384.484.384.584.7

CUB-200

Method \ synthetic pool MM100K200K400K800K1M
Augmented real-only KD79.580.380.781.681.7
KD+Syn (real + MM synthetic)80.881.482.583.483.6

This comparison also separates data from training strategy: at a fixed number of updates, the two methods differ only in whether the updates are spent on augmented real images or on new synthetic images.

Training length alone does not help either. We trained KD on the real data plus MM synthetic images, M∈{0,10K,100K,1M}M \in \{0, 10\text{K}, 100\text{K}, 1\text{M}\}, under four step budgets, each equal to 100 epochs at one of these MM. Smaller pools are therefore repeated for more epochs at the same budget. Bold marks the best MM for each budget, and parentheses give epochs over the training set (real + synthetic).

Aircraft

MM \ steps2.7K10.5K80.8K783.9K
075.5 (100)75.6 (403)75.8 (3,102)74.9 (30,094)
10K78.5 (25.9)80.9 (100)80.5 (776)79.7 (7,525)
100K70.8 (3.3)81.2 (13.0)83.3 (100)81.5 (971)
1M63.2 (0.34)73.5 (1.3)84.3 (10.3)84.7 (100)

CUB-200

MM \ steps3.8K11.6K81.9K785.0K
073.8 (100)74.7 (310)74.9 (2,187)73.6 (20,960)
10K77.0 (32.9)77.5 (100)78.6 (709)76.7 (6,792)
100K75.0 (4.6)79.1 (14.2)80.8 (100)80.5 (959)
1M70.8 (0.48)76.3 (1.5)82.0 (10.4)83.6 (100)

With real data only (MM = 0), accuracy stays flat as the number of updates grows from 2.7K to 783.9K on Aircraft (75.5 → 74.9%) and from 3.8K to 785.0K on CUB (73.8 → 73.6%). At matched updates, the best accuracy rises with MM (Aircraft 75.8 → 84.7%, CUB 74.9 → 83.6%), provided the updates suffice to cover the pool. We have added these results to Section 6.4 and Appendix B.7.

Even counting diffusion fine-tuning (4 h) and pool generation (18 h for 200K images), SynDrop with MM = 200K beats the augmented baseline trained with about 10× as many student steps (the 1M budget) on both datasets, at lower total cost. At a matched total cost of about 120 h, KD+Syn with MM = 1M also beats the augmented real-only baseline trained 4× longer:

MethodTotal hoursAircraftCUB-200
~30 h budget
Augmented real-only KD (1M budget)3081.381.7
KD+Syn (MM = 200K)2884.481.4
SynDrop (MM = 200K)2585.382.0
~120 h budget
Augmented real-only KD (4× the 1M budget)~12082.482.6
KD+Syn (MM = 1M)~12484.783.6

KD+Syn with MM = 200K is better on Aircraft and comparable on CUB. The 4× budget matches the total cost of KD+Syn with MM = 1M, since diffusion fine-tuning and sampling 1M images take about 3× as long as a 1M-budget student run (Appendix B.2). With these 4× more steps, the baseline improves from 81.3 to 82.4% on Aircraft and from 81.7 to 82.6% on CUB, but stays 2.3 and 1.0 pt below KD+Syn.

(continued in the next comment)

(W1). Direct comparison with previous work on synthetic images for distillation under matched computational budgets

KD+Syn and SynDrop outperform DM-KD (Li et al., 2023b), the closest previous work, at about half the computational cost, even when DM-KD is given the same real images. DM-KD targets the data-free setting: it distills on synthetic images only, generated by an off-the-shelf diffusion model, with a KL loss. Its published protocol samples ImageNet-1K classes, so we compared against two adaptations to our setting: (i) DM-KD with SDXL, without fine-tuning, prompted with the target class names, using lower guidance as DM-KD recommends, and (ii) the same as (i), but also trained on the real images as in our method, with our augmentation and temperature. All runs use MM = 100K and the same teacher and student, and both DM-KD variants use as many student updates as KD+Syn.

Hours are for one RTX PRO 6000, as in Appendix B.2. DM-KD skips generator fine-tuning, but SDXL without fine-tuning must sample at its native 1024 px (at 512 px it produces unusable images). This latent has 4× as many positions as our 512 px one, so with the same 20 sampling steps, sampling takes about 3–4× longer:

Method (MM = 100K)Fine-tuningSamplingTrainingTotal hoursAircraftCUB-200
(i) DM-KD, target-class prompts027–36330–3953.457.8
(ii) DM-KD + real images027–36330–3982.278.9
KD+Syn (ours)4931683.380.8
SynDrop (ours)491.514.585.181.1

Even with the same real images, DM-KD stays 2.9 pt below SynDrop on Aircraft (2.2 on CUB). This gap is our contribution beyond existing methods: synthetic data from the fine-tuned generator, distilled with our KD + CE loss, adds 1.1 pt (1.9 on CUB), and ending training on real data only with SynDrop adds another 1.8 pt (0.3 on CUB). We have added this comparison to Appendix B.8.

(W2, Q2). Fewer images processed per epoch, a higher proportion of real images, and gradually reducing the proportion of synthetic images

The gain comes from ending training with a phase in which real images make up a substantial share of each batch, not from processing fewer images. Removing the synthetic images is the simplest way to obtain this phase, but, as the reviewer suggests, raising the real share while retaining synthetic samples recovers most of the gain. All runs in the two tables below share the first 50 epochs and the learning-rate schedule of SynDrop@50 (Aircraft, ResNet-18, MM = 1M).

Fewer images processed per epoch. We varied the data and the number of steps after epoch 50 independently:

After epoch 50~1.4K steps~392K steps
Real only87.1 (SynDrop@50)87.2
Real + synthetic84.584.7 (no drop)

Keeping the synthetic images for the same short phase gives 84.5%, 2.6 pt below SynDrop@50, while real-only training for as many steps as no drop gives 87.2%. The total number of images processed therefore does not explain the gain.

A higher proportion of real images. We kept the synthetic images after epoch 50, drawn without replacement from the full 1M pool, and filled a fixed share of each batch with real images:

Real share per batchSteps after epoch 50Real images processedAcc.
0.3% (no drop)392K167K84.7
50%1.4K86K86.0
50%2.6K166K86.9
25%5.2K166K86.7
100% (SynDrop@50)1.4K167K87.1

Once as many real images are processed as in SynDrop (about 167K), a 25–50% real share recovers most of the gain (86.7–86.9%), whereas the natural 0.3% share with the same number gives 84.7%. Removing the synthetic images matches or slightly exceeds this (87.1%) and needs neither a choice of share nor extra steps.

Gradually reducing the proportion of synthetic images does not reproduce the gain. Starting from the full real + 1M synthetic set at epoch 1, we removed about 10.1K randomly chosen synthetic images at every epoch, so that the synthetic pool shrinks linearly and epoch 100 is real only. This processes about as many synthetic images in total as SynDrop@50, but the real share stays below 25% until the final epoch, and accuracy is 83.1%, below both SynDrop@50 (87.1%) and no drop (84.7%).


Thank you for your constructive feedback. We hope these additional results address your concerns, and we are happy to answer any further questions.

Best regards,

The Authors

Reference

L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In CVPR, 2022.

Z. Li, Y. Li, P. Zhao, R. Song, X. Li, and J. Yang. Is synthetic data from diffusion models ready for knowledge distillation? arXiv preprint arXiv:2305.12954, 2023.

Reviewer_tsTaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study proposes a pipeline to scale knowledge distillation leveraging diffusion-generated synthetic data balanced by class, and removing the synthetic samples midway through training (SynDrop). The pipeline is evaluated across five classification datasets and one segmentation dataset with an RKHS bound to provide practical and theoretical grounding to the study.

Critical strengths
  1. ResNet-18 achieves 87.07% on the Aircraft dataset and 94.99% on the Cars dataset compared to teacher accuracies of 86.62% and 93.32%. SynDrop improves all 20 pairs as shown in Table 1.

  2. Figure 3 demonstrates that SynDrop's accuracy increases with fewer training steps. Appendix B.5 shows that the preference for early dropping under Cross Entropy supports the synthetic-label-noise explanation, but does not isolate it in a controlled manner. Using different Generator and teacher variants and adding segmentation as a separate evaluation criterion broadens the evidence. Tuning α (the loss-balancing knob) and τ (temperature) on real-only validation before fixing them across synthetic runs prevents tuning that favors synthetic-data configurations.

Critical weaknesses
  1. The scaling comparison confounds Synthetic dataset size with a larger number of gradient updates, as when epochs = 100 and batch size = 128, then no-drop updates increase from 2.6k to 784k on the Aircraft dataset and 0.8k to 782k on the Flowers dataset ( discussed in Appendix B.1). Also, stronger baselines like augmented real-only KD and repeated smaller-pool sampling at matched updates are missing. Section B.1 does not provide important augmentation details beyond resizing, given the paper's citation of patient and consistent distillation. However, it's understandable that teacher–student data asymmetry itself is not an invalid comparison. Teacher-student parity on cars does not clearly attribute to knowledge distillation, as ResNet-50 CE+Syn reaches 93.33% without distillation. Also, ResNet-18 KD+Syn already reaches 93.97%, above the teacher's accuracy before SynDrop.

  2. Authors need to bound empirical novelty claims and scaling interpretation. As auxiliary-data training followed by real-data refinement has been discussed as prior art in the same manuscript. The conditional squared-error theorem yields regression bounds that do not necessarily explain classification scaling. Fits of 100.00% accuracy and 0.0000 CE loss asymptotes on Flowers require uncertainty and extrapolation checks.

Questions for authors
  1. Does KD+Syn beat a strong augmented real-only KD baseline and repeated smaller pools, at comparable schedule and gradient updates? Please specify the augmentation types used in the experiment and also present a computational comparison against the accuracy measures.

  2. There seems to be a discrepancy between Figure 3 and Table 1, as reported accuracies in both vary as given: Aircraft student 69.04/68.89, CUB student 70.81/70.31, and CUB teacher 89.80/89.32. Please specify: Are these different runs or configurations, or reporting errors? Also provide repeated-run uncertainty estimates for the principal gains.

  3. Can fitted scaling laws predict accuracy for larger held-out synthetic pools? Please clarify that the 50% savings apply only to student training, since the unamortized end-to-end savings are well below 50% once the synthetic dataset generation costs are included.

3 comments

Dear Reviewer tsTa,

Thank you for your time and the constructive review. We answer each point below.


(W1, Q1). Synthetic dataset size confounded with a larger number of gradient updates

The accuracy gains from a larger synthetic dataset come from the data itself, not from the larger number of gradient updates that comes with it. To separate the two, we trained each pool size M∈{0,10K,100K,1M}M \in \{0, 10\text{K}, 100\text{K}, 1\text{M}\} for the number of steps of a 100-epoch run on each of these pool sizes, so that smaller pools are repeated at matched updates. Bold marks the best MM for each number of steps, and parentheses give the corresponding epochs over the training set (real + synthetic).

Aircraft

MM \ steps2.7K10.5K80.8K783.9K
075.5 (100)75.6 (403)75.8 (3,102)74.9 (30,094)
10K78.5 (25.9)80.9 (100)80.5 (776)79.7 (7,525)
100K70.8 (3.3)81.2 (13.0)83.3 (100)81.5 (971)
1M63.2 (0.34)73.5 (1.3)84.3 (10.3)84.7 (100)

CUB-200

MM \ steps3.8K11.6K81.9K785.0K
073.8 (100)74.7 (310)74.9 (2,187)73.6 (20,960)
10K77.0 (32.9)77.5 (100)78.6 (709)76.7 (6,792)
100K75.0 (4.6)79.1 (14.2)80.8 (100)80.5 (959)
1M70.8 (0.48)76.3 (1.5)82.0 (10.4)83.6 (100)
  • With real data only, accuracy stays flat even with two orders of magnitude more updates.
  • At matched updates, the best accuracy rises with MM (Aircraft 75.8 → 84.7%, CUB 74.9 → 83.6%), provided the updates suffice to cover the pool.
  • Otherwise, more updates do not help: smaller pools saturate or decline, and a 1M pool trained for only 0.34 epochs falls below the real-only student.

We have added these results to Section 6.4 and Appendix B.7.

(W1, Q1). Stronger baselines like augmented real-only KD, and a computational comparison

KD+Syn beats augmented real-only KD at matched training steps, and is better or comparable at matched total compute. Following Beyer et al. (2022), we distilled on real data only with their inception-crop and mixup augmentation. Each column gives the synthetic pool size MM of KD+Syn, and the augmented baseline is trained for the same number of steps as KD+Syn with that pool (100 epochs over real + synthetic data).

Aircraft

Method \ MM100K200K400K800K1M
Augmented real-only KD78.479.379.780.681.3
KD+Syn83.384.484.384.584.7

CUB-200

Method \ MM100K200K400K800K1M
Augmented real-only KD79.580.380.781.681.7
KD+Syn80.881.482.583.483.6

Even when the baseline gets slightly more total compute, KD+Syn is better or comparable. The hours for our methods include fine-tuning the diffusion model (4 h) and generating the pool (18 h).

MethodTotal hoursAircraftCUB-200
Augmented real-only KD (1M budget)3081.381.7
KD+Syn (MM = 200K)2884.481.4
SynDrop (MM = 200K)2585.382.0

SynDrop widens the gap further, reaching higher accuracy on both datasets at even lower cost. A comparison at the 1M budget, with the baseline trained 4× longer, is in progress. (updated: see our follow-up comment below.)

(W2, Q3). Bounding the empirical novelty claims of SynDrop and its 50% savings

We have narrowed our claims about the novelty and savings of SynDrop: the paper now credits the two-stage schedule to prior pretrain-then-refine pipelines, presents SynDrop's contribution as applying it when the auxiliary data must be generated, and states that it saves student training steps only. These pipelines differ from our setting:

  • Yalniz et al. (2019): the auxiliary images are real web images, pseudo-labeled by a teacher.
  • Chen et al. (2020): the auxiliary images are unlabeled real images, used for self-supervised pretraining.
  • Li et al. (2023a): the generative model serves as the teacher, not as a source of training inputs.
  • Basu et al. (2025): generated images are used for pretraining without distillation, for fairness.

SynDrop targets distillation when additional real data are unavailable and must be generated. Because generated images differ from real ones (He et al., 2023; Fan et al., 2024), prolonged training on them can bias the student; dropping them midway removes this bias and improves accuracy once the pool is large (Sec. 6.2). In addition, we have made it clear in the abstract and Section 1 that the roughly 50% saving is in student training cost.

(continued in the next comment)

W2. Regression bounds that do not necessarily explain classification scaling

Proposition 3.1 is a motivating result, not an explanation of the classification scaling. The paper already states that our classification setting with the KD loss departs from it (Sec. 4.1), and we have revised several sentences in Sections 1, 4.2 and 6 to make this distinction explicit.

Q2. Discrepancy between Figure 3 and Table 1

The two took different runs of the same configurations, repeated on different machines (Aircraft student 69.04 vs. 68.89%, CUB student 70.81 vs. 70.31%, CUB teacher 89.80 vs. 89.32%). We now use the Table 1 runs throughout and apologize for the confusion. Such repeats vary by at most about 1 point, an order of magnitude below the main gains (+9.2 and +9.8 points from KD to KD+Syn).

W2. 100.00% accuracy and 0.0000 CE loss asymptotes on Flowers

These are the bounds of our constrained fit (A∞≤100%A_\infty \le 100\%, L∞≥0L_\infty \ge 0), not point estimates. With 95% residual-bootstrap intervals, the data give only A∞≥96.9%A_\infty \ge 96.9\% and L∞≤0.109L_\infty \le 0.109 on Flowers, whereas Aircraft and Cars are bounded on both sides (A∞A_\infty = 88.45% [86.55, 92.52] and 97.02% [95.73, 98.83]). We have added these intervals to Figures 2 and 7.

(W2, Q3). Can fitted scaling laws predict accuracy for larger held-out synthetic pools?

Yes for moderately larger pools, but not precisely far beyond the data. Fitting only MM < 50K, the prediction at 100K is within 0.25 points, comparable to seed variation; at 1M the error is +1.6, +0.9 and −1.4 points on Aircraft, Cars and Flowers.

W1. Attribution of teacher–student parity on Cars

We have made this attribution explicit in Sections 1 and 6.1. On Cars, synthetic data alone bring ResNet-50 to the teacher (CE+Syn 93.33% vs. 93.32%), whereas ResNet-18 needs KD to pass it (91.00% → 93.97%); SynDrop then widens the margin.

(W1, Q1). Augmentation details beyond resizing

We have added them to Appendix B.1: during training, student inputs are resized to 224 × 224, randomly rotated by up to ±15°, horizontally flipped, and normalized with ImageNet statistics.


Thank you again. We are happy to answer any remaining questions, and hope our responses address your concerns.

Best regards,

The Authors

Follow-up on (W1, Q1): computational comparison at the 1M budget

The comparison at the 1M budget is now complete, and KD+Syn remains ahead. Training the augmented real-only baseline 4× longer matches the total cost of KD+Syn with MM = 1M, including diffusion fine-tuning and sampling (Appendix B.2).

MethodTotal hoursAircraftCUB-200
Augmented real-only KD (1M budget)3081.381.7
Augmented real-only KD (4× the 1M budget)~12082.482.6
KD+Syn (MM = 1M)~12484.783.6

With 4× more steps, the baseline gains only about 1 point and stays 2.3 and 1.0 points below KD+Syn.

Reviewer_W5tkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes knowledge distillation with the aid of AI-generated data. The main motivation is that when a student is trained to mimic the teacher's responses on some finite real data points, the student does not get to understand the teacher's behavior on the overall distribution (but only on some finite samples). The authors propose a simple extension to fix this problem; making the students mimic the teacher on significantly more number of images than what is available in the real training dataset. This is done using images produced by generative models. Another version of the method is produced where the generated images are only used in the initial epochs, with the real images being used like normal in the remaining process. The authors show the results on some combinations of teacher and student architecture (ViT/ResNet combination is highlighted) and show improvements over the student's original performance, sometimes even exceeding the teacher's performance.

Critical strengths
  • The authors have done a good job explaining one of the main problems with the vanilla knowledge distillation, where the student only gets the knowledge from some finite number of real samples. The corresponding solution is also explained clearly and logically, where the student learns to mimic the teacher on a much bigger set of samples. This is a very natural use of generative models, something that I believe is being done for the first time.

  • While I am not completely confident on this time, it seems like the first time that I have seen the student achieve better performance than the teacher, especially when there was a big difference in the performance of the individually trained teacher and non-distilled student (e.g., ViT: 86.62% vs ResNet:69.04%). While there are some caveats to this result (please see weaknesses), it is a worthwhile point to be mentioned.

  • The authors also propose a small variant of their method – SynDrop – which uses the generated dataset only for the first half of the training process. This variant, surprisingly, achieves even better performance than the one in which generated dataset is used throughout the training process. While the explanation of this behavior is not very satisfactory, it is again a point worth highlighting.

  • The authors have studied different components of the method; e.g., the effect of teacher's size, guidance scale, generative model. Furthermore, while they have mostly focused on ViT-L + ResNet18 combination, they have considered other combinations as well. Overall, the results on multiple datasets are somewhat consistent, highlighting the universal applicability of synthesized images.

Critical weaknesses
  • While the student's performance rising all the way up to teacher's performance is impressive, it could very well be because of the immense dataset that the generative model (SDXL) was original pre-trained on. So, even though the authors are finetuning the model on some target dataset like Flowers102, the additional information given by the pre-training dataset of SDXL will likely not be completely lost. And if so, then it is not very clear how fair the performance comparison is between the distilled student (which might have used much more information) and the non-distilled student (using only the information from Flowers102, for example). Right now, there is no discussion on this crucial point. In fact, this additional information from the pre-training dataset of SDXL might also explain the student performing better than the teacher itself.

    • To further our understanding regarding this, it will be helpful to conduct an experiment using a generative model NOT pre-trained on any large scale dataset, but trained only on the target dataset (e.g., Flowers102).
    • An alternative is to train the teacher with the same expanded dataset (real + generated) and see if that changes the teacher's performance.
    • Either way, the authors should think of a way to make the game fair when comparing the performance across different types of models in terms of data access. And if the authors believe that unequal access to the dataset is fine, a discussion about the same is important nonetheless.
  • The results will be much convincing if this phenomena holds for some other teacher/student architectures. To get some inspiration, please see some example combinations from [1].

Reference
[1] Contrastive Representation Distillation. Tian et al. ICLR 2020.

Questions for authors
  • Line 40-41. While this makes conceptual sense, it might not be entirely true. There are other works which have found that the student, after mimicking the teacher's response on some finite samples, can end up inheriting some other properties of the teacher even without being asked to learn them [1]. This can also include learning teacher's behavior on unseen data points. I do not point this as a limitation, but just to highlight that knowledge distillation does more than what meets the eye. Some discussion on this topic will be helpful to the readers.

References

[1] What Knowledge Gets Distilled in Knowledge Distillation? Ojha et al. NeurIPS 2023

1 comment

We thank Reviewer W5tk for taking the time to review our paper.

W1. SDXL is pretrained with a large dataset

We would like to state clearly that using the diffusion model trained on a large dataset does not make the comparison unfair and does not undermine the main finding of this paper. In the manuscript, we have already checked for potential sources of unfairness (such as data leakage) and provided additional experiments on the role of the diffusion model’s pretraining dataset.

First, our finding still holds when we replace the diffusion model with a model pretrained on a much smaller dataset. In Section 6.3.1 (Table 2), we reported experimental results for EDM2-S, a diffusion model pretrained on ImageNet-1K rather than a web-scale dataset used in SDXL. We still observe substantial performance gain (e.g., +7.4pp Acc over real-data-only KD on Cars at M=400k) when running our pipeline with EDM2-S. Note that the students are also pretrained with ImageNet-1K, and therefore no “additional” data is used compared with the real-only KD baseline.

Second, we confirm that there is no data leakage between SDXL and the test data. In Appendix B.4, we conducted a copy-detection analysis of all 5M synthetic training images against the corresponding test sets. We found almost no near-duplicates, suggesting that our improvements are unlikely to result from direct reproduction of test images.

Finally, the impact of the large diffusion pretraining dataset is shared with the baseline (CE+Syn) in Table 1. For example, on Stanford Cars with 1M synthetic images for ResNet18, KD+Syn achieves 93.97% accuracy compared to 91.00% with CE+Syn. This suggests that the gains cannot be attributed solely to additional information from the generative model.

The choice of SDXL as a diffusion model is merely due to its high sample quality and is not a necessary component of our pipeline. We will clarify this point in the revised manuscript.

W2. Extension to other teacher/student architectures

We thank the reviewer for this suggestion. We have already investigated the effect of varying teacher architectures in Section 6.3.2 (Figure 5), where we compare ResNet-101, ResNet-152, ViT-B, and ViT-L teachers with a fixed ResNet-18 student on CUB200. We also evaluate two student architectures, ResNet-18 and ResNet-50, in Table 1. Across all combinations, we observe that synthetic data scaling consistently improves student performance across these teachers, with stronger teachers generally yielding greater improvements as the synthetic data pool grows.

To further address the reviewer’s suggestion, we append Appendix B.6, which contains results from additional student models, such as EfficientNet-B0, DeiT-Tiny, and ResNet-34. This provides a more comprehensive view of the robustness of our findings across model architectures.

Q1. Clarification of Line 40-41 (“Student is encouraged to match the teacher only at the training inputs”)

We acknowledge that this sentence is an oversimplification of what is really happening in KD. We tried to emphasize that the standard KD objective does not guarantee matching between the teacher and the student across the entire input domain. To incorporate the reviewer’s feedback, we have updated the manuscript to cite Ojha et al., 2023 and softened our statement.

Thank you again for your constructive feedback. Please let us know if you have any further questions, as we are happy to answer them. If you find your concerns resolved, please consider championing our manuscript for acceptance.

Best regards,

The Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.