- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how large amounts of synthetic images can help small classifiers approach the performance of larger teacher models through knowledge distillation. The authors introduce KD+Syn, which combines diffusion-generated images with real images for distillation, and SynDrop, which removes synthetic images partway through training and continues distillation using only real images. Experiments cover five image classification datasets and one semantic segmentation task. The results show that adding synthetic data substantially improves student performance, while SynDrop further improves accuracy and reduces student training computation when the synthetic dataset is sufficiently large. The ResNet-18 student outperforms its ViT-L teacher on Aircraft and Cars.
Critical strengths
- The method improves small models by adjusting the training data and training strategy. On some tasks, the student matches or exceeds the teacher’s accuracy, demonstrating the practical potential of the proposed training approach.
- The authors examine different synthetic dataset sizes, the timing of synthetic data removal, and teacher–student configurations. These experiments suggest that retaining synthetic data throughout training may not be optimal and provide useful insights into how synthetic data should be used.
Critical weaknesses
- Increasing the synthetic dataset size also substantially increases the amount of training. Although the existing comparisons indicate that teacher supervision is beneficial, they do not adequately establish whether conventional distillation could achieve similar results with sufficiently long training and extensive data augmentation. Consequently, the results do not clearly distinguish the contributions of additional data and improved training strategies to the overall gains. This also complicates the assessment of novelty. Previous work has already investigated synthetic images for distillation, and the lack of direct comparisons with relevant training approaches under matched computational budgets leaves the contribution beyond existing methods insufficiently established.
- Removing synthetic images sharply reduces the number of images processed per epoch and substantially increases the proportion of real images used during training. The performance improvement may therefore arise from the combined effects of these changes.
Questions for authors
- Does KD+Syn still offer an advantage when conventional distillation uses sufficient training data and extensive data augmentation? Could these comparisons help distinguish the effects of additional data from those of enhanced training strategies?
- Could similar gains be achieved by retaining synthetic images while increasing the proportion of real images, or by gradually reducing the proportion of synthetic images?
If the authors satisfactorily address my concerns, I will consider raising my score.
2 comments
(W1). Direct comparison with previous work on synthetic images for distillation under matched computational budgets
KD+Syn and SynDrop outperform DM-KD (Li et al., 2023b), the closest previous work, at about half the computational cost, even when DM-KD is given the same real images. DM-KD targets the data-free setting: it distills on synthetic images only, generated by an off-the-shelf diffusion model, with a KL loss. Its published protocol samples ImageNet-1K classes, so we compared against two adaptations to our setting: (i) DM-KD with SDXL, without fine-tuning, prompted with the target class names, using lower guidance as DM-KD recommends, and (ii) the same as (i), but also trained on the real images as in our method, with our augmentation and temperature. All runs use = 100K and the same teacher and student, and both DM-KD variants use as many student updates as KD+Syn.
Hours are for one RTX PRO 6000, as in Appendix B.2. DM-KD skips generator fine-tuning, but SDXL without fine-tuning must sample at its native 1024 px (at 512 px it produces unusable images). This latent has 4× as many positions as our 512 px one, so with the same 20 sampling steps, sampling takes about 3–4× longer:
| Method ( = 100K) | Fine-tuning | Sampling | Training | Total hours | Aircraft | CUB-200 |
|---|---|---|---|---|---|---|
| (i) DM-KD, target-class prompts | 0 | 27–36 | 3 | 30–39 | 53.4 | 57.8 |
| (ii) DM-KD + real images | 0 | 27–36 | 3 | 30–39 | 82.2 | 78.9 |
| KD+Syn (ours) | 4 | 9 | 3 | 16 | 83.3 | 80.8 |
| SynDrop (ours) | 4 | 9 | 1.5 | 14.5 | 85.1 | 81.1 |
Even with the same real images, DM-KD stays 2.9 pt below SynDrop on Aircraft (2.2 on CUB). This gap is our contribution beyond existing methods: synthetic data from the fine-tuned generator, distilled with our KD + CE loss, adds 1.1 pt (1.9 on CUB), and ending training on real data only with SynDrop adds another 1.8 pt (0.3 on CUB). We have added this comparison to Appendix B.8.
(W2, Q2). Fewer images processed per epoch, a higher proportion of real images, and gradually reducing the proportion of synthetic images
The gain comes from ending training with a phase in which real images make up a substantial share of each batch, not from processing fewer images. Removing the synthetic images is the simplest way to obtain this phase, but, as the reviewer suggests, raising the real share while retaining synthetic samples recovers most of the gain. All runs in the two tables below share the first 50 epochs and the learning-rate schedule of SynDrop@50 (Aircraft, ResNet-18, = 1M).
Fewer images processed per epoch. We varied the data and the number of steps after epoch 50 independently:
| After epoch 50 | ~1.4K steps | ~392K steps |
|---|---|---|
| Real only | 87.1 (SynDrop@50) | 87.2 |
| Real + synthetic | 84.5 | 84.7 (no drop) |
Keeping the synthetic images for the same short phase gives 84.5%, 2.6 pt below SynDrop@50, while real-only training for as many steps as no drop gives 87.2%. The total number of images processed therefore does not explain the gain.
A higher proportion of real images. We kept the synthetic images after epoch 50, drawn without replacement from the full 1M pool, and filled a fixed share of each batch with real images:
| Real share per batch | Steps after epoch 50 | Real images processed | Acc. |
|---|---|---|---|
| 0.3% (no drop) | 392K | 167K | 84.7 |
| 50% | 1.4K | 86K | 86.0 |
| 50% | 2.6K | 166K | 86.9 |
| 25% | 5.2K | 166K | 86.7 |
| 100% (SynDrop@50) | 1.4K | 167K | 87.1 |
Once as many real images are processed as in SynDrop (about 167K), a 25–50% real share recovers most of the gain (86.7–86.9%), whereas the natural 0.3% share with the same number gives 84.7%. Removing the synthetic images matches or slightly exceeds this (87.1%) and needs neither a choice of share nor extra steps.
Gradually reducing the proportion of synthetic images does not reproduce the gain. Starting from the full real + 1M synthetic set at epoch 1, we removed about 10.1K randomly chosen synthetic images at every epoch, so that the synthetic pool shrinks linearly and epoch 100 is real only. This processes about as many synthetic images in total as SynDrop@50, but the real share stays below 25% until the final epoch, and accuracy is 83.1%, below both SynDrop@50 (87.1%) and no drop (84.7%).
Thank you for your constructive feedback. We hope these additional results address your concerns, and we are happy to answer any further questions.
Best regards,
The Authors
Reference
L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In CVPR, 2022.
Z. Li, Y. Li, P. Zhao, R. Song, X. Li, and J. Yang. Is synthetic data from diffusion models ready for knowledge distillation? arXiv preprint arXiv:2305.12954, 2023.
Dear Reviewer ZwUm,
Thank you for your time and the constructive review. We answer each point below.
KD+Syn beats real-only KD with a long schedule and strong augmentation at matched training steps. KD+Syn trains on the real training set together with a pool of M synthetic images, for 100 epochs over this combined set. Following Beyer et al. (2022), we distilled on only real data with inception-crop and mixup augmentation, for the same number of steps as KD+Syn with each M. At the 1M budget, this is 783.9K steps on Aircraft and 785.0K on CUB, i.e., about 30K and 21K epochs over the real training sets. KD+Syn uses no such augmentation and is still better for every pool size, by 3.4–5.1 pt on Aircraft and 1.1–1.9 pt on CUB.
Aircraft
CUB-200
This comparison also separates data from training strategy: at a fixed number of updates, the two methods differ only in whether the updates are spent on augmented real images or on new synthetic images.
Training length alone does not help either. We trained KD on the real data plus M synthetic images, M∈{0,10K,100K,1M}, under four step budgets, each equal to 100 epochs at one of these M. Smaller pools are therefore repeated for more epochs at the same budget. Bold marks the best M for each budget, and parentheses give epochs over the training set (real + synthetic).
Aircraft
CUB-200
With real data only (M = 0), accuracy stays flat as the number of updates grows from 2.7K to 783.9K on Aircraft (75.5 → 74.9%) and from 3.8K to 785.0K on CUB (73.8 → 73.6%). At matched updates, the best accuracy rises with M (Aircraft 75.8 → 84.7%, CUB 74.9 → 83.6%), provided the updates suffice to cover the pool. We have added these results to Section 6.4 and Appendix B.7.
Even counting diffusion fine-tuning (4 h) and pool generation (18 h for 200K images), SynDrop with M = 200K beats the augmented baseline trained with about 10× as many student steps (the 1M budget) on both datasets, at lower total cost. At a matched total cost of about 120 h, KD+Syn with M = 1M also beats the augmented real-only baseline trained 4× longer:
KD+Syn with M = 200K is better on Aircraft and comparable on CUB. The 4× budget matches the total cost of KD+Syn with M = 1M, since diffusion fine-tuning and sampling 1M images take about 3× as long as a 1M-budget student run (Appendix B.2). With these 4× more steps, the baseline improves from 81.3 to 82.4% on Aircraft and from 81.7 to 82.6% on CUB, but stays 2.3 and 1.0 pt below KD+Syn.
(continued in the next comment)