- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies high-resolution dataset distillation under sub-1 IPC-equivalent storage budgets (less than the storage required for one image per class). The previous method, DDiF, stores each synthetic image as an independent small SIREN network. This paper introduces two main modifications: 1. Inspired by STRAINER, the proposed method allows all synthetic images to share an encoder while storing a separate decoder for each image. This approximately halves the per-image storage cost. 2. The method uses two stages of coreset selection for initialization. It first pretrains the shared encoder using samples at median distances from their class centers in feature space, then performs warmup using samples closest to the class centers. Finally, it freezes the shared encoder and optimizes only the decoders using DM or DC. On ImageNet subsets at IPC 0.5 with DM, the proposed method outperforms DDiF by 4.6 and 5.4 percentage points at resolutions of 128×128 and 256×256, respectively. The advantage becomes larger under smaller budgets.
Critical strengths
The experiments are solid and well-rounded, covering two different distillation objectives, different resolutions, cross-architecture generalization, and different budgets.
The results are presented transparently. At approximately matched instance counts of 50 samples per class under DM, the proposed method does not outperform the previous method. At IPC 1.0 under DM, the performance advantage is also small. These results make it clear where the method works best.
Critical weaknesses
-
The core novelty of this paper is that sharing parameters across synthetic samples allows more samples to be stored within the same budget. However, previous works HaBa (Liu et al., NeurIPS 2022) and LinBa (Deng & Russakovsky, NeurIPS 2022) have broadly similar ideas. HaBa uses shared base images and small networks called hallucinators to generate synthetic images. LinBa uses shared memory bases and learned addressing functions to generate images. Although these two methods seem compatible with the pipeline of this task, they are neither cited nor compared in the paper. I would suggest conducting at least some experiments with these two methods and counting the parameters of their shared components toward the budget, as the authors have done for their proposed method.
-
It is not clear how much coreset selection helps the proposed method. Each result is based on only one distillation run, and the reported ± values reflect only the variation across five evaluation classifiers. However, the improvements from coreset selection are only about 1–2 percentage points: +1.6 at IPC 0.5 when comparing the full method with fully random selection, and +1.4 at IPC 1.0 when replacing random pretraining samples with samples at median distances from the class centers. Moreover, the reproduced DDiF curves are not monotonic. Under DM, IPC 0.9 performs 4 points worse than IPC 0.8, with similar behavior under DC. This raises concerns that run-to-run variation could be substantial and potentially comparable to the reported improvements from sample selection. Therefore, more distillation seeds are needed to make the experimental results more convincing.
References
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang. Dataset Distillation via Factorization. NeurIPS 2022.
Zhiwei Deng, Olga Russakovsky. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks. NeurIPS 2022. arXiv:2206.02916.
Questions for authors
-
Since the proposed method claims to reduce storage costs, how would it perform compared with simply storing compressed real images? Based on the byte budget in Appendix A.6.3, an IPC 1.0 budget can store four 128×128 8-bit real images per class. If we use a lower resolution, such as 64×64, it could store 16 images. With JPEG compression at 64×64, it might even fit over 100 images, depending on the compression ratio. How does DISC compare with such a baseline at the same byte budget?
-
Why does DDiF at IPC 0.8 outperform IPC 0.9, and at IPC 0.3 outperform IPC 0.4?
-
What value of M_med is used in the first stage? Considering that the images used during the pretraining stage are not counted toward the final storage budget, why not just use the full training set?
6 comments
Response to Reviewer g7ZR [2/2]
Q1: Compressed real images at the same byte budget. At each budget, every class gets the same number of bytes. JPEG sizes are measured file sizes (64², upsampled to 128² for evaluation). Labels are excluded from every budget. DM, six-subset means:
| Budget | Stored set | Instances/class | Accuracy |
|---|---|---|---|
| IPC 0.25 | JPEG 64², quality 30, random order | 52–73 | 39.6 |
| JPEG 64², quality 30, prototype-nearest | 49–89 | 45.7 | |
| DISC (3-run mean) | 25 | ||
| DISC, 8-bit decoders | 101 | 54.1 | |
| IPC 0.5 | JPEG 64², quality 30, random order | 105–148 | 46.6 |
| JPEG 64², quality 30, prototype-nearest | 98–173 | ||
| DISC (3-run mean) | 50 | 51.5 | |
| DISC, 8-bit decoders | 202 | 56.2 | |
| IPC 1.0 | JPEG 64², quality 30, random order | 211–298 | 53.6 |
| JPEG 64², quality 30, prototype-nearest | 200–339 | ||
| DISC (3-run mean) | 101 | 54.9 | |
| DISC, 8-bit decoders | 405 | 60.1 |
At a fixed byte budget, accuracy tends to grow with the number of stored instances (quality 30 was the best of 30 / 50 / 75 at IPC 1.0). With 8-bit decoders, DISC stores about 1.1–2.1 times as many instances as JPEG and is ahead at every budget. Only the decoders are quantized, after training, and the shared encoder stays at 32 bits and is charged in full. We will add the quantized-decoder results to the revision.
Q2: Why DDiF at IPC 0.9 and 0.4 falls below IPC 0.8 and 0.3. The dips come from the last mini-batch of the evaluation training. Fractional IPCs give image counts that leave a very short last batch at batch size 64 (the batch size in Appendix Table 8, used for both methods). A full search over the curve finds three such cells:
| DM, 128² | Images in the last batch (of 64) | Paper | Remainder-free batch |
|---|---|---|---|
| DDiF, IPC 0.9 | 2 | 46.5 | 50.7 |
| DDiF, IPC 0.4 | 8 | 43.0 | 45.9 |
| DISC, IPC 0.2 | 8 | 43.7 | 46.0 |
Re-evaluating the same checkpoints with a batch size that divides the set removes the dips. DC shows the same pattern (DDiF 0.9: 40.4 → 44.8, DDiF 0.4: 37.1 → 40.1, DISC 0.2: 37.2 → 40.5, re-trained with the paper's code and seed). The main comparison tables (Tables 1 and 2) are unaffected. The affected cells appear only in the budget-scaling summary (Table 4) and in the appendix’s fine-grained IPC table and curve. After correction DISC still leads on average (IPC 0.9: 53.6 vs 50.7, IPC 0.4: 50.3 vs 45.9), and one subset changes winner (Meow at IPC 0.9: DDiF 43.2 vs DISC 40.8). We will use a remainder-free batch for these cells in the revision.
Q3: and the full training set. By default, equals the number of decoders per class (25 / 50 / 101 at IPC 0.25 / 0.5 / 1.0). Pretraining the encoder on the whole class (about 1,300 images per class) at IPC 1.0 gives 54.0 (one run), against 54.9 with 101 images (3-run mean), while pretraining takes about 15 times longer. We therefore keep the small set and will state this in the paper.
References
- Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang. Dataset Distillation via Factorization. NeurIPS 2022.
- Zhiwei Deng, Olga Russakovsky. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks. NeurIPS 2022.
Thank you for your detailed response. My other questions from the initial review have been addressed; only one comparison remains, which will determine my final score.
In the table for Q1, the JPEG baseline was evaluated only at a resolution of 64×64. Furthermore, file sizes were measured on a per-image basis, with each file including its own header. The 8-bit decoders allow DISC to store four times as many samples within the same byte budget, and the shared encoder is accounted for only once. The JPEG baseline should be granted the same consideration: for images of the same quality and dimensions, the quantization tables and standard Huffman tables in the file headers are identical; therefore, a single copy for the entire dataset should suffice and be counted only once.
Please provide the JPEG results for 32×32 and 48×48 resolutions using this accounting method. Specifically, use "prototype-nearest" for image selection, a quality setting of 30, and upsample the images to 128² for evaluation. Please report these results for IPC budgets of 0.25, 0.5, and 1.0, including the number of images stored per class and the standard deviation across the evaluation classifiers.
Additionally, please clarify whether the results for 8-bit DISC represent an average over multiple runs.
If 8-bit DISC still outperforms these JPEG configurations at some of these budgets, I will increase my score.
Response to Reviewer g7ZR [1/2]
We thank the reviewer for the thorough review and the thoughtful comments on related work and the robustness of our results.
W1: HaBa and LinBa. Both are relevant precedents for sharing across synthetic samples, and we will cite and discuss them.
W2: How much coreset selection helps, and run-to-run variation. We repeated the random-selection variants three times with the protocol used for the full method (mean ± std over runs, propagated across subsets as in the paper's tables):
At IPC 0.5, the selection gain is about five times the run-to-run std.
We also repeated distillation three times for both methods on the six 128² ImageNet-Subsets (new runs for DISC and DDiF under one protocol). Each cell is the mean ± std over the three runs, propagated across subsets as above. The 95 % confidence interval of every Δ excludes zero.
The subset winners are those of the submitted table (DISC leads on 35 of 36 cells). At 256² (DM, IPC 1.0), one additional seed per method gives DISC 56.4 and DDiF 53.3. The non-monotonic DDiF curve is not run-to-run variation but an evaluation artifact (Q2).