acceptodds
Under review as a conference paper at ICLR 2027

DISC: Dataset Distillation for Sub-1 IPC Learning under Implicit Shared Representations with Coresets

Abstract

Distilling high-resolution datasets under tight storage budgets requires synthetic representations that are both compact and useful for downstream learning. We introduce Dataset Distillation for Sub-1 IPC Learning under Implicit Shared Representations with Coresets (DISC), a framework for the sub-1 images-per-class (IPC) regime, where the storage budget is smaller than that required for 1 IPC. Building on the storage efficiency of implicit neural representations (INRs), DISC further reduces per-instance storage costs by sharing a coordinate encoder across synthetic images while retaining lightweight instance-specific decoders. This allows more synthetic instances to be stored within the same byte budget. To translate this additional capacity into effective training data, DISC couples parameter sharing with a two-stage coreset initialization: median-distance samples pretrain the shared encoder on non-outlier variation, and prototype-nearest samples warm up the encoder and decoders with representative examples before distillation. On six ImageNet subsets at a 0.5 IPC-equivalent budget, DISC improves mean Top-1 accuracy over DDiF by 4.6 and 5.4 percentage points at and , respectively, under distribution matching. Further improvements under gradient matching and in average cross-architecture performance demonstrate the effectiveness of combining shared representations with stage-specific initialization for a better accuracy–storage trade-off in highly compressed dataset distillation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Dataset Distillation, Implicit Neural Representations, Coreset Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g7ZROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies high-resolution dataset distillation under sub-1 IPC-equivalent storage budgets (less than the storage required for one image per class). The previous method, DDiF, stores each synthetic image as an independent small SIREN network. This paper introduces two main modifications: 1. Inspired by STRAINER, the proposed method allows all synthetic images to share an encoder while storing a separate decoder for each image. This approximately halves the per-image storage cost. 2. The method uses two stages of coreset selection for initialization. It first pretrains the shared encoder using samples at median distances from their class centers in feature space, then performs warmup using samples closest to the class centers. Finally, it freezes the shared encoder and optimizes only the decoders using DM or DC. On ImageNet subsets at IPC 0.5 with DM, the proposed method outperforms DDiF by 4.6 and 5.4 percentage points at resolutions of 128×128 and 256×256, respectively. The advantage becomes larger under smaller budgets.

Critical strengths

The experiments are solid and well-rounded, covering two different distillation objectives, different resolutions, cross-architecture generalization, and different budgets.

The results are presented transparently. At approximately matched instance counts of 50 samples per class under DM, the proposed method does not outperform the previous method. At IPC 1.0 under DM, the performance advantage is also small. These results make it clear where the method works best.

Critical weaknesses
  1. The core novelty of this paper is that sharing parameters across synthetic samples allows more samples to be stored within the same budget. However, previous works HaBa (Liu et al., NeurIPS 2022) and LinBa (Deng & Russakovsky, NeurIPS 2022) have broadly similar ideas. HaBa uses shared base images and small networks called hallucinators to generate synthetic images. LinBa uses shared memory bases and learned addressing functions to generate images. Although these two methods seem compatible with the pipeline of this task, they are neither cited nor compared in the paper. I would suggest conducting at least some experiments with these two methods and counting the parameters of their shared components toward the budget, as the authors have done for their proposed method.

  2. It is not clear how much coreset selection helps the proposed method. Each result is based on only one distillation run, and the reported ± values reflect only the variation across five evaluation classifiers. However, the improvements from coreset selection are only about 1–2 percentage points: +1.6 at IPC 0.5 when comparing the full method with fully random selection, and +1.4 at IPC 1.0 when replacing random pretraining samples with samples at median distances from the class centers. Moreover, the reproduced DDiF curves are not monotonic. Under DM, IPC 0.9 performs 4 points worse than IPC 0.8, with similar behavior under DC. This raises concerns that run-to-run variation could be substantial and potentially comparable to the reported improvements from sample selection. Therefore, more distillation seeds are needed to make the experimental results more convincing.

References

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang. Dataset Distillation via Factorization. NeurIPS 2022.

Zhiwei Deng, Olga Russakovsky. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks. NeurIPS 2022. arXiv:2206.02916.

Questions for authors
  1. Since the proposed method claims to reduce storage costs, how would it perform compared with simply storing compressed real images? Based on the byte budget in Appendix A.6.3, an IPC 1.0 budget can store four 128×128 8-bit real images per class. If we use a lower resolution, such as 64×64, it could store 16 images. With JPEG compression at 64×64, it might even fit over 100 images, depending on the compression ratio. How does DISC compare with such a baseline at the same byte budget?

  2. Why does DDiF at IPC 0.8 outperform IPC 0.9, and at IPC 0.3 outperform IPC 0.4?

  3. What value of M_med is used in the first stage? Considering that the images used during the pretraining stage are not counted toward the final storage budget, why not just use the full training set?

6 comments
AuthorsOpenReview ↗

Response to Reviewer g7ZR [1/2]

We thank the reviewer for the thorough review and the thoughtful comments on related work and the robustness of our results.

W1: HaBa and LinBa. Both are relevant precedents for sharing across synthetic samples, and we will cite and discuss them.

  • HaBa (Liu et al., NeurIPS 2022) needs at least one full-resolution basis per class in its ImageNet configuration, i.e. at least IPC 1.0 (about 1.06 with the hallucinators counted), and it reports no setting below IPC 1.0. Its published 1-basis result on the six subsets (supplementary Table 2) is 41.4 (MTT). DISC reaches 48.5 at IPC 0.25 and 54.9 at IPC 1.0 (DM, 3-run means), and its IPC 0.25 result is higher on all six subsets.
  • LinBa (Deng & Russakovsky, NeurIPS 2022): we ran the official code at 128², IPC 0.5, counting bases and addresses toward the budget as you suggested. Its unrolled training costs about 85 s per iteration at 128², so 1,000 outer iterations (published: 50,000) already take about 24 GPU-h per subset, six times DISC's 3.9 h, and reach 32.8. The same representation optimized with DM under DISC's schedule reaches 35.1, against 51.5 for DISC and 48.3 for DDiF (3-run means).

W2: How much coreset selection helps, and run-to-run variation. We repeated the random-selection variants three times with the protocol used for the full method (mean ± std over runs, propagated across subsets as in the paper's tables):

Comparison (DM, six-subset mean)Full method, 3 runsRandom selection, 3 runsΔ
IPC 0.5: two-stage selection vs fully random51.5 ± 0.250.4 ± 0.2+1.1
IPC 1.0: median-distance vs random pretraining set54.9 ± 0.254.2 ± 0.2+0.7

At IPC 0.5, the selection gain is about five times the run-to-run std.

We also repeated distillation three times for both methods on the six 128² ImageNet-Subsets (new runs for DISC and DDiF under one protocol). Each cell is the mean ± std over the three runs, propagated across subsets as above. The 95 % confidence interval of every Δ excludes zero.

IPCDM DDiFDM DISCΔ DMDC DDiFDC DISCΔ DC
1.053.4 ± 0.454.9 ± 0.2+1.545.9 ± 0.350.5 ± 0.3+4.6
0.548.3 ± 0.351.5 ± 0.2+3.240.9 ± 0.346.9 ± 0.2+6.0
0.2543.1 ± 0.248.5 ± 0.1+5.437.0 ± 0.441.9 ± 0.3+5.0

The subset winners are those of the submitted table (DISC leads on 35 of 36 cells). At 256² (DM, IPC 1.0), one additional seed per method gives DISC 56.4 and DDiF 53.3. The non-monotonic DDiF curve is not run-to-run variation but an evaluation artifact (Q2).

AuthorsOpenReview ↗

Response to Reviewer g7ZR [2/2]

Q1: Compressed real images at the same byte budget. At each budget, every class gets the same number of bytes. JPEG sizes are measured file sizes (64², upsampled to 128² for evaluation). Labels are excluded from every budget. DM, six-subset means:

BudgetStored setInstances/classAccuracy
IPC 0.25JPEG 64², quality 30, random order52–7339.6
JPEG 64², quality 30, prototype-nearest49–8945.7
DISC (3-run mean)2548.5‾\underline{\text{48.5}}
DISC, 8-bit decoders10154.1
IPC 0.5JPEG 64², quality 30, random order105–14846.6
JPEG 64², quality 30, prototype-nearest98–17351.7‾\underline{\text{51.7}}
DISC (3-run mean)5051.5
DISC, 8-bit decoders20256.2
IPC 1.0JPEG 64², quality 30, random order211–29853.6
JPEG 64², quality 30, prototype-nearest200–33958.2‾\underline{\text{58.2}}
DISC (3-run mean)10154.9
DISC, 8-bit decoders40560.1

At a fixed byte budget, accuracy tends to grow with the number of stored instances (quality 30 was the best of 30 / 50 / 75 at IPC 1.0). With 8-bit decoders, DISC stores about 1.1–2.1 times as many instances as JPEG and is ahead at every budget. Only the decoders are quantized, after training, and the shared encoder stays at 32 bits and is charged in full. We will add the quantized-decoder results to the revision.

Q2: Why DDiF at IPC 0.9 and 0.4 falls below IPC 0.8 and 0.3. The dips come from the last mini-batch of the evaluation training. Fractional IPCs give image counts that leave a very short last batch at batch size 64 (the batch size in Appendix Table 8, used for both methods). A full search over the curve finds three such cells:

DM, 128²Images in the last batch (of 64)PaperRemainder-free batch
DDiF, IPC 0.9246.550.7
DDiF, IPC 0.4843.045.9
DISC, IPC 0.2843.746.0

Re-evaluating the same checkpoints with a batch size that divides the set removes the dips. DC shows the same pattern (DDiF 0.9: 40.4 → 44.8, DDiF 0.4: 37.1 → 40.1, DISC 0.2: 37.2 → 40.5, re-trained with the paper's code and seed). The main comparison tables (Tables 1 and 2) are unaffected. The affected cells appear only in the budget-scaling summary (Table 4) and in the appendix’s fine-grained IPC table and curve. After correction DISC still leads on average (IPC 0.9: 53.6 vs 50.7, IPC 0.4: 50.3 vs 45.9), and one subset changes winner (Meow at IPC 0.9: DDiF 43.2 vs DISC 40.8). We will use a remainder-free batch for these cells in the revision.

Q3: MmedM_{\text{med}} and the full training set. By default, MmedM_{\text{med}} equals the number of decoders per class (25 / 50 / 101 at IPC 0.25 / 0.5 / 1.0). Pretraining the encoder on the whole class (about 1,300 images per class) at IPC 1.0 gives 54.0 (one run), against 54.9 with 101 images (3-run mean), while pretraining takes about 15 times longer. We therefore keep the small set and will state this in the paper.

References

  • Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang. Dataset Distillation via Factorization. NeurIPS 2022.
  • Zhiwei Deng, Olga Russakovsky. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks. NeurIPS 2022.

Thank you for your detailed response. My other questions from the initial review have been addressed; only one comparison remains, which will determine my final score.

In the table for Q1, the JPEG baseline was evaluated only at a resolution of 64×64. Furthermore, file sizes were measured on a per-image basis, with each file including its own header. The 8-bit decoders allow DISC to store four times as many samples within the same byte budget, and the shared encoder is accounted for only once. The JPEG baseline should be granted the same consideration: for images of the same quality and dimensions, the quantization tables and standard Huffman tables in the file headers are identical; therefore, a single copy for the entire dataset should suffice and be counted only once.

Please provide the JPEG results for 32×32 and 48×48 resolutions using this accounting method. Specifically, use "prototype-nearest" for image selection, a quality setting of 30, and upsample the images to 128² for evaluation. Please report these results for IPC budgets of 0.25, 0.5, and 1.0, including the number of images stored per class and the standard deviation across the evaluation classifiers.

Additionally, please clarify whether the results for 8-bit DISC represent an average over multiple runs.

If 8-bit DISC still outperforms these JPEG configurations at some of these budgets, I will increase my score.

Reviewer_qEfdOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DISC, a storage-efficient dataset distillation framework that enhances the performance of data set distillation under extremely low storage budgets (Sub-1 IPC). It achieves this by combining shared implicit neural representations and a two-stage Coreset initialization strategy.

Critical strengths
  1. The research problem has practical significance. This paper focuses on the problem of dataset distillation under extremely low storage budgets, particularly in high-resolution image scenarios, which has certain research value.

  2. The experimental performance improvement is relatively stable. DISC outperforms DDiF across different IPC budgets, image resolutions, and distillation targets. It also demonstrates certain advantages in cross-architecture and cross-resolution generalization experiments.

  3. The ablation experiments are quite comprehensive. The paper analyzes the Coreset selection, Encoder sharing strategy, and Encoder freezing strategy, providing experimental support for the effectiveness of each component.

Critical weaknesses
  1. The method's innovation is limited. The core idea of this paper is largely based on existing work. DDiF (ICLR 2025) has already applied implicit neural representations to dataset distillation, while STRAINER (NeurIPS 2024) adopted a shared Encoder and instance-independent Decoder architecture. Additionally, works like HaBa and NSD have explored methods to improve dataset distillation efficiency by leveraging shared representations. The two-stage Coreset strategy proposed in this paper mainly combines a simple sample selection heuristic rule based on feature distance, thus the overall method's contribution is relatively diluted.

  2. Lack of comparison with recent related works. The experiments are mainly centered around DDiF, lacking comparisons with recent storage-efficient dataset distillation methods. Particularly, [Dataset Distillation as Data Compression: ... , ICCV 2025] directly studies the trade-off between accuracy and storage cost, is not included. The absence of such comparisons makes it difficult to fully verify the competitiveness of DISC relative to the latest research progress.

  3. Insufficient evaluation of actual storage efficiency. The paper defines the storage budget using the number of 32-bit parameters, rather than the actual compressed storage size. If comparisons could be made with quantized neural representations or image compression baselines under the same byte budget, it would more strongly support the paper's core conclusion on storage efficiency.

  4. Insufficient verification of robustness and scalability. The standard deviation reported in the paper mainly comes from different evaluation classifiers rather than multiple independent data distillation experiments. Moreover, the main experiments only use six ImageNet subsets with 10 classes each, thus the scalability of the method on larger datasets and the stability under different distillation random seeds have not been adequately verified.

Questions for authors

Refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qEfd [1/2]

We thank the reviewer for the detailed review and the insightful comments on related work and the practical storage efficiency of our method.

W1: Novelty relative to DDiF, STRAINER and prior factorization. DISC's gain is a compression gain: under the same parameter budget it stores about twice as many synthetic instances as DDiF. The contribution is the budget allocation that makes this possible: one encoder for the whole distilled set, charged once, plus a lightweight decoder per instance (IPC 0.5: 50 decoders per class against DDiF's 25 fields), with separate coreset sets for encoder pretraining and decoder warmup. NSD (Yang et al., ECCV 2024), which is MTT-based, reports 50.5 on the six 128² ImageNet-Subsets at IPC 1.0, against 54.9 for DISC under DM (3-run mean, ahead on 5 of 6 subsets). NSD combines shared spectrum tensors and kernel matrices pairwise, whereas DISC shares one coordinate-network encoder. We will cite and discuss HaBa (Liu et al., NeurIPS 2022), LinBa (Deng & Russakovsky, NeurIPS 2022) and NSD.

W2: Comparison with DD-RUO (Bao et al., ICCV 2025). To compare with DD-RUO, which is optimized for compression through quantization and entropy coding, we stored DISC's decoders in 8 bits (quantization only, without entropy coding), which fits four times as many decoders in the same budget. Six-subset means at 128² (DD-RUO values from its Table 3):

ObjectiveMethodIPC 0.25IPC 0.5IPC 1.0
Gradient matching (DC)DISC, 8-bit decoders49.952.957.7
GM-RUO, reported (≈ IPC 1.0)not reportednot reported49.5
Distribution matching (DM)DISC, 8-bit decoders54.156.260.1
DM-RUO, reported (≈ IPC 1.0)not reportednot reported59.1

DD-RUO is a coding scheme (quantized latents with a learned entropy model), which could also be applied on top of DISC's decoders. We will add this comparison and cite DD-RUO.

W3: Actual storage: quantized representations and compressed images. At each budget, every class gets the same number of bytes. JPEG sizes are measured file sizes (64², upsampled to 128² for evaluation). Labels are excluded from every budget. DM, six-subset means:

BudgetStored setInstances/classAccuracy
IPC 0.25JPEG 64², quality 30, random order52–7339.6
JPEG 64², quality 30, prototype-nearest49–8945.7
DISC (3-run mean)2548.5‾\underline{\text{48.5}}
DISC, 8-bit decoders10154.1
IPC 0.5JPEG 64², quality 30, random order105–14846.6
JPEG 64², quality 30, prototype-nearest98–17351.7‾\underline{\text{51.7}}
DISC (3-run mean)5051.5
DISC, 8-bit decoders20256.2
IPC 1.0JPEG 64², quality 30, random order211–29853.6
JPEG 64², quality 30, prototype-nearest200–33958.2‾\underline{\text{58.2}}
DISC (3-run mean)10154.9
DISC, 8-bit decoders40560.1

At a fixed byte budget, accuracy tends to grow with the number of stored instances (quality 30 was the best of 30 / 50 / 75 at IPC 1.0). With 8-bit decoders, DISC stores about 1.1–2.1 times as many instances as JPEG and is ahead at every budget. Only the decoders are quantized, after training, and the shared encoder stays at 32 bits and is charged in full. 8-bit storage applies to DDiF's fields as well. It would multiply both methods' instance counts by the same factor, so the shared encoder's roughly twofold instance advantage remains. We will add the quantized-decoder results to the revision.

AuthorsOpenReview ↗

Response to Reviewer qEfd [2/2]

W4: Distillation seeds and scale. We repeated distillation three times for both methods on the six 128² ImageNet-Subsets (new runs for DISC and DDiF under one protocol). Each cell is the mean ± std over the three runs, with the per-subset stds propagated to the six-subset mean as in the paper's tables. The 95 % confidence interval of every Δ excludes zero.

IPCDM DDiFDM DISCΔ DMDC DDiFDC DISCΔ DC
1.053.4 ± 0.454.9 ± 0.2+1.545.9 ± 0.350.5 ± 0.3+4.6
0.548.3 ± 0.351.5 ± 0.2+3.240.9 ± 0.346.9 ± 0.2+6.0
0.2543.1 ± 0.248.5 ± 0.1+5.437.0 ± 0.441.9 ± 0.3+5.0

The subset winners are those of the submitted table (DISC leads on 35 of 36 cells). At 256² (DM, IPC 1.0), one additional seed per method gives DISC 56.4 and DDiF 53.3.

For scale, we added ImageNet-100 (the paper's source and 128² resolution, ten times the classes) and CUB-200-2011 (fine-grained, non-ImageNet, 200 classes, 64², official split). DM, one run per cell. Vanilla DM stores pixel images, so it has only an IPC 1.0 cell:

DatasetIPCVanilla DMDDiFDISCΔ (DISC − DDiF)
ImageNet-100 · 128²0.25n/a15.919.4+3.5
0.5n/a19.921.9+2.0
1.05.823.922.4‾\underline{\text{22.4}}−1.5
CUB-200 · 64²0.25n/a8.011.9+3.9
0.5n/a12.517.4+4.9
1.03.718.4‾\underline{\text{18.4}}22.2+3.8

DISC improves over DDiF at all CUB-200 budgets and at ImageNet-100 IPC 0.25 and 0.5. At ImageNet-100 IPC 1.0, DDiF is ahead by 1.5 pp. For reference, training on the whole CUB-200 training set reaches 27.7 under the same evaluation (mean of the five evaluation classifiers), so DISC's IPC 1.0 set recovers 80 % of it.

The revision will separate evaluation-classifier variation from distillation-run variation and add both datasets.

References

  • Youneng Bao, Yiping Liu, Zhuo Chen, Yongsheng Liang, Mu Li, Kede Ma. Dataset Distillation as Data Compression: A Rate-Utility Perspective. ICCV 2025.
  • Shaolei Yang, Shen Cheng, Mingbo Hong, Haoqiang Fan, Xing Wei, Shuaicheng Liu. Neural Spectral Decomposition for Dataset Distillation. ECCV 2024.
  • Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang. Dataset Distillation via Factorization. NeurIPS 2022.
  • Zhiwei Deng, Olga Russakovsky. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks. NeurIPS 2022.
AuthorsOpenReview ↗

Update to the 8-bit DISC values and additional JPEG results (W2, W3)

Update of our tables for W2 and W3.

  • The 8-bit DISC values under DM at IPC 0.25 and 0.5 are now 54.8 and 57.6 (previously 54.1 and 56.2). Because of the limited time during the rebuttal period, the earlier runs did not use the same hyperparameter setting as the IPC 1.0 run (60.1, unchanged). The new values use the same setting at all three budgets and are means of two runs.
  • In the W3 table, the 8-bit DISC row at IPC 0.25 should read 100 decoders per class, not 101. This was a labelling error in the table only.

The DC values and the comparison with DD-RUO at IPC 1.0 are unchanged.

Additional JPEG results. Following Reviewer g7ZR's request, we also evaluated JPEG at 32² and 48² with the header charged once for the whole dataset. These results are in our reply to Reviewer g7ZR.

Reviewer_GSrEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DISC, which combines shared implicit neural representations with two-stage coreset initialization to accommodate more effective synthetic training samples under extremely limited storage budgets. Experiments on ImageNet subsets and CIFAR-10 demonstrate an improved trade-off between storage cost and classification accuracy in dataset distillation.

Critical strengths

1.Clear motivation. The paper addresses dataset distillation under extremely limited storage budgets, using a shared encoder to reduce the storage cost per synthetic sample.

2.Notable improvements under low budgets. DISC achieves higher classification accuracy than the main baselines across multiple extremely low-budget settings, improving the trade-off between storage cost and performance.

Critical weaknesses

1.Questionable topic categorization. This paper focuses on image dataset distillation, and all experiments concern visual classification tasks. Therefore, “applications to computer vision, audio, language, and other modalities” may be a more appropriate primary area than “other topics in machine learning.”

2.Lack of direct validation of coverage and representativeness. Although the results show that the selection strategy improves classification accuracy, they do not directly establish that median-distance samples better cover intra-class variation or that prototype-nearest samples are more representative. Additional experiments should separately validate the intra-class coverage of median-distance samples and the representativeness of prototype-nearest samples.

3.Limited dataset coverage. Although I am not an expert in this field, I believe this task warrants evaluation on a broader range of datasets. The experiments focus on six ImageNet subsets and CIFAR-10, leaving the evidence limited to two dataset sources and insufficiently convincing. Expanding the dataset coverage would provide stronger evidence of the method’s effectiveness.

Questions for authors

Given that the research and experiments focus on image dataset distillation, why did the authors select “other topics in machine learning” as the primary area rather than “applications to computer vision, audio, language, and other modalities”?

Beyond improvements in classification accuracy, can the authors provide direct quantitative analyses or visualizations to validate the coverage of intra-class variation by median-distance samples and the representativeness of prototype-nearest samples, respectively?

Have the authors evaluated the method on datasets beyond the ImageNet subsets and CIFAR-10? In particular, does the method retain its advantages with more classes or different data distributions? If such results are available, please include them to support the method’s broader effectiveness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GSrE

We thank the reviewer for the careful review and the helpful comments on the validation and generality of our method.

W2/Q2: Coverage and representativeness. The two rules do different jobs, and direct measurements confirm it. From median-distance selection we wanted two properties: to keep the class's variance as well as random selection does, and, unlike random selection, to include no outliers. Prototype-nearest picks the most representative images. Measured in the paper's feature space, 30 samples per class, averaged over the six subsets:

SelectionClass variance retained (random = 1.00)Selected samples in the class's farthest 10 %1-NN test accuracy
Median-distance (encoder pretraining)0.990 %63.8
Prototype-nearest (decoder warmup)0.680 %68.4
Random (seeds 0–2)1.0010.8 %65.4
Farthest from prototype1.21100 %43.3

Median-distance retains the same variance as random selection but takes no sample from the farthest 10 % of the class, whereas random selection places 10.8 % of its samples there. Over ten random draws per class, this share ranges from 0 to 30 %, and the 1-NN accuracy moves by up to 9 pp between seeds. As a point cover (mean distance from each class sample to its nearest selected sample), median-distance is close to random (0.777 vs 0.766). Its role is to drop the tail while keeping the spread, not to cover every point. Prototype-nearest gives the highest 1-NN accuracy (+3.0 pp over random): its images are the most representative of their class. (1-NN labels each test image with the class of its nearest selected sample, and no network is trained.) We will replace the "coverage" wording in the paper with this description and add the table and PCA plots of the selections to the appendix.

W3/Q3: More classes and other distributions. We added ImageNet-100 (the paper's source and 128² resolution, ten times the classes) and CUB-200-2011 (fine-grained, non-ImageNet, 200 classes, 64², official split). DM, one run per cell. Vanilla DM stores pixel images, so it has only an IPC 1.0 cell:

DatasetIPCVanilla DMDDiFDISCΔ (DISC − DDiF)
ImageNet-100 · 128²0.25n/a15.919.4+3.5
0.5n/a19.921.9+2.0
1.05.823.922.4‾\underline{\text{22.4}}−1.5
CUB-200 · 64²0.25n/a8.011.9+3.9
0.5n/a12.517.4+4.9
1.03.718.4‾\underline{\text{18.4}}22.2+3.8

DISC improves over DDiF at all CUB-200 budgets and at ImageNet-100 IPC 0.25 and 0.5. At ImageNet-100 IPC 1.0, DDiF is ahead by 1.5 pp. For reference, training on the whole CUB-200 training set reaches 27.7 under the same evaluation (mean of the five evaluation classifiers), so DISC's IPC 1.0 set recovers 80 % of it. We will add both datasets to the revision.

W1/Q1: Primary area. We followed the primary area of DDiF (ICLR 2025), our direct predecessor and main baseline.

Thank you for the response and additional experiments. I have one further question: prototype-nearest selection improves the average 1-NN accuracy over random selection by 3 percentage points, while random selection reportedly varies by up to 9 pp across seeds. This variation seems relatively large. Is these results support your claim?

AuthorsOpenReview ↗

Additional Response to Reviewer GSrE

Thank you for the follow-up question. The two numbers describe different things, and our earlier wording did not separate them clearly.

The 9 pp figure is the largest range we observed within a single subset (Meow) across ten random draws. The 3.0 pp figure is a gap between averages over the six subsets, which our earlier reply computed from the first three random draws. Over all ten draws, the gap is 3.2 pp.

To compare the two rules at the same level, we average the six subsets for each random draw and then take the mean and standard deviation over the ten draws:

1-NN test accuracy (%), 30 samples per classSix-subset average
Random, ten draws (mean ± std over draws)65.2 ± 0.7
Random, highest of the ten draws66.3
Prototype-nearest (deterministic for a fixed feature space)68.4

At this level, the variation across random draws is small (±0.7 pp), and prototype-nearest is above even the highest of the ten draws. Per subset, it is above all ten random draws on Nette, Woof and Fruit, above eight of them on Yellow and nine on Meow, and on par with the random average on Squawk.

So yes, these results support our claim: averaged over the six subsets, prototype-nearest selects samples that are more representative of their class than random selection does in the feature space used for selection, although the margin varies by subset. We will state this in the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global response to Reviewers

We thank the reviewers for their careful and constructive reviews. Several points were raised by more than one reviewer (more datasets, variation across distillation runs, storage measured in bytes, comparison with recent storage-efficient methods). We answer each point in the reviewer's own thread, so that every reply is self-contained. A revised PDF with the changes marked in red will be uploaded before the end of the discussion period.

AuthorsOpenReview ↗

Summary: how the reviewers' main concerns were addressed

We thank the reviewers and the AC for their time and constructive comments. The reviews shared a few main concerns. Below we summarize how the discussion and the revised PDF (changes in red) address each of them.

1. Is DISC actually storage-efficient in bytes? (qEfd, g7ZR) Both reviewers asked whether the parameter budget translates into real storage savings against compressed real images and recent compression-based distillation. With 8-bit decoders and no entropy coding, DISC is ahead on the six-subset average of JPEG-compressed real images at every budget, under the accounting g7ZR specified (header counted once, 32² and 48², quality 30). g7ZR had stated that this comparison would decide the final score, confirmed that the condition was met, and raised the score. DISC is also ahead of DD-RUO (ICCV 2025), which qEfd asked us to compare with, on the six-subset average at the budget DD-RUO reports, under both objectives. (Section 4.9, Appendix D.5)

2. Are the gains robust across distillation runs? (qEfd, g7ZR) We repeated distillation three times per cell for both DISC and DDiF under both objectives. The average margins over DDiF are +1.5 to +6.0pp, their 95 % confidence intervals exclude zero at every budget, and DISC leads on 35 of the 36 cells. The coreset-selection gains (+1.1 and +0.7pp) exceed the run-to-run standard deviation (0.2pp). The non-monotonic DDiF curve that raised this concern comes from a short last evaluation batch, and the revision corrects the affected cells. (Sections 4.2, 4.3, 4.7, Appendix C.4.1, D.1)

3. Does DISC generalize beyond ImageNet-Subsets? (GSrE, qEfd) On ImageNet-100 (ten times the classes) and CUB-200-2011 (fine-grained, outside ImageNet), DISC improves over DDiF on both datasets at IPC 0.25 and 0.5, by 2.0 to 4.9pp, and on CUB-200-2011 at IPC 1.0 as well. (Section 4.9, Appendix D.4)

4. Do the selection rules do what we claim? (GSrE) Measured directly in the feature space, median-distance selection keeps the class variance of random selection while taking no sample from the farthest 10 % of the class, and prototype-nearest selection gives the highest 1-NN accuracy, above the best of ten random draws. (Section 3.4, Appendix D.2)

5. Comparison with methods that share components across samples (qEfd, g7ZR) HaBa, LinBa and NSD also share components across synthetic samples. We now discuss them and compare with them, charging shared components to the budget as the reviewers suggested. DISC's design, one coordinate-network encoder charged once with a lightweight decoder per instance and coreset-based initialization, roughly doubles the number of stored instances at the same budget. LinBa at IPC 0.5 stays well below DISC (35.1 vs. 51.5). HaBa, which needs at least IPC 1.0, is below DISC at IPC 0.25 (41.4 vs. 48.5), and NSD is below DISC at the same IPC 1 budget (50.5 vs. 54.9). (Section 2, Appendix D.6)

We believe these results resolve the main concerns raised in the reviews. The updated 8-bit DISC values and the decoder-count correction are posted in the threads of g7ZR and qEfd, and the revision uses them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.