acceptodds
Under review as a conference paper at ICLR 2027

Patch-Level Codebook Parameterization for Dataset Distillation

Abstract

Dataset distillation aims to synthesize a compact dataset that retains the essential knowledge of a large-scale dataset, enabling efficient model training while significantly reducing storage and computational costs. Recent parameterization methods enhance storage efficiency through compact or shared representations, but recurring local structures may still be redundantly encoded across synthetic images, limiting storage efficiency. In this paper, we propose to parameterize a synthetic dataset using patch-level learnable codebooks and patch index maps, such that each synthetic image is generated by assembling learned representative patches according to the index map. By explicitly sharing patch-level representations across synthetic images, our method reduces repeated encoding of recurring local structures and improves the use of limited storage budgets. Furthermore, we employ multiple codebooks at different stages and combine their patches additively to further improve representation power. Extensive experiments on standard benchmarks demonstrate that our method achieves state-of-the-art performance across various dataset distillation methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dataset Distillation, Parameterization for Dataset Distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jc9eOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a parameterization for dataset distillation in which synthetic images are assembled from globally shared, learnable patch codebooks according to per-image discrete index maps. Patches are stored as DCT coefficients. Index maps are optimized through a low-rank factorization with Gumbel-Softmax and stored as integer indices after training. The authors further stack several codebooks additively in a multi-stage design to increase the number of patch combinations per location under a fixed storage budget.

Critical strengths
  1. The core idea is simple and well motivated. Natural images contain many repeated local structures, and storing them once in a shared patch codebook is an intuitive way to avoid encoding the same content repeatedly.
  2. The method can plug into GM, DM, TM.
  3. The gains are large and consistent in Table 1 and 2.
  4. The paper ablates each main design choice.
Critical weaknesses
  1. The novelty over existing shared-component methods is not clearly established. SPEED (Wei et al., 2023, in line 633) already learns a shared dictionary of epitomes and composes each synthetic image from sparse selections of it, and HaBa and RTP also share bases across images.
  2. The DCT motivation is weak for the patch sizes actually used. The frequency principle concerns whole networks and images, but a 4×4 patch has only 16 coefficients, so it is unclear what meaningful low-frequency bias this representation introduces at that scale.
  3. It remains unclear whether the gains come from better images or simply from more images. In Table 5, the method is slightly below plain TM on CIFAR-10 (71.3 vs 71.6) and below SRe2L on ImageNet-1K (20.8 vs 21.3). This suggests most of the improvement comes from the larger TPC, which weakens the claim that shared patches improve optimization.
  4. Distillation cost is not reported. The paper measures decoding time but not the training time or memory of the Gumbel-based index optimization.
  5. Standard deviations are missing in most tables.
  6. In lines 349 to 351, the paper attributes the larger gains on ImageNet subsets to higher resolution containing more local redundancy. However, larger images also contain proportionally more non-redundant content, so the fraction of redundancy may not increase with resolution. A more direct explanation is that the ImageNet setting uses a larger stride, which shrinks the index maps and allows many more synthetic images under the same budget.
Questions for authors
  1. Section 3.2 states that the low-rank dimension satisfies d ≪ |P|, but the default settings use d=16 or d=32 with codebook sizes as small as 16. In that regime the factorization is not low-rank at all, so what role does the expander E actually play, and why is it still needed?
5 comments
AuthorsOpenReview ↗

Response for reviewer jc9e (1/3)

W1. Novelty over existing shared-component methods

We thank the reviewer for this comment. HaBa, RTP, SPEED, and our method all construct synthetic images from components shared across images. The two most important differences lie in the sharing unit and in how each local region of a synthetic image is parameterized.

HaBaRTPSPEEDOurs
Sharing unitWhole imageWhole imageLocal regionLocal region
Shared componentBase images + networksBase imagesFeature tokens + decoderCodebook patches
ParameterizationNetworkWeighted sumWeighted sum + decoderSelection

HaBa and RTP share whole-image bases, so a local pattern recurring at different positions is not shared. SPEED shares at the local level, but it mixes the shared feature tokens with real-valued weights assigned to each local region and passes them through a learned decoder, so multiple real-valued weights should be stored at each local region. Our method also shares at the local level, but selects one of the codebook patches for each local region at each stage, spending only a small storage budget of a single index.

The benefit of local sharing is shown in Supp. Table 19, where accuracy decreases as the patch size approaches the image size. To support the benefit of selection, we replaced the selection scheme in our method with real-valued weighting over the codebook patches (CIFAR-10, TPC=50).

CompositionBudgetAcc.
Weighted sum13.7470.9
Selection (ours)0.1171.3

The weighted sum requires 125$\times$ more storage space than ours but does not improve accuracy.

Furthermore, unlike SPEED, our method needs no decoder. SPEED passes every local region through a learned decoder to synthesize an image, whereas our method composes a synthetic image by selecting the codebook patches, which requires 24 to 1771$\times$ fewer FLOPs per image than SPEED. The total is computed over all synthetic images under the same storage budget, where our method synthesizes more images than SPEED.

MetricMethodCIFAR-10 IPC=1CIFAR-10 IPC=10CIFAR-10 IPC=50CIFAR-100 IPC=1CIFAR-100 IPC=10CIFAR-100 IPC=50
MFLOPs per imageSPEED0.5934.13843.7027.68643.70243.838
Ours0.0250.0250.0250.0250.0350.035
Total GFLOPsSPEED0.1015.131118.8708.455270.954438.382
Ours0.0120.1230.3700.1230.8871.419

Under the same storage budget, our method also outperforms SPEED in all settings of Table 1.

In the revised paper, we clarified this difference in the related work and provided the comparative experimental results in Tables 9 and 12.

W2. Motivation of DCT

Small patches in natural images are usually dominated by low-frequency contents, for example, in CIFAR-10, 98.6% of the energy of 4 ×\times 4 image patches lies in the three lowest-frequency coefficients among the 16 DCT coefficients. We therefore represent codebook patches in the DCT domain to encourage low-frequency bias. To support this, we conducted a simple image-fitting experiment. Starting from an identical initialization, we fit real images with the pixel and DCT parameterizations, respectively, under the setting of our codebook initialization stage. We then split each image into 4 ×\times 4 patches and measured the PSNR on the patches (Full) and on their low-frequency components (Low-freq.) by considering the three lowest-frequency DCT coefficients only. Results are averaged over 5 independent runs.

DatasetPSNRParam.250100020005000
CIFAR-10FullPixel22.6123.0422.4421.50
CIFAR-10FullDCT22.6423.0823.2623.05
CIFAR-10Low-freq.Pixel23.7024.0523.2822.18
CIFAR-10Low-freq.DCT23.7324.1224.3124.01
CIFAR-100FullPixel21.1522.0922.0222.32
CIFAR-100FullDCT21.5022.6922.7522.71
CIFAR-100Low-freq.Pixel22.0923.0322.9223.24
CIFAR-100Low-freq.DCT22.5423.7923.8223.72

The DCT parameterization reconstructs the original images more accurately throughout training. At every iteration the gap is larger in low-frequency PSNR than in full PSNR, which indicates that the gain is concentrated in the low-frequency components.

AuthorsOpenReview ↗

Response for reviewer jc9e (2/3)

W3. Gains from better images or more images

Thanks for this insightful comment. The goal of parameterization in DD is to represent a synthetic dataset that preserves as much training signal as possible within a given storage budget. In the matched-TPC comparison in Table 5, TM and SRe$^2$L directly optimize raw pixels without imposing any constraints on the image space. Their performance therefore serves as a reference upper bound for what parameterization methods can achieve at the same TPC. Thus, a parameterization method should be evaluated by how closely it approaches this bound, rather than whether it surpasses it. We further extend our evaluation to ImageNet-1K at TPC=50 and include a comparison with FreD.

TPCMethodBudget (MB)Acc.
10SRe$^2$L5742.221.3
+FreD234.46.2
+DDiF134.414.6
+Ours45.620.8
50SRe$^2$L28710.946.8
+FreD1171.922.5
+DDiF672.042.6
+Ours228.145.5

Our method stays within 0.5 and 1.3 % of SRe$^2$L at TPC 10 and 50. In contrast, DDiF drops by 6.7 and 4.2 %, and FreD by 15.1 and 24.3 %. Our method also requires the least storage among the parameterizations in all settings. These results show that our gain comes not only from using more images but also from preserving more training signal per image than existing parameterizations.

W4. Computational cost of distillation.

According to the reviewer's suggestion, we compared the GPU memory usage, distillation time, and synthesis time across DDiF, RUO, and our method using TM at IPC=1 on two NVIDIA RTX PRO 6000 GPUs. Each method uses the hyperparameters specified in its original paper, with TPC determined by the IPC=1 budget. GPU memory usage (in GB) is measured as the sum of the peak memory usage on the two GPUs. Synthesis time (in ms) denotes the time required to generate synthetic images from the stored parameters. Distillation time (in hours) denotes the total time required to optimize the synthetic dataset, including Gumbel-based index optimization in our method.

DatasetMethodGPU Memory (GB)Distillation Time (h)Synthesis Time (ms)
CIFAR-10DDiF12.96.231.71
CIFAR-10RUO36.46.271.46
CIFAR-10Ours44.24.30.81
CIFAR-100DDiF29.513.9181.29
CIFAR-100RUO45.016.0770.00
CIFAR-100Ours44.06.76.69

Our method achieves the shortest distillation and synthesis times on both CIFAR-10 and CIFAR-100, while maintaining nearly constant GPU memory usage across the two datasets. We have discussed these results in the last paragraph of Section 4.2 and incorporated them into Tables 8 and 9 of the revised manuscript, consolidating them with the synthesis time comparison against RUO previously reported in Table 20 of the original supplementary material.

W5. Missing standard deviations

All results of our method reported in the original manuscript were already averaged over five independent runs. In the revised manuscript, we have additionally reported the corresponding standard deviations in the main experimental results (Tables 2, 3, 4, and 6).

AuthorsOpenReview ↗

Response for reviewer jc9e (3/3)

W6. Effect of larger stride and TPC on ImageNet subsets

Thanks for this valuable comment. To examine whether the gain comes from the larger TPC enabled by a larger stride, we varied the patch size and stride on ImageNette. We kept the storage budget within IPC 1 and all other hyperparameters fixed.

Patch/StrideGridTPCBudgetAccuracy
4/263 ×\times 63321.8472.1
8/431 ×\times 311281.8580.8
16/815 ×\times 154661.8779.6

A larger TPC does not necessarily improve accuracy. Increasing the stride from 2 to 4 raises TPC from 32 to 128 and improves accuracy by 8.7 %. Increasing it further to 8, however, lowers accuracy to 79.6% despite about 3.6$\times$ more TPC. Smaller patches represent local structures more finely but leave fewer images within the budget, whereas larger patches allow more images but represent local structures more coarsely. Thus, TPC alone does not explain the gain, and the performance is determined by the balance between the number of synthetic images and the granularity of the shared patches.

In the revised manuscript, we revised our claims and added these results to the supplementary material (Supp. Table 20).

Q1. Role of the expander E\boldsymbol{E}

The condition d≪∣P∣d \ll |\mathcal{P}| in Section 3.2 describes the single-stage case. In the multi-stage setting (Section 3.3), E\mathbf{E} expands dd to L∣P∣L|\mathcal{P}| rather than ∣P∣|\mathcal{P}|. For example, with L=2L=2 and ∣P∣=16|\mathcal{P}|=16, d=16d=16 is expanded to 32, so the factorization remains low-rank in all our settings. We clarified this in Sections 3.2 and 3.3 of the revised version.

In addition, to examine the role of E\mathbf{E}, we compare our method with a variant that directly learns the selection scores of each image without E\mathbf{E} on CIFAR-100 at IPC=1. Params denotes the size of the parameters optimized during distillation. After distillation, only the selected patch indices are stored, so both have the same storage cost.

MethodParams (MB)Acc.
Ours68.6745.8
w/o E\mathbf{E}137.3345.2

With E\mathbf{E}, the learnable parameters are halved, and accuracy is slightly higher.

Reviewer_DL7POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new parameterization method for dataset distillation to reduce redundancies caused by the recurrence of local structures within and across images. Instead of storing pixels, each synthetic image is assembled from patches of globally shared, learnable codebooks according to a per-image patch index map. Codebook patches are represented as DCT coefficients, and the index maps are produced from low-rank logits (a per-image tensor multiplied by a shared expander), with the Gumbel-Softmax trick to enable end-to-end training. The codebook consists of multiple stages, and the codes from stages are summed, which increases the possible number of patch combinations. Because each image is cheap to store, many more training images per class (TPC) fit within the same storage budget represented as IPC. The parameterization is agnostic to the distillation objective and is combined with GM, DM, TM, and SRe2L, with evaluations on CIFAR-10/100, ImageNet subsets, and ImageNet-1K.

Critical strengths

The approach is simple yet effective, showing large and consistent gains throughout the experiments (e.g., 67.0% vs. 62.7% for RUO on ImageNet subsets at IPC=1 with TM).

Critical weaknesses
  • Novelty relative to shared-dictionary methods needs sharper positioning. Especially, in my understanding, SPEED also parameterizes synthetic images through patch-level encoding. Although SPEED introduces this parameterization in the latent space of a decoder network, it still produces each patch with a codebook and combines them to generate synthetic images. The paper should state precisely what is new.

  • Storage accounting on ImageNet-1K is unclear. In SRe2L-style training, soft labels for every image are stored or regenerated, and since many more images per class are stored at IPC=1, the label cost grows accordingly. It is unclear whether soft labels count toward the budget. D3S (“Large Scale Dataset Distillation with Domain Shift”, ICML 2024) shows that training with more iterations, and hence more soft labels, improves performance. This factor should therefore be controlled to ensure a fair comparison across methods.

Questions for authors
  • What is the most important factor that distinguishes the proposed method from SPEED?

  • In ImageNet-1k experiments, was the number of iterations to train networks fixed to match the storage budget induced by storing soft labels?

  • (Minor) I recommend adding citations to tables. Especially for Table 1, HMN is not even in the paper's references.

3 comments
AuthorsOpenReview ↗

Response for reviewer DL7P

W1/Q1. Differences from SPEED.

We thank the reviewer for this comment. SPEED and our method construct synthetic images from components shared across images like previous sharing-based parameterizations such as HaBa, RTP, and NSD. However, the key difference from SPEED is how each local region of a synthetic image is parameterized. Our method selects one of the codebook patches for each local region at each stage, spending only a small storage budget of a single index. In contrast, SPEED mixes the shared feature tokens with real-valued weights assigned to each local region and passes them through a learned decoder, so multiple real-valued weights should be stored at each local region. In order to support the benefit of our method over SPEED, we first compared the performance by replacing the selection scheme in our method with real-valued weighting over the codebook patches (CIFAR-10, TPC=50).

CompositionBudgetAcc.
Weighted sum13.7470.9
Selection (ours)0.1171.3

The weighted sum requires 125$\times$ more storage space than ours but does not improve accuracy. Storing recurring patterns at each location does not contribute to accuracy, and enforcing their reuse parameterizes synthetic images more efficiently.

Furthermore, our method needs no decoder. A synthetic image is composed by selecting the codebook patches, which requires 24 to 1771$\times$ fewer FLOPs per image than SPEED. The total is computed over all synthetic images under the same storage budget, where our method synthesizes more images than SPEED.

MetricMethodCIFAR-10 IPC=1CIFAR-10 IPC=10CIFAR-10 IPC=50CIFAR-100 IPC=1CIFAR-100 IPC=10CIFAR-100 IPC=50
MFLOPs per imageSPEED0.5934.13843.7027.68643.70243.838
Ours0.0250.0250.0250.0250.0350.035
Total GFLOPsSPEED0.1015.131118.8708.455270.954438.382
Ours0.0120.1230.3700.1230.8871.419

Under the same storage budget, our method also outperforms SPEED in all settings of Table 1.

In the revised paper, we clarified the difference of our method from SPEED in the related work and provided the comparative experimental results in Tables 9 and 12.

W2/Q2. Storage budget of soft labels.

On ImageNet-1K, we follow the evaluation protocol of SRe$^2$L, where soft labels are not stored but generated by the teacher model at each iteration. Thus, the associated storage cost is that of the teacher model, which is identical across all methods and IPC settings. Accordingly, the storage budget in the experiment of Table 6 excludes soft labels.

However, the number of iterations is not fixed in Table 6. A method with more synthetic images is trained for more iterations and receives more soft labels. As the reviewer pointed out, we conducted two experiments on ImageNet-1K to control this issue.

(1) Matched TPC (ImageNet-1K). All methods use the same number of synthetic images and are trained under a fixed number of iterations, so they receive the same number of soft labels.

TPCMethodBudget (MB)IterationsSoft labels (10610^6)Acc.
10SRe$^2$L5742.23,0003.021.3
10+FreD234.43,0003.06.2
10+DDiF134.43,0003.014.6
10+Ours45.63,0003.020.8
50SRe$^2$L28710.914,70015.046.8
50+FreD1171.914,70015.022.5
50+DDiF672.014,70015.042.6
50+Ours228.114,70015.045.5

(2) Similar numbers of iterations (IPC=10). Each method uses its synthetic images under the IPC=10 budget. We use comparable numbers of training iterations across methods to ensure that they are trained with approximately the same number of soft labels.

MethodBudget (MB)IterationsSoft labels (10610^6)Acc.
SRe$^2$L5742.23,0003.021.3
+FreD5625.03,0553.17.1
+DDiF5644.53,2883.421.1
+Ours4562.02,9313.025.8

Our method achieves the highest accuracy among all parameterization methods in both additional experimental settings.

Q3. Citations.

We thank the reviewer for pointing this out. We have added the citations to the tables and the missing reference for HMN.

I thank the authors for their rebuttal. My concerns have been adequately addressed. I recommend that the authors add the experimental results regarding a fixed number of iterations under the same IPC to the paper. I will maintain my score and recommend acceptance.

We sincerely thank the reviewer for recommending acceptance. As recommended, we have added the experimental results with comparable numbers of iterations under the same IPC to Table 24 in Appendix B.8 (Comparable Number of Soft Labels) of the revised paper.

Reviewer_XuxmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a patch-level codebook parameterization method for dataset distillation. The method aims to reduce redundant storage of local visual patterns across synthetic images. Specifically, the authors introduce globally shared learnable codebooks and discrete patch index maps to construct synthetic images. A multi-stage codebook framework is further introduced to improve the representation capacity under limited storage budgets. The proposed method can be combined with different dataset distillation objectives, including GM, DM, TM, and SRe2L. Experimental results on CIFAR-10/100, ImageNet subsets, and ImageNet-1K demonstrate promising performance and storage efficiency.

Critical strengths
  1. The paper addresses a meaningful problem by exploring patch-level redundancy to improve the storage efficiency of dataset distillation.
  2. The proposed framework is simple and flexible, and the shared codebook design can be integrated with different dataset distillation methods.
Critical weaknesses
  1. The effectiveness of patch-level sharing could be further analyzed. The paper mainly uses classification accuracy and storage reduction to demonstrate the benefits of shared codebooks. However, there is limited quantitative analysis of how frequently codebook patches are reused across different images and classes. Such analysis would help better support the motivation of reducing redundant local representations.
  2. The computational cost during dataset distillation needs further evaluation. Although the paper provides an analysis of storage efficiency and synthetic image generation time, the proposed method requires optimizing shared codebooks and learnable index maps. It would be helpful to report the total distillation time and peak GPU memory usage compared with existing parameterization methods.
  3. The comparisons with generative model-based dataset distillation methods could be more comprehensive. Although the paper includes GLaD and H-GLaD, it lacks comparisons with recent diffusion-based dataset distillation methods. Adding these baselines under comparable storage budgets would further strengthen the experimental evaluation.
  4. The structural consistency of synthetic images generated by patch combinations needs further investigation. Although the proposed method uses real-image initialization and overlapping patch assembly, it is unclear whether combining patches from shared codebooks can preserve the global structure and semantic consistency of natural images. The authors should further explain how the method avoids unreasonable patch combinations and provide more quantitative or qualitative analysis to demonstrate the structural consistency of the generated images.
Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response for reviewer Xuxm (1/2)

W1. Quantitative analysis of patch usage.

We thank the reviewer for this insightful comment. In the table below, we report the selection ratio of each codebook patch, along with the minimum and maximum selection ratios across 500 images and across 10 classes at two respective stages, under the CIFAR-10 setting (IPC=1, L=2L=2, ∣P∣=16|\boldsymbol{\mathcal{P}}|=16).

Patch indexStage 1, Overall (%)Stage 1, across images, min-max (%)Stage 1, across classes, min-max (%)Stage 2, Overall (%)Stage 2, across images, min-max (%)Stage 2, across classes, min-max (%)
13.70.0-11.12.0-5.511.82.2-48.47.8-21.6
24.20.0-11.12.0-6.38.51.3-20.46.7-10.5
32.40.0-10.21.6-3.38.31.3-15.66.6-10.0
47.30.0-30.22.9-10.91.00.0-16.90.1-2.4
52.80.0-9.81.6-4.04.80.4-12.43.3-5.8
611.21.3-47.66.1-16.72.00.0-8.40.8-3.3
76.30.4-15.15.3-7.52.10.0-8.41.4-3.0
89.01.8-26.76.4-12.09.01.8-19.67.7-11.0
95.40.9-13.84.1-6.87.60.4-21.85.9-8.7
108.70.9-34.25.3-15.77.90.4-19.15.2-11.1
111.90.0-9.30.9-2.34.60.4-11.63.6-5.8
125.40.0-24.03.0-8.55.20.0-21.33.1-8.2
138.80.9-16.97.7-10.12.50.0-7.61.8-3.2
144.90.4-12.02.9-6.46.10.0-45.81.5-12.9
1512.81.3-30.29.4-15.33.40.0-11.62.6-4.3
165.20.0-34.21.5-9.415.24.0-38.711.5-18.0

All patches are utilized, with an average selection ratio of 6.25 % and standard deviations of 3.1 and 3.7 % at Stages 1 and 2, respectively, indicating that no single patch dominates the selection. As shown in Figure 3, the most frequently selected patches mostly encode smooth color regions, while the least frequently selected ones mostly capture structural patterns, such as edges. More than half of the patches are used in every image, and every patch is used in every class. This indicates that local structures are not specific to individual images or classes but recur widely across images, both within and across classes, which supports our motivation. In the revised paper, we added the selection ratio of each codebook patch to Figure 3 and discussed the results in Section 4.3.

W2. Further evaluation of computational cost.

According to the reviewer's suggestion, we compared the GPU memory usage, distillation time, and synthesis time across DDiF, RUO, and our method using TM at IPC=1 on two NVIDIA RTX PRO 6000 GPUs. Each method uses the hyperparameters specified in its original paper, with TPC determined by the IPC=1 budget. GPU memory usage (in GB) is measured as the sum of the peak memory usage on the two GPUs. Synthesis time (in ms) denotes the time required to generate synthetic images from the stored parameters. Distillation time (in hours) denotes the total time required to optimize the synthetic dataset, including Gumbel-based index optimization in our method.

DatasetMethodGPU Memory (GB)Distillation Time (h)Synthesis Time (ms)
CIFAR-10DDiF12.96.231.71
CIFAR-10RUO36.46.271.46
CIFAR-10Ours44.24.30.81
CIFAR-100DDiF29.513.9181.29
CIFAR-100RUO45.016.0770.00
CIFAR-100Ours44.06.76.69

Our method achieves the shortest distillation and synthesis times on both CIFAR-10 and CIFAR-100, while maintaining nearly constant GPU memory usage across the two datasets. We have discussed these results in the last paragraph of Section 4.2 and incorporated them into Tables 8 and 9 of the revised manuscript, consolidating them with the synthesis time comparison against RUO previously reported in Table 20 of the original supplementary material.

AuthorsOpenReview ↗

Response for reviewer Xuxm (2/2)

W3. Comparison with diffusion-based method.

According to the reviewer's suggestion, we newly compared LD3M [1], which extends GLaD by optimizing latent codes of a pretrained latent diffusion model, in Table 4.

DatasetGLaDH-GLaDLD3MOurs
Nette30.430.832.067.4
Woof17.117.419.940.9
Fruit21.121.521.444.5
Yellow–––62.2
Meow19.620.122.143.1
Squawk28.228.830.465.2
Average23.323.725.253.9

LD3M generally improves over the GAN-based methods GLaD and H-GLaD, but its performance remains substantially lower than ours on all subsets, even though our method does not rely on a pretrained generator.

[1] Moser, Brian, et al. "Unlocking dataset distillation with diffusion models." Advances in Neural Information Processing Systems 38 (2025): 72581-72614.

W4. Structural consistency of synthetic images.

We respectfully note that the goal of DD is not to generate natural images but to preserve the training signal of the original dataset, such as the feature distribution, gradients, or training trajectories. Accordingly, our patch combinations are selected by their contribution to the distillation objective rather than by visual naturalness. Therefore, combinations that do not help preserve the training signal are discouraged during optimization, without any explicit constraint on visual naturalness.

Moreover, the structural consistency of our synthetic images does not come from real-image initialization. When initialized from random noise, our method still achieves 70.3 %, close to the 71.3 % with real-image initialization (Supp. Table 22), and the resulting images show class-consistent global structures similar to those from real-image initialization (Supp. Fig. 5). This indicates that the structural and semantic consistency of the patch compositions emerges from optimizing the distillation objective rather than from the initialization.

AuthorsOpenReview ↗

Kind reminder regarding our response

Dear Reviewer Xuxm,

As the discussion period is approaching its end, we just wanted to politely check whether our rebuttal has addressed your concerns, or if there is anything we need to further clarify.

Comments on the paper

1 comment

Dear reviewers,

The authors have submitted their response. Please carefully check whether the new content has addressed your concerns, particularly novelty against previous methods, modular effectiveness, and storage/compute consumption. If not, please follow up with the authors ASAP so that the authors can further respond before the discussion period ends (Nov 18).

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.