- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a parameterization for dataset distillation in which synthetic images are assembled from globally shared, learnable patch codebooks according to per-image discrete index maps. Patches are stored as DCT coefficients. Index maps are optimized through a low-rank factorization with Gumbel-Softmax and stored as integer indices after training. The authors further stack several codebooks additively in a multi-stage design to increase the number of patch combinations per location under a fixed storage budget.
Critical strengths
- The core idea is simple and well motivated. Natural images contain many repeated local structures, and storing them once in a shared patch codebook is an intuitive way to avoid encoding the same content repeatedly.
- The method can plug into GM, DM, TM.
- The gains are large and consistent in Table 1 and 2.
- The paper ablates each main design choice.
Critical weaknesses
- The novelty over existing shared-component methods is not clearly established. SPEED (Wei et al., 2023, in line 633) already learns a shared dictionary of epitomes and composes each synthetic image from sparse selections of it, and HaBa and RTP also share bases across images.
- The DCT motivation is weak for the patch sizes actually used. The frequency principle concerns whole networks and images, but a 4×4 patch has only 16 coefficients, so it is unclear what meaningful low-frequency bias this representation introduces at that scale.
- It remains unclear whether the gains come from better images or simply from more images. In Table 5, the method is slightly below plain TM on CIFAR-10 (71.3 vs 71.6) and below SRe2L on ImageNet-1K (20.8 vs 21.3). This suggests most of the improvement comes from the larger TPC, which weakens the claim that shared patches improve optimization.
- Distillation cost is not reported. The paper measures decoding time but not the training time or memory of the Gumbel-based index optimization.
- Standard deviations are missing in most tables.
- In lines 349 to 351, the paper attributes the larger gains on ImageNet subsets to higher resolution containing more local redundancy. However, larger images also contain proportionally more non-redundant content, so the fraction of redundancy may not increase with resolution. A more direct explanation is that the ImageNet setting uses a larger stride, which shrinks the index maps and allows many more synthetic images under the same budget.
Questions for authors
- Section 3.2 states that the low-rank dimension satisfies d ≪ |P|, but the default settings use d=16 or d=32 with codebook sizes as small as 16. In that regime the factorization is not low-rank at all, so what role does the expander E actually play, and why is it still needed?
5 comments
Response for reviewer jc9e (2/3)
W3. Gains from better images or more images
Thanks for this insightful comment. The goal of parameterization in DD is to represent a synthetic dataset that preserves as much training signal as possible within a given storage budget. In the matched-TPC comparison in Table 5, TM and SRe$^2$L directly optimize raw pixels without imposing any constraints on the image space. Their performance therefore serves as a reference upper bound for what parameterization methods can achieve at the same TPC. Thus, a parameterization method should be evaluated by how closely it approaches this bound, rather than whether it surpasses it. We further extend our evaluation to ImageNet-1K at TPC=50 and include a comparison with FreD.
| TPC | Method | Budget (MB) | Acc. |
|---|---|---|---|
| 10 | SRe$^2$L | 5742.2 | 21.3 |
| +FreD | 234.4 | 6.2 | |
| +DDiF | 134.4 | 14.6 | |
| +Ours | 45.6 | 20.8 | |
| 50 | SRe$^2$L | 28710.9 | 46.8 |
| +FreD | 1171.9 | 22.5 | |
| +DDiF | 672.0 | 42.6 | |
| +Ours | 228.1 | 45.5 |
Our method stays within 0.5 and 1.3 % of SRe$^2$L at TPC 10 and 50. In contrast, DDiF drops by 6.7 and 4.2 %, and FreD by 15.1 and 24.3 %. Our method also requires the least storage among the parameterizations in all settings. These results show that our gain comes not only from using more images but also from preserving more training signal per image than existing parameterizations.
W4. Computational cost of distillation.
According to the reviewer's suggestion, we compared the GPU memory usage, distillation time, and synthesis time across DDiF, RUO, and our method using TM at IPC=1 on two NVIDIA RTX PRO 6000 GPUs. Each method uses the hyperparameters specified in its original paper, with TPC determined by the IPC=1 budget. GPU memory usage (in GB) is measured as the sum of the peak memory usage on the two GPUs. Synthesis time (in ms) denotes the time required to generate synthetic images from the stored parameters. Distillation time (in hours) denotes the total time required to optimize the synthetic dataset, including Gumbel-based index optimization in our method.
| Dataset | Method | GPU Memory (GB) | Distillation Time (h) | Synthesis Time (ms) |
|---|---|---|---|---|
| CIFAR-10 | DDiF | 12.9 | 6.2 | 31.71 |
| CIFAR-10 | RUO | 36.4 | 6.2 | 71.46 |
| CIFAR-10 | Ours | 44.2 | 4.3 | 0.81 |
| CIFAR-100 | DDiF | 29.5 | 13.9 | 181.29 |
| CIFAR-100 | RUO | 45.0 | 16.0 | 770.00 |
| CIFAR-100 | Ours | 44.0 | 6.7 | 6.69 |
Our method achieves the shortest distillation and synthesis times on both CIFAR-10 and CIFAR-100, while maintaining nearly constant GPU memory usage across the two datasets. We have discussed these results in the last paragraph of Section 4.2 and incorporated them into Tables 8 and 9 of the revised manuscript, consolidating them with the synthesis time comparison against RUO previously reported in Table 20 of the original supplementary material.
W5. Missing standard deviations
All results of our method reported in the original manuscript were already averaged over five independent runs. In the revised manuscript, we have additionally reported the corresponding standard deviations in the main experimental results (Tables 2, 3, 4, and 6).
Response for reviewer jc9e (3/3)
W6. Effect of larger stride and TPC on ImageNet subsets
Thanks for this valuable comment. To examine whether the gain comes from the larger TPC enabled by a larger stride, we varied the patch size and stride on ImageNette. We kept the storage budget within IPC 1 and all other hyperparameters fixed.
| Patch/Stride | Grid | TPC | Budget | Accuracy |
|---|---|---|---|---|
| 4/2 | 63 63 | 32 | 1.84 | 72.1 |
| 8/4 | 31 31 | 128 | 1.85 | 80.8 |
| 16/8 | 15 15 | 466 | 1.87 | 79.6 |
A larger TPC does not necessarily improve accuracy. Increasing the stride from 2 to 4 raises TPC from 32 to 128 and improves accuracy by 8.7 %. Increasing it further to 8, however, lowers accuracy to 79.6% despite about 3.6$\times$ more TPC. Smaller patches represent local structures more finely but leave fewer images within the budget, whereas larger patches allow more images but represent local structures more coarsely. Thus, TPC alone does not explain the gain, and the performance is determined by the balance between the number of synthetic images and the granularity of the shared patches.
In the revised manuscript, we revised our claims and added these results to the supplementary material (Supp. Table 20).
Q1. Role of the expander
The condition in Section 3.2 describes the single-stage case. In the multi-stage setting (Section 3.3), expands to rather than . For example, with and , is expanded to 32, so the factorization remains low-rank in all our settings. We clarified this in Sections 3.2 and 3.3 of the revised version.
In addition, to examine the role of , we compare our method with a variant that directly learns the selection scores of each image without on CIFAR-100 at IPC=1. Params denotes the size of the parameters optimized during distillation. After distillation, only the selected patch indices are stored, so both have the same storage cost.
| Method | Params (MB) | Acc. |
|---|---|---|
| Ours | 68.67 | 45.8 |
| w/o | 137.33 | 45.2 |
With , the learnable parameters are halved, and accuracy is slightly higher.
Response for reviewer jc9e (1/3)
W1. Novelty over existing shared-component methods
We thank the reviewer for this comment. HaBa, RTP, SPEED, and our method all construct synthetic images from components shared across images. The two most important differences lie in the sharing unit and in how each local region of a synthetic image is parameterized.
HaBa and RTP share whole-image bases, so a local pattern recurring at different positions is not shared. SPEED shares at the local level, but it mixes the shared feature tokens with real-valued weights assigned to each local region and passes them through a learned decoder, so multiple real-valued weights should be stored at each local region. Our method also shares at the local level, but selects one of the codebook patches for each local region at each stage, spending only a small storage budget of a single index.
The benefit of local sharing is shown in Supp. Table 19, where accuracy decreases as the patch size approaches the image size. To support the benefit of selection, we replaced the selection scheme in our method with real-valued weighting over the codebook patches (CIFAR-10, TPC=50).
The weighted sum requires 125$\times$ more storage space than ours but does not improve accuracy.
Furthermore, unlike SPEED, our method needs no decoder. SPEED passes every local region through a learned decoder to synthesize an image, whereas our method composes a synthetic image by selecting the codebook patches, which requires 24 to 1771$\times$ fewer FLOPs per image than SPEED. The total is computed over all synthetic images under the same storage budget, where our method synthesizes more images than SPEED.
Under the same storage budget, our method also outperforms SPEED in all settings of Table 1.
In the revised paper, we clarified this difference in the related work and provided the comparative experimental results in Tables 9 and 12.
W2. Motivation of DCT
Small patches in natural images are usually dominated by low-frequency contents, for example, in CIFAR-10, 98.6% of the energy of 4 × 4 image patches lies in the three lowest-frequency coefficients among the 16 DCT coefficients. We therefore represent codebook patches in the DCT domain to encourage low-frequency bias. To support this, we conducted a simple image-fitting experiment. Starting from an identical initialization, we fit real images with the pixel and DCT parameterizations, respectively, under the setting of our codebook initialization stage. We then split each image into 4 × 4 patches and measured the PSNR on the patches (Full) and on their low-frequency components (Low-freq.) by considering the three lowest-frequency DCT coefficients only. Results are averaged over 5 independent runs.
The DCT parameterization reconstructs the original images more accurately throughout training. At every iteration the gap is larger in low-frequency PSNR than in full PSNR, which indicates that the gain is concentrated in the low-frequency components.