- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes FunDD, which distills a dataset into a distribution rather than a fixed set of images. A Functa implicit neural representation is meta-learned on the training data, and each image is encoded as a latent modulation vector. Each class is then fitted in closed form with a Gaussian over its modulations. The distilled dataset consists of these class Gaussians plus the shared decoder. During classifier training, new modulations are drawn at every step and used once. They are either decoded into images or fed directly to an MLP operating on modulations. A temperature controls how far the draws spread from the class mean.
Critical strengths
- The method is simple to distill. It needs no bilevel optimization and the Gaussian is calculated instead of training.
- The finding that a classifier can be trained directly on modulations, without decoding images, points to a lightweight training on distilled data.
Critical weaknesses
- Fitting a Gaussian per class on latents and sampling from it during training is standard generative modeling. The paper does not explain what separates FunDD from training on samples of any class-conditional generator.
- A single Gaussian per class is a weak model of the data. It cannot represent classes. The maximum entropy argument in Section 3.2 only says the Gaussian adds no structure beyond the first two moments, not that it fits the data.
- In Table 1, there is no results from the full real dataset.
- In Tables 1 and 2, FunDD sees T * C * DCU distinct samples while baselines see only C · IPC. It's not a fair comparison.
- FunDD outperforms the baselines by a large margin only at IPC 1, and this margin largely disappears at IPC 10. At IPC 1, each baseline trains on a single stored image per class, while FunDD draws a new image per class at every step, so the large gains mainly reflect seeing far more distinct images.
- Tables 1 and 2 omit many recent dataset distillation methods, such as Minimax diffusion, D4M, MGD3 and IGD, SRe2L and RDED, many of which report results on the same ImageNet subsets.
- In line 317-323, the paper say that the comparison does not match storage and argues that this mismatch is “the point of the paradigm” (line 320). However, reducing storage is a central purpose of dataset distillation, and in the image-space setting FunDD stores more than the fixed sets. It instead changes the goal from compressing a dataset to generating training data.
Questions for authors
- In Table 4, ResNet-18 accuracy on CIFAR-10 drops from 60.0 to 58.9 when DCU grows from 100 to 200, although the classifier sees double amount of images. What causes this drop?
- In Table 5, how do you separate the effect of reconstruction quality from the effect of the larger MLP input size?