acceptodds
Withdrawn from ICLR 2027

FunDD: Functa Distribution Sampling for Generative Dataset Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A distilled dataset is a small fixed set of synthetic images that leads to similar training performance compared to its original dataset. However, a classifier trained on it sees the same images for hundreds of epochs, its storage grows with every image and with resolution, and a new budget needs a new distillation run. We propose Functa Distribution Distillation (FunDD), which distills a dataset into a distribution instead of a set. Each image is encoded as a latent modulation of a Functa implicit neural representation meta-learned on the dataset, and each class is fitted with a Gaussian over these modulations in closed form. The distilled dataset consists of the class Gaussians and the shared Functa decoder. During classifier training, new modulations are drawn at every step and used once, either decoded into images or fed directly to a classifier in modulation space. On CIFAR-10 and ImageNet subsets, classifiers trained this way exceed classifiers trained on fixed distilled sets from GLaD by 10.2 and 11.3 percentage points on average. Because every sample is used once, the training budget is the number of samples drawn, and at a fixed budget accuracy does not depend on how the samples are grouped into updates. The stored representation is 32 MiB for ten classes and does not change with the number of samples drawn or with image resolution, so the amount of training data a distilled dataset can provide is no longer bounded by what it stores.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Dataset Distillation, Functa, Synthetic Training Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zWyvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes FunDD, which distills a dataset into a distribution rather than a fixed set of images. A Functa implicit neural representation is meta-learned on the training data, and each image is encoded as a latent modulation vector. Each class is then fitted in closed form with a Gaussian over its modulations. The distilled dataset consists of these class Gaussians plus the shared decoder. During classifier training, new modulations are drawn at every step and used once. They are either decoded into images or fed directly to an MLP operating on modulations. A temperature controls how far the draws spread from the class mean.

Critical strengths
  1. The method is simple to distill. It needs no bilevel optimization and the Gaussian is calculated instead of training.
  2. The finding that a classifier can be trained directly on modulations, without decoding images, points to a lightweight training on distilled data.
Critical weaknesses
  1. Fitting a Gaussian per class on latents and sampling from it during training is standard generative modeling. The paper does not explain what separates FunDD from training on samples of any class-conditional generator.
  2. A single Gaussian per class is a weak model of the data. It cannot represent classes. The maximum entropy argument in Section 3.2 only says the Gaussian adds no structure beyond the first two moments, not that it fits the data.
  3. In Table 1, there is no results from the full real dataset.
  4. In Tables 1 and 2, FunDD sees T * C * DCU distinct samples while baselines see only C · IPC. It's not a fair comparison.
  5. FunDD outperforms the baselines by a large margin only at IPC 1, and this margin largely disappears at IPC 10. At IPC 1, each baseline trains on a single stored image per class, while FunDD draws a new image per class at every step, so the large gains mainly reflect seeing far more distinct images.
  6. Tables 1 and 2 omit many recent dataset distillation methods, such as Minimax diffusion, D4M, MGD3 and IGD, SRe2L and RDED, many of which report results on the same ImageNet subsets.
  7. In line 317-323, the paper say that the comparison does not match storage and argues that this mismatch is “the point of the paradigm” (line 320). However, reducing storage is a central purpose of dataset distillation, and in the image-space setting FunDD stores more than the fixed sets. It instead changes the goal from compressing a dataset to generating training data.
Questions for authors
  1. In Table 4, ResNet-18 accuracy on CIFAR-10 drops from 60.0 to 58.9 when DCU grows from 100 to 200, although the classifier sees double amount of images. What causes this drop?
  2. In Table 5, how do you separate the effect of reconstruction quality from the effect of the larger MLP input size?
Reviewer_m9X9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FunDD represents a dataset with a shared Functa decoder and a Gaussian fitted to each class latent modulations. During classifier training, it draws fresh modulations at every update and either decodes them into images or feeds them directly to an MLP. Gaussian fitting is closed form, while the decoder requires meta-training. Experiments on CIFAR-10 and ImageNet subsets compare the method with fixed distilled datasets and examine the benefits of resampling.

Critical strengths

The distributional representation is simple and reusable: the sampling budget can change without repeating distillation. Fixed-versus-resampled controls show that refreshing samples contributes substantially to accuracy. The supplementary learning curves support approximate batch-size invariance on CIFAR-10 under the tested training schedule, across several architectures. The storage accounting includes the decoder and clearly identifies the budgets at which the representation becomes smaller than fixed image sets.

Critical weaknesses
  • The maximum-entropy argument characterizes the fitted Gaussian, but does not explain which class-discriminative structure survives nonlinear decoding, particularly for multimodal classes. The validation-selected temperature further changes the covariance, so the contributions of moment matching and variance contraction remain difficult to distinguish. Fig4 provides empirical support for approximate batch-size invariance, but eq9 does not justify this observation beyond the tested settings.
  • The appendix makes the storage trade-off explicit, but the main accuracy comparisons still use different storage budgets. Matching steps and batch size does not establish an accuracy advantage at equal compression. Comparisons with stored real modulations through the same decoder would help isolate the value of Gaussian compression from the learned representation. The closely related R1 is also cited but not evaluated.

R1: T Qin et al. Distributional dataset distillation with subtask decomposition

Questions for authors
  1. The connection to DM is determined by matching latent moments. What justifies expecting these moments to preserve class-discriminative structure after nonlinear decoding, especially for multimodal classes?
  2. Given the temperature study, what evidence separates the benefit of matching class moments from the benefit of suppressing within-class variation?
  3. What assumptions are needed to justify selecting batch size solely for hardware efficiency? Fig4 supports this empirically under the tested schedule.
  4. At matched total storage, how does FunDD compare with R1 and with replay of real modulations through FunDD decoder?

R1: T Qin et al. Distributional dataset distillation with subtask decomposition

Reviewer_azXVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new method named Functa Distribution Distillation (FUNDD), aiming at generating distilled training data given the original dataset, which can be leveraged to train the model to obtain the similar performance to the training setting with the original dataset. More specifically, by leveraging a Functa decoder meta-learned on the original dataset and the class-level Gaussian distribution over its latent modulation space, the distilled training data can be sampled from modulation space on the fly during the training stage of the target model such as ResNet. Compared with the distilled dataset with fixed size, FUNDD can enable the target model to see more diverse training data across steps. Experimental results on different settings show that the target model trained on the distilled data from FUNDD can usually has better performance than the baselines.

Critical strengths

The high-level idea of FUNDD is reasonable. Compared with a fixed distilled dataset, the training data generated on the fly can provide much more diverse input for the target model to learn.

Critical weaknesses

The critical weaknesses of this paper are listed as follows.

  1. The novelty of this paper is limited. The high-level idea of leveraging a generative method to produce training data on the fly has already been explored, such as D3 mentioned in Section 2.1.
  2. The experimental results are not convincing enough. All the baselines use fixed distilled dataset for the model trainings. The authors should include more baselines that generate training data on the fly for comparison, such as D3.
  3. According to the settings of experiments on CIFAR-10 in Table 4, the total number of data viewed in the training stage (1000 steps) should already reach or exceed the total amount of the original dataset of CIFAR-10. However, all the evaluated image classifiers have much lower accuracy than what is typically expected when training them on the original dataset with a comparable amount of data. For CIFAR-10, an accuracy over 90% is expected for classifiers such as ResNet-18. It would be better if the author could clarify where this huge gap comes from and explain in which cases, when users have access to the original dataset, they will choose to use a distilled dataset to train their model even with huge performance degradation.
  4. This paper is not well self-contained. The process of how to meta-learn the Functa decoder is highly simplified and thus can prevent the readers to understand the methods clearly. It would be better if the authors can provide a pseudo code to show the process of meta-learning.
Questions for authors

Could you help me understand that what the major use cases are for your proposed FUNDD? For the model training, the storage to save training data is usually not the bottleneck. The runtime memory, training speed and computational cost are much more important. Does FUNDD have advantage on these aspects?

Section 4.6 only reports the per-image decoding time. Will it introduce overhead on the aspects like end-to-end training time, runtime memory usage and computational cost since the training data have to be generated on the fly? For model training, when will we want to use a distilled dataset even if we have access to the whole training dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.