FunDD: Functa Distribution Sampling for Generative Dataset Distillation

A distilled dataset is a small fixed set of synthetic images that leads to similar training performance compared to its original dataset. However, a classifier trained on it sees the same images for hundreds of epochs, its storage grows with every image and with resolution, and a new budget needs a new distillation run. We propose Functa Distribution Distillation (FunDD), which distills a dataset into a distribution instead of a set. Each image is encoded as a latent modulation of a Functa implicit neural representation meta-learned on the dataset, and each class is fitted with a Gaussian over these modulations in closed form. The distilled dataset consists of the class Gaussians and the shared Functa decoder. During classifier training, new modulations are drawn at every step and used once, either decoded into images or fed directly to a classifier in modulation space. On CIFAR-10 and ImageNet subsets, classifiers trained this way exceed classifiers trained on fixed distilled sets from GLaD by 10.2 and 11.3 percentage points on average. Because every sample is used once, the training budget is the number of samples drawn, and at a fixed budget accuracy does not depend on how the samples are grouped into updates. The stored representation is 32 MiB for ten classes and does not change with the number of samples drawn or with image resolution, so the amount of training data a distilled dataset can provide is no longer bounded by what it stores.