Why Fair Dataset Distillation Fails on Complex Distributions: A Geometric Analysis and a Spread-Preserving Remedy

Fair dataset distillation compresses a large training set into a small synthetic surrogate that should not amplify disparities across protected-attribute (PA) groups. Existing PA-aware methods differ in how they choose the per-class matching target used to align synthetic images with the real distribution, but agree on its form. Each reduces a class to a single point in feature space. We show that this shared form, not the choice of point, is a limitation. Any objective whose per-class target is a single vector depends on first moments alone, so it can match group means while discarding within-group geometry. Decomposing the squared Bures-Wasserstein distance between each group's real and synthetic feature distributions separates a centroid term, which such objectives control, from a covariance term, which none of them constrain. No relocation of the point can recover this covariance term. The residual grows with within-group variance, which is large for complex real-world distributions such as faces. We propose Fair Geometric Distillation (FGD), whose per-class target is instead a set of per-group coresets of real images approximating each group's distribution under a balanced budget, so the second moment of each group enters the objective. At one fixed regularisation setting, FGD attains the lowest per-group Wasserstein error on all six benchmarks and cuts CelebA DEO${}_M$ from 11.81 to 3.65 at 30 images per class (IPC), a 69% reduction over FairDD and 66% over the strongest prior fair method at that budget, and stays within 3.6 accuracy points of the no-fairness baseline at every budget. Gains are smaller where PA cells are sparse. Control experiments show that neither preventing variance collapse, nor balanced per-group sampling, nor coreset selection without distillation reproduces these gains individually. Progress on fair dataset distillation therefore depends on the form of the per-class target, not the choice of point within it.