acceptodds
Under review as a conference paper at ICLR 2027

Why Fair Dataset Distillation Fails on Complex Distributions: A Geometric Analysis and a Spread-Preserving Remedy

Abstract

Fair dataset distillation compresses a large training set into a small synthetic surrogate that should not amplify disparities across protected-attribute (PA) groups. Existing PA-aware methods differ in how they choose the per-class matching target used to align synthetic images with the real distribution, but agree on its form. Each reduces a class to a single point in feature space. We show that this shared form, not the choice of point, is a limitation. Any objective whose per-class target is a single vector depends on first moments alone, so it can match group means while discarding within-group geometry. Decomposing the squared Bures-Wasserstein distance between each group's real and synthetic feature distributions separates a centroid term, which such objectives control, from a covariance term, which none of them constrain. No relocation of the point can recover this covariance term. The residual grows with within-group variance, which is large for complex real-world distributions such as faces. We propose Fair Geometric Distillation (FGD), whose per-class target is instead a set of per-group coresets of real images approximating each group's distribution under a balanced budget, so the second moment of each group enters the objective. At one fixed regularisation setting, FGD attains the lowest per-group Wasserstein error on all six benchmarks and cuts CelebA DEO from 11.81 to 3.65 at 30 images per class (IPC), a 69% reduction over FairDD and 66% over the strongest prior fair method at that budget, and stays within 3.6 accuracy points of the no-fairness baseline at every budget. Gains are smaller where PA cells are sparse. Control experiments show that neither preventing variance collapse, nor balanced per-group sampling, nor coreset selection without distillation reproduces these gains individually. Progress on fair dataset distillation therefore depends on the form of the per-class target, not the choice of point within it.

Primary area:
optimization
Keywords:
dataset distillation, dataset condensation, algorithmic fairness, group fairness, coresets, optimal transport, Wasserstein distance, protected attributes

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Yu5ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on fair dataset distillation. Existing PA-aware methods differ in their class-wise alignment targets, yet all represent each class as a single point in feature space. This can miss differences in how widely or unevenly each group’s data is distributed. Authors propose Fair Geometric Distillation (FGD), which uses small, group-specific sets of real examples as targets for the synthetic data. FGD achieves sustainable improvement across six benchmarks. Main innovation of this paper is reframing fair dataset distillation as a distribution-geometry problem: matching group means alone can miss within-group variation, which may contribute to fairness failures on complex data.

Critical strengths

1 This paper shows a clear limitation of exisiting fairness-aware dataset distillation: aligning each group to a single feature-space target may overlook within-group variation. A complete theoretical analysis further supports this argument.

2 The evaluation experiment examines both group-level geometric metrics and downstream fairness across selected six benchmarks. FGD improves the reported geometric metrics across the datasets and achieves the lowest maximum equalized-odds gap in 7 of 9 face-dataset and IPC settings. This result provides useful evidence that the proposed approach FGD can improve fairness-related outcomes in practice, while making its accuracy–fairness trade-offs visible.

Critical weaknesses

Major:

1 Paper’s geometric claim is not fully supported by the proposed objective. In equation 8, synthetic examples are matched one-sidedly to their nearest target medoids, with a group-mean regularizer. But this process does not ensure that all medoids are represented or that within-group covariance is matched.

2 The experimental evidence does not make the robustness of the fairness improvements clear. The main tables report means over five seeds without variability estimates, while the appendix notes that some control differences are within seed noise.

3 Some equations need further explanations.

Minor:

1 An additional visualization could make the proposed method easier to follow.

2 Additional experiments to substantiate the paper’s central claims could better shape this paper. Experiments that can directly assess within-group distribution coverage can be beneficial.

Questions for authors

1 Several equations would benefit from additional clarification. First, for Eq. 8, please specify the direction of the nearest-medoid matching. Do multiple synthetic examples match the same medoid? How the group-mean regularizer is computed and weighted?

2 How does FGD perform as protected-group sizes become more imbalanced or sparse?

2 comments

We thank the reviewer for the positive assessment of the presentation and the geometric perspective and for concrete requests that strengthened the paper.

Major 1: Eq. (8) does not ensure all medoids are represented or that covariance is matched

A fair observation about the nearest-medoid step, and we measured it: under 100 fresh feature networks not used in distillation, it covers 61% of the re-solved medoids at IPC 10 and 33% at IPC 50.

FGD's design allows a principled fix. Every (class, group) cell has exactly as many medoids as synthetic slots, so the matching can be made one-to-one with a Hungarian assignment σ\sigma at negligible cost. We call this FGD-assign:

Prop. A. For every feature network θ\theta, Lassign=min⁡σ1M∑i=1M∥ϕθ(x~i)−tσ(i)∥2=W22(μ^Sθ,μ^Tθ) ≥ ∥mS−mT∥2+B2(ΣS,ΣT)\mathcal{L}_{\text{assign}}=\min_\sigma\frac1M\sum_{i=1}^{M}\big\Vert \phi_\theta(\tilde x_i)-t_{\sigma(i)}\big\Vert ^2=W_2^2\big(\hat\mu_S^\theta,\hat\mu_T^\theta\big)\ \ge\ \Vert m_S-m_T\Vert ^2+\mathfrak{B}^2(\Sigma_S,\Sigma_T) (Gelbrich bound), so it controls the mean and the full covariance. Zero loss holds if and only if every medoid is represented exactly once.

This delivers what the reviewer asked for, and it improves the method:

  • Coverage increases to 85% (IPC 10) and 65% (IPC 50), with at most ~6 points per medoid (vs up to ~16);
  • accuracy rises in all nine dataset × IPC settings (IPC 10/30/50), by 1.8–6.4 points;
  • fairness is preserved or improved on CelebA and UTKFace, and nearest-medoid FGD remains the fairest method on bFFHQ.

The revision presents both variants.

Major 3 / Q1: Clarifying Eq. (8)

Within each (class yy, group aa) cell, at each training step:

  1. A random feature network θ\theta is sampled.
  2. My,aM_{y,a} medoids {tj}\lbrace t_j\rbrace are chosen by k-median from a random subsample R~y,a\tilde R_{y,a} of the cell's real pool, where My,aM_{y,a} is the cell's number of synthetic slots.
  3. Direction. Each synthetic feature is matched to its nearest medoid and the cell cost is 1My,a∑imin⁡j∥ϕθ(x~i)−tj∥2\frac{1}{M_{y,a}}\sum_i\min_j\Vert \phi_\theta(\tilde x_i)-t_j\Vert ^2. Gradients flow only into the synthetic images and the medoids are fixed targets.
  4. Multiplicity. Several synthetic points may share a medoid in Eq. (8); FGD-assign makes the matching one-to-one.
  5. Regulariser. λ∥ϕθ‾(Sy,a)−ϕθ‾(R~y,a)∥2\lambda \big\Vert \overline{\phi_\theta}(S_{y,a})-\overline{\phi_\theta}(\tilde R_{y,a})\big\Vert ^2 (the difference between the mean features of the synthetic set and of the real subsample) on the same subsample, with a single global λ=0.25\lambda=0.25.
  6. Cell costs are averaged uniformly over the groups of a class, independent of group frequency, and summed over classes.

We will rewrite §4 with these steps and move the training-loop diagram into the main text.

Q2: Performance as protected groups become more imbalanced

We subsampled CelebA's minority group to bias ratios 0.7–0.99 and compared FGD and FGD-assign with FairDD on identical data. Seed-paired difference to FairDD (Δ\DeltaAcc / Δ\DeltaDEO$_M$):

Bias ratioFGD, IPC 10FGD, IPC 50FGD-assign, IPC 10FGD-assign, IPC 50
0.70−5.9 / −2.6−10.6 / −10.2−1.5 / −1.3−1.9 / −5.0
0.90−5.1 / −1.1−6.6 / −8.6−1.6 / −2.8−2.6 / −8.1
0.95−4.9 / −1.1−5.9 / −9.2−2.2 / −3.8−1.0 / −6.1
0.99−3.6 / −7.8−5.6 / −8.3−0.3 / −6.8+1.0 / −8.6
  • FGD's advantage is largest at the most extreme ratio (−7.8 at 0.99, IPC 10) and steady at 8–10 points at IPC 50, and thus its fairness is robust to imbalance.
  • FGD-assign keeps the robustness and removes most of the accuracy cost: within 0.3–2.6 points of FairDD at every ratio, with DEO$_M$ lower by 5.0–8.6 at IPC 50. At ratio 0.99, IPC 50, it is better on both axes (+1.0 Acc, −8.6 ± 2.4 DEO$_M$).

This is what FGD is designed for so each minority cell keeps its own slots and real targets however rare the group becomes, whereas a class-shared target is increasingly dominated by the majority.

Major 2: Variability

We add seed spreads to the main table. Bold: lowest DEO in each column.

MethodIPCCelebA DEO$_M$ ↓CelebA DEO$_A$ ↓CelebA Acc ↑bFFHQ DEO$_M$ ↓bFFHQ DEO$_A$ ↓bFFHQ Acc ↑UTKFace DEO$_M$ ↓UTKFace DEO$_A$ ↓UTKFace Acc ↑
DM1030.01 ± 2.4728.85 ± 2.2660.97 ± 1.1344.32 ± 1.3334.76 ± 0.1764.90 ± 0.6822.29 ± 2.2518.64 ± 2.0661.24 ± 0.29
DM3034.45 ± 2.8932.67 ± 1.6862.45 ± 0.7153.68 ± 1.5442.92 ± 0.9765.02 ± 0.0922.87 ± 1.4420.46 ± 1.2463.48 ± 1.29
DM5040.26 ± 2.5738.81 ± 2.1463.57 ± 0.7860.88 ± 1.9350.56 ± 0.9065.28 ± 0.1922.50 ± 1.7120.31 ± 1.8664.97 ± 1.15
FairDD109.37 ± 1.625.37 ± 0.7364.37 ± 0.2015.84 ± 1.8110.68 ± 1.2365.46 ± 1.3818.37 ± 1.4014.44 ± 1.1061.98 ± 0.49
FairDD3011.81 ± 0.938.02 ± 1.0866.10 ± 0.8417.96 ± 1.8911.44 ± 1.2068.46 ± 0.5419.06 ± 1.6016.56 ± 0.9163.98 ± 1.61
FairDD5014.08 ± 1.199.87 ± 1.6768.50 ± 0.8419.76 ± 1.4514.96 ± 1.7469.00 ± 1.1418.99 ± 0.7216.43 ± 1.2065.19 ± 0.41
COBRA1010.00 ± 0.816.36 ± 1.0764.94 ± 0.1920.24 ± 0.8111.40 ± 0.7468.30 ± 0.3421.00 ± 1.6417.43 ± 0.7549.95 ± 0.45
COBRA3010.65 ± 1.045.98 ± 0.5667.04 ± 0.8516.56 ± 1.9910.56 ± 2.7370.56 ± 0.7721.80 ± 1.4016.93 ± 1.0450.97 ± 0.55
COBRA5015.32 ± 1.7710.65 ± 1.4168.48 ± 0.6214.96 ± 1.8910.00 ± 1.5371.40 ± 0.4622.20 ± 1.5017.24 ± 0.7253.80 ± 0.51
FGD105.01 ± 1.213.18 ± 0.6959.49 ± 0.4613.18 ± 1.048.60 ± 1.7665.82 ± 1.1421.82 ± 1.4218.18 ± 1.1760.07 ± 0.54
FGD303.65 ± 0.952.64 ± 0.6059.01 ± 1.7416.01 ± 1.1810.48 ± 0.9868.28 ± 0.4018.99 ± 0.4015.58 ± 1.4059.99 ± 0.77
FGD505.31 ± 1.353.01 ± 0.6359.99 ± 1.0617.44 ± 1.8811.99 ± 0.8067.01 ± 0.4914.98 ± 1.0811.74 ± 1.8658.96 ± 0.89

Minor 2: Directly assessing within-group coverage

We add per-cell coverage diagnostics, all evaluated in fresh feature networks not used during distillation (100 networks, all seeds at IPC 10):

  • fraction of medoids matched;
  • maximum multiplicity (points per medoid);
  • W2W_2 to the medoids;
  • Bures discrepancy B(ΣS,ΣR)\mathfrak{B}(\Sigma_S,\Sigma_R) to the real cell.

Each cell: FGD / FGD-assign / GroupMoment (mean + full covariance on FGD's allocation, best β\beta); per-cell averages over 100 fresh networks and 3 seeds (W2W_2 and Bures normalised per cell).

Dataset, IPCCoverage ↑Max. multiplicity ↓W2W_2 to medoids ↓Bures to real cell ↓
CelebA 100.41 / 0.63 / 0.703.3 / 2.6 / 2.30.89 / 0.77 / 1.141.02 / 0.92 / 1.13
bFFHQ 100.43 / 0.74 / 0.754.4 / 2.3 / 2.21.06 / 0.83 / 1.241.03 / 0.94 / 1.17
UTKFace 100.61 / 0.89 / 0.853.0 / 2.0 / 2.01.03 / 0.90 / 1.171.05 / 1.03 / 1.22
  • FGD-assign has the lowest W2W_2 to the medoids in all three settings.
  • Its coverage is within 0.07 of mean + covariance matching, and higher on UTKFace IPC 10.
  • It also has the lowest Bures discrepancy to the real cell in all three settings, below GroupMoment, which matches covariance explicitly. This is Prop. A in practice: the one-to-one coreset loss controls the covariance as a by-product.
Reviewer_t49xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that mean-based fair dataset distillation can lose within-group variation. It uses a Wasserstein lower bound to study this problem and proposes Fair Geometric Distillation (FGD), replacing centroid targets with sets of real examples and a nearest-neighbor matching loss. Experiments compare fairness, accuracy, and feature geometry, with controls for variance regularization, balanced sampling, and coreset selection. Supplementary code provides further implementation details.

Critical strengths

S1. The focus on within-group geometry gives a useful perspective on the limitations of mean-based fair distillation. In a fixed feature space, groups with the same mean can still differ substantially in spread, so matching centroids alone cannot distinguish them. The Wasserstein lower bound helps formalize this distinction.

S2. Using sets of real representatives offers a concrete alternative to a single centroid target. FGD provides several reference locations within each class–group cell and combines nearest-neighbor matching with a cell-wise mean regularizer. The targets therefore remain tied to observed group data.

S3. The experiments report sizeable fairness gains and include controls for their source. On CelebA at IPC=30, FGD reduces DEOM from FairDD's 11.81 to 3.65. The variance-regularization, balanced-sampling, and coreset-only comparisons help examine alternative explanations.

Critical weaknesses

W1. The matching loss does not ensure coverage of the target distribution. In Equation (8), several synthetic points can select the same medoid while other modes remain uncovered, and the mean regularizer does not prevent this. Thus, control of within-group covariance beyond proximity to real examples remains unestablished.

W2. The fixed-feature analysis does not yet explain the full distillation procedure. Training repeatedly resamples the feature network, so limitations of a mean loss at one step do not establish insensitivity to spread across changing representations. Although collapse is acknowledged as empirical, identifying target form as its main cause still needs justification.

W3. The experiments leave the fairness–accuracy trade-off and the reliability of the gains unresolved. On CelebA at IPC=50, FGD has 59.99% accuracy versus 68.50% for FairDD, so the fairness advantage needs to be assessed at comparable predictive performance. Seed variability and the exact reproduction configuration also need clarification: the README uses lambda=1, while the paper uses 0.25.

Questions for authors

Q1. What in the objective prevents several synthetic points from matching the same medoid? Is there a coverage or assignment constraint that is not included in Equation (8)?

Q2. How does the fixed-feature argument extend to the resampled feature networks used in training? What distinguishes a limitation of the target from a collapse caused by optimization?

Q3. Does the fairness advantage remain at matched accuracy, with uncertainty reported for the main methods? A per-group covariance-matching control and the exact supplementary configuration would help interpret this comparison.

4 comments

We thank the reviewer for a careful reading, including the supplementary code. Each of the three weaknesses led to a concrete improvement, addressed below with new experiments.

W1 / Q1: Nothing prevents several synthetic points from matching the same medoid

Correctly pointed that for Eq. (8), its nearest-medoid step has no assignment constraint. Under 100 fresh feature networks it covers 61% of the medoids at IPC 10 and 33% at IPC 50.

FGD's structure makes the fix natural. Each (class, group) cell has exactly as many medoids as synthetic slots, so the matching can be made one-to-one with a Hungarian assignment σ\sigma at negligible cost. This variant, FGD-assign, gives the constraint reviewer asked about, with a guarantee:

Prop. A. For every feature network θ\theta, Lassign=min⁡σ1M∑i=1M∥ϕθ(x~i)−tσ(i)∥2=W22(μ^Sθ,μ^Tθ) ≥ ∥mS−mT∥2+B2(ΣS,ΣT)\mathcal{L}_{\text{assign}}=\min_\sigma\frac1M\sum_{i=1}^{M}\big\Vert \phi_\theta(\tilde x_i)-t_{\sigma(i)}\big\Vert ^2=W_2^2\big(\hat\mu_S^\theta,\hat\mu_T^\theta\big)\ \ge\ \Vert m_S-m_T\Vert ^2+\mathfrak{B}^2(\Sigma_S,\Sigma_T) (Gelbrich bound). Zero loss holds iff every medoid is covered exactly once.

Empirically, under fresh networks:

  • coverage increases to 85% (IPC 10) and 65% (IPC 50);
  • at most ~6 points share a medoid, versus up to ~16;
  • it has the lowest W2W_2 to the medoids of all methods.

Accuracy also rises in every setting (3 datasets × IPC 10/30/50), by 1.8–6.4 points over FGD, while fairness is preserved or improved on CelebA and UTKFace. Plain FGD remains the fairest method on bFFHQ. We present both the FGD-assign with a guarantee and the original matching as the fairness-maximizing option.

W2 / Q2: Fixed features vs. resampled networks, target vs. optimisation

Under resampling, mean matching becomes an MMD with the random-network kernel k(x,x′)=Eθ⟨ϕθ(x),ϕθ(x′)⟩,k(x,x')=\mathbb{E}_\theta\langle\phi_\theta(x),\phi_\theta(x')\rangle, so it is not variance-blind in the strict per-θ\theta sense and we revise §3 accordingly. Two new results separate target from optimisation:

  • The target, measured without any optimisation. We scored fixed sets with FairDD's own objective, averaged over 200 fresh networks. Group-balanced random samples of real images, which have the real spread by construction, score 4–7× worse than FairDD's distilled set at IPC 10. A spread-preserving nn-point set estimates each group mean only up to sampling error Tr⁡Σ/n\operatorname{Tr}\Sigma/n, whereas a set unconstrained in spread can match the means almost exactly. At practical budgets the FairDD objective therefore gives no reward for spread which is a property of the target, independent of the optimiser.
  • Optimiser held fixed and target is varied. With identical allocation, pools, initialisation and optimiser (seed-paired Δ\DeltaAcc / Δ\DeltaDEO$_M$):
StepCelebA 10CelebA 30CelebA 50bFFHQ 10bFFHQ 30bFFHQ 50UTK 10UTK 30UTK 50
FGD allocation + per-cell mean, vs FairDD−0.9 / +1.1−0.7 / −3.4−1.7 / −0.6+2.7 / −9.1+2.6 / −1.5+1.7 / −1.4−2.1 / 0.0−1.5 / +0.2−1.9 / −6.2
+ full Bures covariance, vs previous+0.2 / −0.9−0.5 / −0.9−0.3 / −1.7+0.1 / +0.7−0.3 / +1.0−0.3 / +1.5−0.4 / −2.5−0.6 / −2.9−0.2 / +0.3
coreset target (FGD-assign), vs best of above−1.5 / −2.3−1.0 / +0.2−0.3 / −1.6−2.3 / −1.6−3.0 / −6.5−2.3 / −8.0+0.6 / +4.8−1.9 / −0.1−1.6 / −1.3
Bures without per-group allocation, vs FairDD+0.1 / 0.0+0.3 / −0.10.0 / +0.3+2.1 / +9.1−0.6 / +11.7−0.4 / +18.5−0.1 / +1.3−0.3 / −0.10.0 / +0.3
  • Both of FGD's components contribute: per-group allocation (largest where minority cells are smallest) and the coreset target (a further 1.6–8.0 DEO$_M$ on CelebA and bFFHQ at IPC 10/50 and 6.5 on bFFHQ at IPC 30).
  • Per-group covariance adds little, and without per-group targets it never helps.
  • On UTKFace at IPC 10 (only 2–3 slots per cell, four groups per class) a 2–3-medoid coreset and a Gaussian summary carry nearly the same information, so mean + covariance is ahead. As cells grow, the gap closes like −0.1 DEO$_M$ at IPC 30 (7–8 slots) and −1.3 at IPC 50.

W3 / Q3: Matched accuracy, uncertainty and configuration

Matched accuracy.

  • On bFFHQ, FGD-assign matches FairDD's accuracy (67.6 vs 65.2, 73 vs 69) with a reduced DEO$_M$ (15.6 vs 15.8 at IPC 10 and 19.1 vs 19.7 at IPC 50).
  • On CelebA at IPC 50, the setting you highlight, FGD-assign reduces the accuracy gap to FairDD within our protocol from 8.3 to 2.2 points, while keeping a lower DEO$_M$ (−9.2).
  • Higher-λ\lambda operating points (λ∈{0.25,1,4}\lambda\in\lbrace 0.25,1,4\rbrace): on CelebA IPC 10, λ=1\lambda=1 is within 1.1 accuracy points of FairDD with lower DEO$_M$ (−1.7), on UTKFace IPC 10, λ=4\lambda=4 is within 0.9 points at comparable DEO$_M$ while on UTKFace IPC 50, λ=1\lambda=1 gives −8.5 ± 1.7 DEO$_M$ for −4.3 Acc.

Uncertainty. We add seed spreads to the main table. Bold: lowest DEO in each column.

MethodIPCCelebA DEO$_M$ ↓CelebA DEO$_A$ ↓CelebA Acc ↑bFFHQ DEO$_M$ ↓bFFHQ DEO$_A$ ↓bFFHQ Acc ↑UTKFace DEO$_M$ ↓UTKFace DEO$_A$ ↓UTKFace Acc ↑
DM1030.01 ± 2.4728.85 ± 2.2660.97 ± 1.1344.32 ± 1.3334.76 ± 0.1764.90 ± 0.6822.29 ± 2.2518.64 ± 2.0661.24 ± 0.29
DM3034.45 ± 2.8932.67 ± 1.6862.45 ± 0.7153.68 ± 1.5442.92 ± 0.9765.02 ± 0.0922.87 ± 1.4420.46 ± 1.2463.48 ± 1.29
DM5040.26 ± 2.5738.81 ± 2.1463.57 ± 0.7860.88 ± 1.9350.56 ± 0.9065.28 ± 0.1922.50 ± 1.7120.31 ± 1.8664.97 ± 1.15
FairDD109.37 ± 1.625.37 ± 0.7364.37 ± 0.2015.84 ± 1.8110.68 ± 1.2365.46 ± 1.3818.37 ± 1.4014.44 ± 1.1061.98 ± 0.49
FairDD3011.81 ± 0.938.02 ± 1.0866.10 ± 0.8417.96 ± 1.8911.44 ± 1.2068.46 ± 0.5419.06 ± 1.6016.56 ± 0.9163.98 ± 1.61
FairDD5014.08 ± 1.199.87 ± 1.6768.50 ± 0.8419.76 ± 1.4514.96 ± 1.7469.00 ± 1.1418.99 ± 0.7216.43 ± 1.2065.19 ± 0.41
COBRA1010.00 ± 0.816.36 ± 1.0764.94 ± 0.1920.24 ± 0.8111.40 ± 0.7468.30 ± 0.3421.00 ± 1.6417.43 ± 0.7549.95 ± 0.45
COBRA3010.65 ± 1.045.98 ± 0.5667.04 ± 0.8516.56 ± 1.9910.56 ± 2.7370.56 ± 0.7721.80 ± 1.4016.93 ± 1.0450.97 ± 0.55
COBRA5015.32 ± 1.7710.65 ± 1.4168.48 ± 0.6214.96 ± 1.8910.00 ± 1.5371.40 ± 0.4622.20 ± 1.5017.24 ± 0.7253.80 ± 0.51
FGD105.01 ± 1.213.18 ± 0.6959.49 ± 0.4613.18 ± 1.048.60 ± 1.7665.82 ± 1.1421.82 ± 1.4218.18 ± 1.1760.07 ± 0.54
FGD303.65 ± 0.952.64 ± 0.6059.01 ± 1.7416.01 ± 1.1810.48 ± 0.9868.28 ± 0.4018.99 ± 0.4015.58 ± 1.4059.99 ± 0.77
FGD505.31 ± 1.353.01 ± 0.6359.99 ± 1.0617.44 ± 1.8811.99 ± 0.8067.01 ± 0.4914.98 ± 1.0811.74 ± 1.8658.96 ± 0.89

Configuration. Every reported FGD number uses λ=0.25\lambda=0.25. The README's λ=1\lambda=1 was a stale example and we have corrected it, set 0.25 as the code default.

Dear Reviewer,

As the author response period is nearing its deadline, we wanted to check if there are any remaining concerns or questions from your end that we could help clarify within the timeframe. If our responses have addressed your concerns satisfactorily, we would greatly appreciate it if you could consider indicating your willingness to raise the score.

Thank you again for your time and thoughtful feedback.

Dear Reviewer t49x,

Thank you again for your detailed review. As the discussion period is nearing its end, we wanted to point to where the revised paper addresses each of your concerns, with all changes marked in blue.

  • Coverage (W1). FGD-assign matches points to medoids one-to-one, and Proposition 2 shows zero loss requires every medoid to be covered. On UTKFace, coverage rises from 61% to 85% at IPC 10 (Tables 17–18).
  • Target vs. optimiser (W2). With optimisation held fixed, FairDD's loss is lowest near the real spread, and more spread never raises it across random real subsets (Appendix V). The objective misses which groups the spread covers, so the limitation lies in the target.
  • Accuracy and uncertainty (W3). FGD-assign raises accuracy in all nine settings by 1.8 to 6.4 points and matches the no-fairness baseline on CelebA (Appendix S). Seed standard deviations are in Table 21, and all results use λ=0.25\lambda = 0.25.

We hope these revisions address your concerns, and we would be grateful if you could consider raising the score.

Thank you

Reviewer_ugv2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The submission argues that matching feature means can fail to preserve within-group geometry in fair DD. It uses a Wasserstein analysis to motivate FGD, which replaces centroid targets with real medoids for each class and demographic group. Synthetic samples are optimized using a nearest-neighbor loss and a mean-matching regularizer. Experiments report geometric and fairness improvements on several benchmarks.

Critical strengths

FGD addresses a relevant problem and reports substantial fairness improvements, particularly on CelebA. The geometric diagnostics and controls for variance, balanced sampling, and coreset selection provide useful insight into the method. The authors also acknowledge the accuracy trade-off and that better geometry does not consistently predict better fairness, which makes the empirical discussion more informative.

Critical weaknesses

I think the main issue is the gap between the geometric argument and the claimed fairness benefit:

  • The variance-blindness argument assumes a fixed feature map, while the actual optimization resamples nonlinear maps. A sufficiently rich feature family can constrain more than the mean, so the extension needs justification. The proposed loss also does not explicitly enforce medoid coverage or mass. For a fixed feature map, the objective can admit many zero-loss configurations that match the coreset mean but fail to preserve its spread, because medoid coverage and mass allocation are not explicitly enforced. It remains unclear whether feature resampling rules out these configurations.
  • Better within-group geometry does not imply smaller prediction disparities. The fairness gains come with noticeable accuracy costs, and the results do not clearly separate improved disadvantaged-group performance from reduced performance elsewhere. The trace-only and balanced-sampling controls also leave full covariance matching and dedicated group allocation unresolved. I encourage the authors to do comparisons at matched accuracy and controls that hold group allocation fixed to establish what the coreset objective contributes.
Questions for authors
  1. Proposition 1 assumes a fixed feature map, but optimization resamples nonlinear maps. Why does variance blindness still hold when a rich family of feature expectations can constrain more than the mean?
  2. What property of the expected FGD loss would rule out the zero-loss covariance mismatch described above?
  3. What links lower within-group Wasserstein error to smaller equalized-odds gaps when groups can remain different in predictive difficulty?
  4. Can a mean-and-full-covariance baseline recover the gains under identical group allocation, real pools, and initialization? This would isolate the value of the medoid objective beyond dedicated group allocation.
  5. How do the group-conditional error rates change when FGD and the baselines are compared at matched overall accuracy?
8 comments

We thank the reviewer for a precise and constructive review. Your questions specify exactly the controls needed to separate FGD's components, and we run experiments on them:

  • (i) a rigorous treatment of resampled feature maps, with a measurement of what the averaged objective rewards;
  • (ii) a one-to-one matching step that makes the cell loss an exact Wasserstein distance with a coverage guarantee (Prop. A) and improves accuracy in every setting;
  • (iii) the allocation based results (mean, mean + full covariance and coreset targets under identical allocation, pools and initialisation);
  • (iv) matched-accuracy comparisons.

Q1: Variance-blindness under resampled nonlinear maps

The reviewer rightly points out that Proposition 1 is a per-θ\theta statement. Averaged over θ\theta, mean matching becomes a kernel MMD:

Eθ∥ESϕθ−EPϕθ∥2=MMDk2(S,P),k(x,x′)=Eθ⟨ϕθ(x),ϕθ(x′)⟩.\mathbb{E}_\theta\big\Vert \mathbb{E}_S\phi_\theta-\mathbb{E}_P\phi_\theta\big\Vert ^2=\mathrm{MMD}_k^2(S,P),\qquad k(x,x')=\mathbb{E}_\theta\langle\phi_\theta(x),\phi_\theta(x')\rangle.

So the averaged objective constrains more than the mean, to the extent that kk is rich. The question is whether it does so at practical budgets, and we measured this directly. Using FairDD's own objective, averaged over 200 fresh networks, we scored (a) FairDD's distilled set and (b) group-balanced random samples of real images of the same size, which have exactly the real spread.

The real samples score 4–7× worse at IPC 10 (normalized loss 0.11–0.13 vs 0.015–0.029) and 1.6–3× worse at IPC 50.

The reason is that an nn-point set with the real spread estimates each group mean only up to sampling error, E∥m^n−m∥2=Tr⁡Σ/n;\mathbb{E}\Vert \hat m_n-m\Vert ^2=\operatorname{Tr}\Sigma/n; indeed the real samples' loss falls roughly as 1/IPC1/\text{IPC}. A set free of the real spread can drive this error to zero. So at practical budgets FairDD's objective gives no reward for spread, and preserving spread costs loss. This is the precise sense in which variance-blindness survives resampling. §3 will present (1) the per-θ\theta statement, (2) the kernel identity, (3) this finite-budget analysis, and "provably variance-blind" will be replaced by this formulation.

Q2: What rules out zero-loss covariance mismatch?

This led to the most useful change in the revision. FGD gives each (class, group) cell a set of real targets: My,aM_{y,a} k-median medoids for My,aM_{y,a} synthetic slots. Equal sizes let the matching be made one-to-one with a Hungarian assignment σ\sigma at negligible cost. This variant, FGD-assign, has the guarantee the reviewer asked for:

Prop. A. For every feature network θ\theta, Lassign=min⁡σ1M∑i=1M∥ϕθ(x~i)−tσ(i)∥2=W22(μ^Sθ,μ^Tθ) ≥ ∥mS−mT∥2+B2(ΣS,ΣT).\mathcal{L}_{\text{assign}}=\min_\sigma\frac1M\sum_{i=1}^{M}\big\Vert \phi_\theta(\tilde x_i)-t_{\sigma(i)}\big\Vert ^2=W_2^2\big(\hat\mu_S^\theta,\hat\mu_T^\theta\big)\ \ge\ \Vert m_S-m_T\Vert ^2+\mathfrak{B}^2(\Sigma_S,\Sigma_T). Zero loss holds if and only if the synthetic features coincide with the coreset (coverage, mass and covariance matched) and any nonzero loss upper-bounds the mean and covariance mismatch.

Under 100 fresh networks, FGD-assign covers 85% of medoids at IPC 10 and 65% at IPC 50 (vs 61% and 33%), with at most ~6 points per medoid (vs ~16), and the lowest W2W_2 to the medoids of all methods.

FGD-assign − FGD (Δ\DeltaAcc / Δ\DeltaDEO$_M$):

CelebA 10 / 30 / 50bFFHQ 10 / 30 / 50UTKFace 10 / 30 / 50
Δ\DeltaAcc+2.6 / +4.6 / +6.1+1.8 / +4.3 / +6.4+2.9 / +2.4 / +3.4
Δ\DeltaDEO$_M$−2.3 / −2.1 / −0.4+2.4 / +2.9 / +1.7−0.3 / −0.2 / +0.1

Accuracy rises in all nine settings. Fairness is preserved or improved on CelebA and UTKFace on all 3 IPCs. On bFFHQ, the nearest-medoid FGD stays the fairest method. We present both the FGD-assign with a guarantee and the original matching as the fairness-maximizing option.

Q3: Why should lower within-group W2W_2 reduce equalized-odds gaps?

We agree that no general implication holds, and Appendix K mentions this as well. The mechanism our ablations support (Q4) is more specific:

  1. Each group gets its own slots and its own real-data target, so a minority group is never represented only through a pooled class statistic.
  2. Within a cell, the synthetic points must cover the group's support rather than one location.

Together these remove a group-dependent source of error, a minority group reduced to a few points. They do not remove intrinsic difficulty differences between groups, and we will say so.

Q4: Mean + full covariance under identical allocation, pools, and initialization

Seed-paired changes (Δ\DeltaAcc / Δ\DeltaDEO$_M$; negative DEO$_M$ is better):

StepCelebA 10CelebA 30CelebA 50bFFHQ 10bFFHQ 30bFFHQ 50UTK 10UTK 30UTK 50
FGD allocation + per-cell mean, vs FairDD−0.9 / +1.1−0.7 / −3.4−1.7 / −0.6+2.7 / −9.1+2.6 / −1.5+1.7 / −1.4−2.1 / 0.0−1.5 / +0.2−1.9 / −6.2
+ full Bures covariance, vs previous+0.2 / −0.9−0.5 / −0.9−0.3 / −1.7+0.1 / +0.7−0.3 / +1.0−0.3 / +1.5−0.4 / −2.5−0.6 / −2.9−0.2 / +0.3
coreset target (FGD-assign), vs best of above−1.5 / −2.3−1.0 / +0.2−0.3 / −1.6−2.3 / −1.6−3.0 / −6.5−2.3 / −8.0+0.6 / +4.8−1.9 / −0.1−1.6 / −1.3
Bures without per-group allocation, vs FairDD+0.1 / 0.0+0.3 / −0.10.0 / +0.3+2.1 / +9.1−0.6 / +11.7−0.4 / +18.5−0.1 / +1.3−0.3 / −0.10.0 / +0.3
  • Both of FGD's components matter. Per-group allocation gives the largest single gain where minority cells are smallest; the coreset target adds a further 1.6–8.0 DEO$_M$ on CelebA and bFFHQ at IPC 10/50, and 6.5 on bFFHQ at IPC 30.
  • Full covariance adds little, with or without the allocation.
  • UTKFace at IPC 10 favours mean + covariance which is a budget effect. UTKFace has four groups per class, so a cell has only 2–3 slots at IPC 10. With 2–3 points, a 2–3-medoid coreset and a Gaussian summary carry nearly the same information, leaving no room for multimodal structure. The gap closes as cells grow: +4.8 at IPC 10, −0.1 at IPC 30 (7–8 slots) and −1.3 at IPC 50 (12–13 slots).

Q5: Group-conditional errors at matched accuracy

  • bFFHQ: FGD-assign matches FairDD's accuracy and reduces its DEO$_M$: 67.6 vs 65.2 Acc with 15.6 vs 15.8 DEO$_M$ (IPC 10); 73 vs 69 with 19.1 vs 19.7 (IPC 50).
  • CelebA: the accuracy gap to FairDD shrinks to ~2 points, with lower DEO$_M$ (−6.7 at IPC 10, −9.2 at IPC 50 against FairDD).
  • Higher-λ\lambda operating points (λ∈{0.25,1,4}\lambda\in\lbrace 0.25,1,4\rbrace): on CelebA IPC 10, λ=1\lambda=1 is within 1.1 points of FairDD's accuracy with lower DEO$_M$ (−1.7); on UTKFace IPC 10, λ=4\lambda=4 is within 0.9 points at comparable DEO$_M$; on UTKFace IPC 50, λ=1\lambda=1 gives −8.5 ± 1.7 DEO$_M$ for −4.3 Acc.
  • Per-group accuracies (worst / best group per class, averaged over classes):
    • bFFHQ: FGD-assign raises the worst group: 61.4 vs 58.1 (IPC 10) and 66.9 vs 62.4 (IPC 50); the worst single cell rises from 53.4 to 58.5 and from 57.5 to 64.5.
    • CelebA: the worst group stays at FairDD's level (58.3 vs 60.4; 62.5 vs 62.8) and the smaller gap comes mostly from a lower best group (63.7 vs 66.5; 67.4 vs 71.9).
    • We will report both patterns so the source of each DEO reduction is visible.
Reviewer_ugv2OpenReview ↗

Further Questions

Thank you for the substantive revisions. The one-to-one assignment addresses the zero-loss coverage issue for FGD-assign, and the additional allocation/covariance controls and group-level results help clarify the contribution.

  • Lower loss for optimized synthetic samples than randomly sampled real images does not establish that preserving spread is penalized. Could you provide evidence separating the effect of spread from optimization?

Comments on the paper

3 comments

Dear reviewers,

The authors have submitted their response. Please carefully check whether the new content has addressed your concerns, particularly the accuracy trade-off, coverage of the target distribution, and objective mechanism analysis. If not, please follow up with the authors ASAP so that the authors can further respond before the discussion period ends (Nov 18).

Best,

AC

We thank the reviewers for their careful reading of the paper and their constructive feedback.

Strengths highlighted by the reviewers

  • R1 (ugv2) recognised that FGD "addresses a relevant problem and reports substantial fairness improvements, particularly on CelebA". The reviewer found that the geometric diagnostics and controls "provide useful insight into the method", and that acknowledging the accuracy trade-off makes the empirical discussion more informative.
  • R2 (t49x) described the focus on within-group geometry as "a useful perspective on the limitations of mean-based fair distillation", noting that "the Wasserstein lower bound helps formalize this distinction". The reviewer valued that FGD's targets "remain tied to observed group data", and highlighted the sizeable fairness gains and the controls for their source.
  • R3 (Yu5R) found that the paper "shows a clear limitation of existing fairness-aware dataset distillation", supported by "a complete theoretical analysis", and that the evaluation gives useful evidence FGD "can improve fairness-related outcomes in practice".

R1 (ugv2)

  • Variance-blindness under resampled networks (Q1). Averaged over networks, mean matching is a kernel MMD. Real samples carrying the true spread still score 4–7× worse than FairDD's set under its own objective, so the objective gives no reward for spread at practical budgets (§3.3, App. P).
  • Zero-loss covariance mismatch (Q2). We added FGD-assign, a one-to-one Hungarian matching. Proposition 2 shows its cell loss is exactly W22\mathcal{W}_2^2 to the coreset, so zero loss forces coverage and covariance to match (§4, App. Q).
  • Geometry vs. equalised odds (Q3). We narrowed the claim: per-group slots keep minority groups from collapsing, but differences in difficulty between groups remain (§6).
  • Mean + full-covariance control (Q4). Under identical allocation, the coreset target lowers DEOM\mathrm{DEO}_M by a further 1.6–8.0 points, while full covariance adds little (Table 3, App. R).
  • Matched accuracy (Q5). On bFFHQ, FGD-assign beats FairDD's accuracy (67.6 vs 65.2; 73.0 vs 69.0) at equal or lower DEOM\mathrm{DEO}_M, and lifts the worst group by 3.3–4.5 points (App. S).

R2 (t49x)

  • Medoid coverage (W1). FGD-assign adds the assignment constraint. On UTKFace, coverage rises from 61% to 85% at IPC 10 and from 33% to 65% at IPC 50 (Table 17).
  • Resampled networks; target vs. optimiser (W2). The kernel-MMD analysis measures the objective without any optimisation, and the attribution holds the optimiser fixed. The lack of reward for spread comes from the target itself (App. P, R).
  • Accuracy and uncertainty (W3). FGD-assign raises accuracy in all nine settings by 1.8–6.4 points, and on CelebA matches the no-fairness baseline's accuracy. Seed standard deviations are now reported for all main results, with two near-ties marked (Table 21).
  • Configuration. All results use λ=0.25\lambda = 0.25, and the README's stale λ=1\lambda = 1 has been corrected.

R3 (Yu5R)

  • Coverage and covariance under Eq. (8) (Major 1, Minor 2). FGD-assign with Proposition 2. Per-cell diagnostics show it has the lowest Bures discrepancy to the real cell, below a target that matches covariance explicitly (Table 18).
  • Variability of gains (Major 2). Seed standard deviations for all main results (Table 21).
  • Clarity of Eq. (8) (Major 3). §4 now walks through one training step: subsample, matching direction, multiplicity, regulariser and averaging.
  • Group imbalance (Q2). At bias ratios from 0.70 to 0.99, FGD's fairness advantage holds at every ratio, and FGD-assign stays within 0.3–2.6 accuracy points of FairDD (Table 20).

Revisions. All changes are in blue, using the additional page. The main text now states variance-blindness per feature map (§3.3), details the training step and adds FGD-assign with Proposition 2 (§4), and adds component attribution and matched accuracy (§5). Appendices P–U add the resampled-network analysis, the proof and coverage diagnostics, attribution with the exact Bures computation, group-conditional accuracy, imbalance robustness and seed standard deviations.

We sincerely thank all reviewers and the AC for their constructive engagement, which has substantially improved the paper.

In summary, our work identifies a limitation shared by existing fair dataset distillation methods. Each one matches the synthetic data of a class to a single target point in feature space. Such an objective can align the group means while ignoring how each group’s data is spread, so minority groups can lose their within-group structure. We propose Fair Geometric Distillation (FGD), which replaces this single point with a small coreset of real images for every class and protected group, so each group’s spread enters the objective. Its variant FGD-assign matches synthetic images to these coreset images one-to-one, which makes each per-group loss an exact Wasserstein distance and guarantees that every coreset image is covered at zero loss.

Strengths highlighted by the reviewers

R1 (ugv2) – FGD "addresses a relevant problem and reports substantial fairness improvements, particularly on CelebA", and the geometric diagnostics and controls "provide useful insight into the method".

R2 (t49x) – The focus on within-group geometry "gives a useful perspective on the limitations of mean-based fair distillation", the targets "remain tied to observed group data", and the experiments "report sizeable fairness gains and include controls for their source".

R3 (Yu5R) – The paper "shows a clear limitation of existing fairness-aware dataset distillation", supported by "a complete theoretical analysis", and offers "useful evidence that the proposed approach FGD can improve fairness-related outcomes in practice".

What changed during the discussion

  • Coverage is now built into the objective. FGD-assign matches synthetic points to medoids one-to-one, and Proposition 2 shows that zero loss requires every medoid to be covered with mass and covariance matched. On UTKFace, coverage rises from 61% to 85% at IPC 10 and from 33% to 65% at IPC 50.
  • The accuracy cost has largely closed. FGD-assign raises accuracy over FGD in all nine settings by 1.8 to 6.4 points and matches the no-fairness baseline's accuracy on CelebA at every budget. On bFFHQ it exceeds FairDD's accuracy (67.6 vs 65.2 and 73.0 vs 69.0) at equal or lower DEOM\mathrm{DEO}_M and lifts the worst group by 3.3 to 4.5 points.
  • The source of the gains is isolated. Under identical allocation, pools and initialisation, the coreset target lowers DEOM\mathrm{DEO}_M by a further 1.6 to 8.0 points on CelebA and bFFHQ at IPC 10 and 50, while full covariance matching adds little.
  • The theory is stated more precisely. Variance-blindness is a per-network property, and the objective, when averaged over networks, sees a class’s total spread but not which groups that spread covers, as the two new controls in Appendix V show.
  • The results are more reliable. Seed standard deviations are reported for all main results, FGD's fairness advantage holds at every bias ratio from 0.70 to 0.99, and all numbers use λ=0.25\lambda = 0.25.

How each reviewer's concerns were addressed

R1 (ugv2) – The reviewer raised five questions on resampled feature maps, zero-loss covariance mismatch, the link to equalized odds, a mean plus full covariance control, and matched accuracy. Each now has a dedicated result in §3.3, §4, §5, §6 and Appendices P to S. The follow-up on separating spread from optimisation led to the two controls in Appendix V, and the reviewer confirmed that the revisions "substantially address my concerns".

R2 (t49x) – The three concerns on medoid coverage, fixed versus resampled features, and the fairness–accuracy trade-off overlap closely with those of R1 and R3, which both reviewers found addressed. FGD-assign with Proposition 2 resolves coverage, Appendix V separates the target from the optimiser, Appendix S covers matched accuracy, Table 21 reports seed variability, and λ=0.25\lambda = 0.25 is used throughout.

R3 (Yu5R) – Coverage and covariance under Eq. (8) are addressed by FGD-assign, whose per-cell Bures discrepancy to the real data is lower than FGD's and lower even than a target that matches covariance explicitly (Table 18). Seed standard deviations (Table 21), a step-by-step account of the training step in §4, and the imbalance study (Table 20) cover the remaining points. The reviewer noted that the response "has addressed my concerns" and "substantially strengthened my confidence in the work".

All changes are marked in blue in the revised manuscript, using the additional page and Appendices P to V.

We are grateful to the reviewers for their thoughtful feedback, which has strengthened the clarity, rigour and practical value of this work, and to the AC for their support and careful oversight.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.