- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies dataset distillation (DD) when only noisy training labels are available. DeDD treats the observed one-vs-rest class-conditional distributions as mixtures, estimates their proportions using TEDN, and recovers clean class feature means through closed-form mixture inversion. It then optimizes synthetic samples to match these recovered targets, using a distribution-matching objective with a positive/negative prototype margin loss. Theorem 1 establishes expected-gradient consistency for the base matching loss under known mixture proportions. Theorem 2 gives a conditional loss-deviation bound for fixed-feature ridge-regression probes, and Appendix A.5 analyzes perturbations from estimated mixture parameters. The experiments cover synthetic and human-annotated label noise on multiple image datasets, along with component ablations, cross-architecture transfer, and computational comparisons.
Critical strengths
- DeDD takes a useful and distinct approach to noisy-label DD: it corrects class-level matching targets while leaving individual labels unchanged, using established MPE and mixture-inversion techniques. Equations (5) to (9) give an interpretable procedure for recovering clean prototypes. Theorem 1 connects the method to clean-label distribution matching under the stated conditions, and Appendix A.5 quantifies its sensitivity to estimated mixture parameters.
- The experiments provide substantial evidence, with broad comparisons under simulated and real-world label noise in Tables 1 and 2 and the appendix. DeDD outperforms the noise-robust RDC baseline in representative CIFAR, CIFAR-N, and ImageNet-subset settings. The ablation in Table 5 shows that decontamination accounts for most of the improvement within the method. Tests across architectures and alternative noise structures suggest that the gains extend beyond a single architecture or synthetic corruption protocol.
Critical weaknesses
-
The comparisons leave some uncertainty about how much the proposed components contribute. On CIFAR-10 at IPC 50, the configuration with neither component in Table 5 achieves 62.57%, compared with 43.96% for DM in Table 1. Differences beyond the two components may therefore account for a substantial part of the reported gap. Appendix D.1 states that baselines use their official implementations and clean-label hyperparameters; Appendix D.4 describes DeDD's initialization, augmentation, and multi-formation recipe. The within-method ablation supports an independent gain from decontamination. Evaluating DM under DeDD's recipe would help determine how much of the overall improvement comes from the proposed mechanism.
-
The abstract, contribution summary, and conclusion should describe the scope of the theoretical guarantees consistently. Theorem 1 establishes an expected-gradient identity for the base matching loss under true mixture proportions. It does not establish equivalent finite-step optimization trajectories for the full objective, which also includes the prototype-margin term. Theorem 2 applies to fixed-feature ridge-regression probes, and its main bound assumes matched class proportions and second moments. Appendix A.4 gives additional terms when these assumptions are relaxed, but the paper does not quantify their magnitude in the experimental setting. Appendix A.5 addresses estimated mixture parameters, though its perturbation bounds depend strongly on the inversion margin. Comparing final accuracy with oracle and estimated parameters, especially at small positive inversion margins, would help clarify where these guarantees hold in practice.
Questions for authors
- Why does the "neither component" configuration in Table 5 achieve 62.57% accuracy, compared with 43.96% for DM in Table 1 on CIFAR-10 at IPC 50? Could you run DM with DeDD's initialization, augmentation, multi-formation, and optimization recipe, then add the proposed components one at a time? Please clarify whether the two tables use the same noise realizations, evaluation protocol, and seed aggregation.
- Figure 1 reports mixture-parameter estimation errors. Could you compare final distilled-set accuracy with oracle and TEDN-estimated proportions across noise severities, including cases with small positive inversion margins? Please report the corresponding parameter errors, inversion margins, and clipping threshold so that the results can be related to the analysis in Appendix A.5.
- Appendix D.4 states that TEDN runs once before distillation and typically finishes within a few minutes. Could you report its wall-clock time and peak GPU memory for the datasets in Table 3, along with the end-to-end costs of DeDD and RDC? These measurements would show how much of the per-iteration savings carries over to the overall computational cost.
I appreciate the author's detailed response for the comparisons with methods such as soft labeling, evaluations at higher resolutions, and efficiency. Most of my concerns have been addressed, and I tend to raise my score to positive.