acceptodds
Under review as a conference paper at ICLR 2027

DeDD: Decontaminating Dataset Distillation under Label Noise

Abstract

Dataset distillation (DD) compresses a large training set into a small synthetic set on which models can be trained to achieve accuracy comparable to that obtained on the original data. Most existing DD methods, however, assume that accurate gold labels are readily available, whereas annotations of large-scale datasets are inevitably corrupted by noise. Under such label noise, existing DD methods degrade substantially, even when preceded by an additional label-cleaning stage. To address this problem, we propose Decontaminating Dataset Distillation (DeDD), a framework that distills a clean synthetic set directly from label-corrupted training data. By modeling the observed class-conditional distributions as mixtures of their latent clean counterparts and analytically inverting this mixture, our method recovers unbiased estimates of the clean class prototypes and matches the synthetic set to them in a single stage, without explicit label cleaning or sample selection. Theoretically, our objective yields, in expectation, the same optimization gradients as distillation on clean data. Experiments with simulated and real-world label noise demonstrate that DeDD outperforms state-of-the-art DD methods. The code package is available at https://anonymous.4open.science/r/DeDD_ICLR27/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dataset Distillation, Weakly Supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sn8XOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dataset distillation (DD) when only noisy training labels are available. DeDD treats the observed one-vs-rest class-conditional distributions as mixtures, estimates their proportions using TEDN, and recovers clean class feature means through closed-form mixture inversion. It then optimizes synthetic samples to match these recovered targets, using a distribution-matching objective with a positive/negative prototype margin loss. Theorem 1 establishes expected-gradient consistency for the base matching loss under known mixture proportions. Theorem 2 gives a conditional loss-deviation bound for fixed-feature ridge-regression probes, and Appendix A.5 analyzes perturbations from estimated mixture parameters. The experiments cover synthetic and human-annotated label noise on multiple image datasets, along with component ablations, cross-architecture transfer, and computational comparisons.

Critical strengths
  1. DeDD takes a useful and distinct approach to noisy-label DD: it corrects class-level matching targets while leaving individual labels unchanged, using established MPE and mixture-inversion techniques. Equations (5) to (9) give an interpretable procedure for recovering clean prototypes. Theorem 1 connects the method to clean-label distribution matching under the stated conditions, and Appendix A.5 quantifies its sensitivity to estimated mixture parameters.
  2. The experiments provide substantial evidence, with broad comparisons under simulated and real-world label noise in Tables 1 and 2 and the appendix. DeDD outperforms the noise-robust RDC baseline in representative CIFAR, CIFAR-N, and ImageNet-subset settings. The ablation in Table 5 shows that decontamination accounts for most of the improvement within the method. Tests across architectures and alternative noise structures suggest that the gains extend beyond a single architecture or synthetic corruption protocol.
Critical weaknesses
  1. The comparisons leave some uncertainty about how much the proposed components contribute. On CIFAR-10 at IPC 50, the configuration with neither component in Table 5 achieves 62.57%, compared with 43.96% for DM in Table 1. Differences beyond the two components may therefore account for a substantial part of the reported gap. Appendix D.1 states that baselines use their official implementations and clean-label hyperparameters; Appendix D.4 describes DeDD's initialization, augmentation, and multi-formation recipe. The within-method ablation supports an independent gain from decontamination. Evaluating DM under DeDD's recipe would help determine how much of the overall improvement comes from the proposed mechanism.

  2. The abstract, contribution summary, and conclusion should describe the scope of the theoretical guarantees consistently. Theorem 1 establishes an expected-gradient identity for the base matching loss under true mixture proportions. It does not establish equivalent finite-step optimization trajectories for the full objective, which also includes the prototype-margin term. Theorem 2 applies to fixed-feature ridge-regression probes, and its main bound assumes matched class proportions and second moments. Appendix A.4 gives additional terms when these assumptions are relaxed, but the paper does not quantify their magnitude in the experimental setting. Appendix A.5 addresses estimated mixture parameters, though its perturbation bounds depend strongly on the inversion margin. Comparing final accuracy with oracle and estimated parameters, especially at small positive inversion margins, would help clarify where these guarantees hold in practice.

Questions for authors
  1. Why does the "neither component" configuration in Table 5 achieve 62.57% accuracy, compared with 43.96% for DM in Table 1 on CIFAR-10 at IPC 50? Could you run DM with DeDD's initialization, augmentation, multi-formation, and optimization recipe, then add the proposed components one at a time? Please clarify whether the two tables use the same noise realizations, evaluation protocol, and seed aggregation.
  2. Figure 1 reports mixture-parameter estimation errors. Could you compare final distilled-set accuracy with oracle and TEDN-estimated proportions across noise severities, including cases with small positive inversion margins? Please report the corresponding parameter errors, inversion margins, and clipping threshold so that the results can be related to the analysis in Appendix A.5.
  3. Appendix D.4 states that TEDN runs once before distillation and typically finishes within a few minutes. Could you report its wall-clock time and peak GPU memory for the datasets in Table 3, along with the end-to-end costs of DeDD and RDC? These measurements would show how much of the per-iteration savings carries over to the overall computational cost.
Reviewer_p8MoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeDD to solve the significant performance degradation issue for dataset distillation caused by noisy labels, especially in the large-scale dataset distillation task. Previous methods for DD with noisy labels usually include an additional label-cleaning stage or sample selection. DeDD directly distills synthetic data directly from label-corrupted training data. Specifically, it models the observed class-conditional distributions as a linear mixture of clean class distributions. It estimates the mixture parameters using introduced mixture proportion estimation and recovers the estimates of the clean class-wise feature means through analytical closed-form mixture inversion, which then are adopted as the optimization targets to distill the synthetic data through distribution matching dataset distillation in a single stage. During synthetic data distillation process, it also introduces prototype margin loss improving inter-class separation. It also provides the theoretical analysis for the decontamination consistency and the loss deviation bound to support the proposed method. Experimental results show the effectiveness of the proposed method under both simulated and real-world label noise settings.

Critical strengths
  1. The proposed method DeDD do not need to recover and correct the noisy labels. It is based on the distribution matching dataset distillation framework, for which the optimization target is matching the class-wise feature statistics. So, this paper transforms the problem into the recover unbiased estimates of the clean class prototypes rather than correct each noisy label. The design is natural and make sense.
  2. The paper provides theoretical analysis for the decontamination consistency and the loss deviation bound to support the proposed method, which also helps to justify the proposed optimization objective.
  3. The paper conducts experiments across multi-resolution datasets and both simulated and real-world noisy label settings. It also includes many ablation studies and tsne visualization analysis to validate the effectiveness of the proposed method.
Critical weaknesses
  1. The paper excludes the DD methods with teacher-generated soft labels, e.g., SRe2L, for external priors, which may because the teacher models are assumed to be trained on clean data. However, in paper, DeDD still requires to train an extra ResNet-50 model on the noisy data for mixture proportion estimation. As soft labels can also capture the information of the class-wise distributions, it is natural to compare the proposed mixture proportion estimation with soft label methods, but the soft labels generated with teachers trained on the noisy data for a fair comparison, to demonstrate its effectiveness.
  2. The paper states that noisy labels especially occur in large-scale datasets, as it is hard to check. However, the proposed DeDD method is implemented based on the distribution matching method, which is challenging to scale up to large-scale and high-resolution datasets, e.g., the full original-resolution ImageNet-1K. The experiments in the paper are also limited to relatively lower-resolution and smaller dataset. The practical value and scaling up capability of DeDD are limited.
  3. DeDD requires training an extra ResNet-50 network, with almost the whole original noisy training data, while the downstream target network is much smaller ConvNet.
Questions for authors
  1. Does the proposed method still have advantages compared with soft label method while the soft labels are generated by the teacher trained on the noisy data?
  2. Can the proposed method DeDD scale up to larger datasets, e.g., the full original-sized ImageNet-1K or integrated with other baseline methods?
  3. What is the total end-to-end runtime of DeDD, including the ResNet-50 training?
1 comment

I appreciate the author's detailed response for the comparisons with methods such as soft labeling, evaluations at higher resolutions, and efficiency. Most of my concerns have been addressed, and I tend to raise my score to positive.

Reviewer_xe2DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DeDD to address dataset distillation under label noise. It models class-conditional distributions as mixtures of clean distributions, which estimates mixture parameters via MPE to recover clean class means. These estimates are incorporated into a distribution matching objective with a prototype margin loss to improve inter-class separation. Experiments on multiple benchmarks with synthetic and real-world label noise demonstrate competitive performance of the proposed method.

Critical strengths
  1. This paper proposes a one-stage dataset distillation framework under label noise, which integrates noise decontamination into the distribution matching process to recover clean class-wise unbiased estimates, without requiring a separate noise detection and filtering stage.
  2. This paper provides a relatively comprehensive theoretical analysis and proof to support the proposed method, including the gradient consistency analysis and the loss deviation bound analysis.
  3. Experiments on multiple benchmarks under both synthetic and real-world label noise demonstrate the effectiveness of the proposed method and its competitive performance against both two-stage and one-stage approaches.
Critical weaknesses
  1. Equation (2) and Equation (4) have same mathematical forms, although they are introduced for clean-label and noisy-label settings, respectively. The definitions of these two formulas are redundant, while this is conceptually reasonable because the ideal distillation objective is defined in the clean data distribution.
  2. In Lemma 1, DeDD relies on a class-wise instance-independent noise assumption to recover clean class prototypes from contaminated class-conditional distributions. However, real-world annotation noise may depend on image difficulty or inter-class similarity. Although the appendix evaluates several alternative noise patterns, the effectiveness of the method to instance-dependent noise remains insufficiently validated, and the limitations of the assumption have not been discussed.
  3. Experiments primarily evaluate DeDD under a fixed synthetic noise rate, but do not systematically discuss the method's performance under different noise rates. Therefore, it is currently unclear whether the method remains effective under more severe label corruption. In addition, the paper uses different synthetic noise rates across datasets (30% corruption on Tiny-ImageNet and 40% on all others), but the reason for these choices is not sufficiently explained.
  4. In Table 1 and 8, DeDD does not outperform the two-stage DATM baseline on CIFAR-10N at IPC 50, whereas it performs strongly in most other settings. The subsection of experimental results only acknowledges this exception but does not provide a detailed analysis of the underlying reasons.
Questions for authors
  1. Could authors clarify the relationship between Equation (2) and Equation (4)? In addition, please explain the necessity of the same formula appearing twice.
  2. How robust is DeDD to instance-dependent label noise? Could authors clarify this limitation and provide additional experiments to verify wether DeDD can generalize to instance-dependent label noise?
  3. Could authors evaluate DeDD under a range of synthetic noise rates on the same datasets to assess its robustness to more severe label corruption? In addition, could authors explain why a 30% corruption rate is used for Tiny-ImageNet, while 40% is used for the other datasets?
  4. Could authors provide a more detailed analysis of why DeDD underperforms the two-stage DATM baseline on CIFAR-10N at IPC 50, despite achieving strong results in most other settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.