acceptodds
Withdrawn from ICLR 2027

PSM: Dataset Distillation via Difficulty-Aware Precise Statistical Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dataset distillation (DD) condenses a large original dataset into a small distilled dataset with high training utility. Decoupled statistical matching methods substantially reduce distillation time and memory overhead while achieving strong performance. However, they typically supervise all distilled samples using running statistics estimated from the entire original dataset. These statistics mainly capture the average feature distribution while overlooking differences in sample difficulty, limiting their ability to characterize the difficulty structure of the original data. To address this issue, we propose Precise Statistical Matching (PSM) by difficulty. After pretraining, PSM uses the Global Precision Score (GPS) to estimate image difficulty, ranks the samples within each class, and partitions each class into IPC (images per class) difficulty groups. During distillation, Statistics Updated Again (SUA) updates the teacher’s batch normalization (BN) running statistics through forward passes on original samples from each group, providing difficulty-specific supervision for the corresponding distilled batch. Meanwhile, Initial Sample Screening (ISS) initializes distilled samples using original images from the corresponding difficulty group, providing an effective starting point for precise matching. Experiments across multiple datasets and model architectures demonstrate that PSM broadens the difficulty range of distilled samples and improves downstream performance in most evaluated settings. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dataset distillation, statistical matching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FpffOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PSM improves decoupled dataset distillation by accounting for sample difficulty that existing methods ignore. It scores image difficulty with GPS, splits each class into difficulty groups, and uses group-specific BN statistics (SUA) and matching initializations (ISS) to supervise distilled samples. Experiments show PSM broadens the difficulty range of distilled data and boosts performance across datasets and architectures.

Critical strengths
  1. This paper is simle but effective ,easy to follow.

  2. The thought of sample difficulty in statistical matching-based methods is new to me, which maybe important for DD.

  3. The insight is intuitive and the experiments are robust.

Critical weaknesses
  1. Although the motivation regarding sample difficulty is intuitive, it still requires stronger experimental evidence or support from previous research findings. This is because earlier studies have also shown that under a smaller IPC, only simple samples are needed. This may be worth further investigation.

  2. I still have some concerns about the fairness and security of this type of DD paradigms. They rely on soft labels, and prior research has shown that under identical settings they can even perform worse than randomly selected images. In addition, they introduce the risk of teacher information leakage. Moreover, evaluation must use the same image augmentation scheme and training logic employed during distillation, which also compromises the flexibility of downstream tasks.

  3. PSM appears to be usable as a plug-in that can be integrated into existing methods, such as SRE2L and GVBSM. The corresponding experimental results should be presented

minor: line 68 :Imagenette->ImageNette

Questions for authors
  1. Regardless of the results presented in this paper, is there any relevant evidence or analysis showing that covering the range of difficulty in the original dataset is important for DD?

  2. Regarding ISS, for example, when IPC = 10, does it mean dividing the data into 10 difficulty groups of equal size, and then randomly selecting one sample from each difficulty group, or choosing the one with medium difficulty, as the initialization?

  3. Refer to Weaknesses 3. I still think that, under identical settings, a comparison with coreset-based methods is necessary regardless of whether the results are better or worse. Could the authors provide the results of methods such as random, k-center, and herding under this framework?

  4. Is the teacher statistics update performed in real time, or is each batch updated using the original statistics as an anchor?

Research on difficulty in DD is not novel, but the idea of directly combining difficulty with statistics and updating teacher statistics is insightful and may have a positive impact on DD research. Therefore, if the authors can properly address all of my concerns, I would be happy to raise my score.

Reviewer_yw1AOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose PSM, a decoupled dataset distillation method that assigns each distilled batch its own BatchNorm (BN) statistical target. Note that previous approaches, such as SRe2L match every batch to the same global BN statistics of the teacher. PSM ranks samples by difficulty (easy-hard), splits them into groups, and adapts the teacher’s BN statistics to each group.

Critical strengths

Existing decoupled methods (e.g., SRe2L, FADRM) match every batch to the statistics captured by BatchNorm layers after training. As a result, each sample batch has the same target. In contrast, the authors reformulate how to use the BatchNorm statistics for each batch. In more detail, each distillation batch has its own target. In other words, the BatchNorm statistics are updated to the difficulty-assigned group. The idea is simple, changes the supervision signal itself rather than adding a module, and is well motivated in principle. This is the main reason I see potential in the paper, although, as discussed in the weaknesses, the empirical evidence that it improves performance is not yet convincing.

Critical weaknesses

There is a discrepancy between the results of FADRM/FADRM+ reported in Table 1 and the results reported in the original paper. Note that the presented paper states: “For PSM, we follow the post-evaluation protocol of FADRM (290)”. For example, the results of FADRM on ImageNet-1k (IPC=10, ResNet-50) report 54.5% in the original paper, whereas 45.6% (-8.9 pp) is found in Table 1. This behavior is also found for FADRM+; for instance, the settings ImageWoof/ResNet-18/IPC=10 show 57.3% in the original paper and 47.2% in this paper. Other settings showed the same pattern. As a result, the claim that PSM and PSM+ outperform FADRM and FADRM+ cannot be presented as a general conclusion compared to the original paper on FADRM, considering PSM outperforms FADRM in 10 of 28 settings and PSM+ in 6 of 28. Two further observations suggest some differences in the evaluation pipeline (PSM vs FADRM). In the first case, the full-dataset accuracy of ResNet-18 in Tiny-ImageNet is 60.3% in this paper; however, FADRM reports 68.9%. In the second case, FADRM+ never reaches lower results than FADRM in the original paper, whereas the presented paper shows a few cases (for example, Tiny-ImageNet - ResNet 50).

Figure 3 shows the different difficulty distributions between the Original Dataset, PSM, and FADRM. Note that FADRM does not follow the original difficulty distribution of the original dataset; however, the result for CIFAR-100 (ResNet-18, IPC50) is higher at 71.0% (original paper) than 69.7% PSM. This comparison challenges the limitations of homogeneous supervision.

Questions for authors

Could the authors explain the reasons for the differences between the reported and published results of FADRM?

Reviewer_a5EfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that decoupled distillation methods, which match every distilled batch to the teacher's global BN statistics, produce distilled images with uniform difficulty, and proposes matching difficulty-specific statistics. The proposed method, PSM, ranks real images by a BN-statistics-based difficulty score (GPS) and splits each class into IPC difficulty groups. For each batch of synthetic images, it re-estimates the teacher's BN statistics on the corresponding group via forward passes (SUA) and initializes the batch with real images from that group (ISS). Further, a multi-teacher variant (PSM+) aggregates the rankings across teachers. The method is evaluated on CIFAR-10/100, Tiny-ImageNet, ImageNette, ImageWoof, and ImageNet-1K with ResNet-18/50 students.

Critical strengths

A clear observation that global BN statistics give every batch the same target. This is also supported by empirical analysis (Fig. 1b, Fig. 5).

Critical weaknesses
  1. Controlled comparisons are needed to show what drives the gains.
  • Grouping. SUA gives each batch a different target, which adds diversity regardless of how groups are formed, yet no random- or cluster-grouping control is reported. Also, if distilled-data difficulty (Fig. 1b, 3a, 5) is measured with GPS, the high correlation holds by construction, and an independent measure (e.g., EL2N [A] or forgetting) is needed.

  • Selection vs. distillation. In the soft-label setting, random real images and strong coresets match or outperform state-of-the-art distillation according to many recent works [B, C, D, E, F, G]. The key baseline, training directly on the GPS-selected initialization images under the same soft-label protocol, is missing.

  • Tuning. Gains are mostly ≤1% and within one standard deviation, with notable drops (e.g., ImageWoof ResNet-50 IPC 1: −5.1), while rfwdr_{fwd} and the screening method are tuned per dataset on test accuracy (Sec. 5.3, Fig. 4) and baselines use defaults.

  1. Limited novelty. It seems that the only novelty of this paper comes from the difficulty score, since grouping real images and supervising with a subset of real images for a group of synthetic images has already been explored (e.g., LPLD, DWA [H], BPS, DREAM). Initializing synthetic images with real ones is not novel at all since it has been the default practice in dataset distillation.

[A] Mansheej Paul, "Deep Learning on a Data Diet: Finding Important Examples Early in Training," NeurIPS 2021.

[B] Tian Qin, "A Label is Worth a Thousand Images in Dataset Distillation," NeurIPS 2024.

[C] Lingao Xiao, "Rethinking Large-scale Dataset Compression: Shifting Focus From Labels to Images," arXiv 2025.

[D] Zekai Li, "DD-Ranking: Rethinking the Evaluation of Dataset Distillation," NeurIPS 2026.

[E] Xinhao Zhong, "Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation," ICLR 2026.

[F] Priyam Dey, "Rethinking Dataset Distillation: Hard Truths about Soft Labels," CVPR 2026.

[G] Trisha Mittal, "Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?," NeurIPS 2026.

[H] Jiawei Du, "Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment," NeurIPS 2024.

Questions for authors
  • What are the results with random or cluster-based groups?

  • How is distilled-data difficulty measured in Fig. 1b, 3a, and 5?

  • How does PSM compare with (a) random real images, (b) the GPS-selected initialization images without optimization, and (c) standard difficulty-based coresets (e.g., forgetting- or EL2N-based), all trained with the same teacher soft labels and schedule?

  • How does PSM differ from, and compare with, LPLD's class-wise BN statistics and DWA's batch-specific weight perturbation under the same FADRM pipeline?

  • How were rfwdr_{fwd} and the screening method selected, and how does one fixed setting perform across datasets?

  • At IPC 1 only ISS is active, yet ImageNette drops by 3.2. Is the selected screening method not effective here?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.