acceptodds
Withdrawn from ICLR 2027

Dataset Distillation via Saliency-Guided Initialization and Adaptive Gaussian Mixture Frequency Sampling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dataset distillation aims to compress a large training set into a compact synthetic set while preserving its training utility. However, in existing distribution matching methods, representation construction and frequency sampling are typically treated as independent components, which may introduce alignment bias between different representation states. In this paper, we propose a novel dataset distillation framework, termed Saliency-guided Initialization with Gaussian Mixture Sampling (SIGMS), which addresses both weak synthetic-data initialization and the limited adaptability of frequency sampling to varying representation states through two complementary mechanisms. Firstly, we propose SaliPatch, a saliency-guided patch initialization module that selects and stitches informative patches from real images, reducing background interference and improving the quality of synthetic initialization. Secondly, we jointly introduce Stage-Conditioned Feature Interpolation and Adaptive SampleNet, where intermediate representations are constructed between the frozen initial and expert models, while frequency sampling is adaptively adjusted to the corresponding representation states through a shared stage variable, thereby reducing the mismatch between feature representations and frequency probing during distribution alignment. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that SIGMS achieves state-of-the-art performance at IPC=1 and maintains competitive performance at higher IPC settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dataset Distillation, Distribution Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CQdnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SIGMS, a dataset distillation framework that combines SaliPatch initialization with stage-conditioned characteristic-function matching. Its key motivation is that existing methods typically treat representation-state construction and frequency-space probing as independent components, which may cause a mismatch between feature representations and sampled frequency arguments. SIGMS addresses this issue by coupling Stage-Conditioned Feature Interpolation with Adaptive SampleNet through a shared stage variable. It consistently outperforms NCFM on several datasets at IPC=1, demonstrating its effectiveness under highly compressed settings. However, SIGMS requires approximately 3.5 times the distillation time of NCFM, while providing only modest improvements at higher IPC settings. Overall, this is a reasonably well-evaluated but incremental work with limited computational efficiency.

Critical strengths
  1. Existing methods typically treat representation-state construction and frequency-space probing independently, so the sampled frequency arguments may not adapt well to different representation states. Coordinating the two through a shared stage variable is a reasonable and insightful idea.

  2. SIGMS consistently outperforms its direct baseline, NCFM, across multiple datasets at IPC=1. The ablation results also show that shared stage conditioning is more effective than using Adaptive SampleNet independently, providing empirical support for the proposed stage-coordinated design.

Critical weaknesses
  1. SIGMS incurs substantial computational overhead. On CIFAR-10 at IPC=1, its distillation time increases from 3.02 hours for NCFM to 10.65 hours—approximately 3.5 times longer—while the improvements at higher IPC settings are generally modest.

  2. The technical novelty appears limited. SaliPatch is closely related to existing patch-based initialization methods, while feature interpolation and learnable frequency sampling build on DANCE and NCFM, respectively. The main distinction of SIGMS is the use of a shared stage variable to coordinate these components, and the significance of this contribution requires stronger justification.

Questions for authors
  1. SIGMS requires approximately 3.5×3.5\times the distillation time of NCFM. Could the authors provide a comparison under a matched computational budget or a similar sampler parameter count to determine whether the gains mainly result from longer optimization and a larger Adaptive SampleNet?

  2. According to Eqs. (7)–(9), the shared stage variable λ\lambda primarily affects frequency sampling through the temperature TλT_{\lambda} and interpolation coefficient ρ(λ)\rho(\lambda), while the means and variances of the Gaussian mixture do not appear to be directly conditioned on λ\lambda. Could the authors clarify how Adaptive SampleNet adapts to different representation states and how this mechanism differs from a predefined stage schedule?

Reviewer_GErnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a DM-based dataset distillation method built upon NCFM. The proposed method consists of three components: SaliPatch initialization, stage-conditioned feature interpolation, and adaptive SampleNet. SaliPatch selects and stitches salient patches using CAM to initialize each synthetic image. Stage-conditioned feature interpolation matches feature distributions using a network interpolated between a randomly initialized model and a fully trained model. Similarly, adaptive SampleNet interpolates between frequencies sampled from a GMM and a base Gaussian distribution. The interpolation ratio is determined by the same stage variable. Experimental results on several datasets demonstrate improvements over NCFM.

Critical strengths
  1. The paper effectively integrates existing techniques (RDED, DANCE, and NCFM) into a unified framework.
  2. The experimental results demonstrate clear performance improvements over previous state-of-the-art methods.
  3. The paper is clearly written, and the proposed method is easy to follow.
Critical weaknesses
  1. Limited novelty. The proposed method effectively combines existing techniques into a single pipeline, with initialization resembling RDED, interpolation resembling DANCE, and sampling based on NCFM. However, it remains unclear what unique technical contributions the paper introduces beyond this combination.

  2. Insufficient justification. The motivations behind coupling and sampling are not sufficiently explained. In addition, the paper does not justify the choice of scheduling functions, such as λ1.2\lambda^{1.2}, λ1.5\lambda^{1.5}, and 1−0.5λ1-0.5\lambda, which appear to be heuristic.

Q1 Please justify why coupling should be fundamentally better than using the components without coupling.

Q2 Please clarify the exact experimental settings in Table 4. Specifically, what do independent and shared λ\lambda mean?

  1. Unclear novelty of SaliPatch initialization. The novelty of SaliPatch initialization is unclear, and its benefits are not sufficiently validated within the same pipeline.

Q3 Please provide an ablation study comparing SaliPatch selection with alternative selection strategies, such as random selection, Herding, K-center, and RDED's realism-based selection.

  1. Unclear fairness of comparison. It is unclear whether the experimental settings are fairly matched to NCFM.

Q4 SIGMS appears to use four times as many frequency arguments as NCFM. Therefore, an ablation study on the number of frequency arguments is needed to ensure a fair comparison.

Questions for authors

Please address the questions raised in Weaknesses 2, 3, and 4.

Reviewer_CWRVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SIGMS, a dataset distillation framework combining saliency-guided initialization with stage-conditioned characteristic-function matching. SaliPatch selects salient, diverse patches from real images and combines them into initial synthetic images. During optimization, features are interpolated between frozen initial and expert networks, while the same stage variable controls Gaussian-mixture frequency sampling. Experiments report improvements over the manuscript’s NCFM baseline, particularly at one image per class (1-IPC).

Critical strengths
  1. The motivation is reasonable. The initialization quality and the choice of feature representations can substantially affect distillation. Improving the synthetic starting point while adapting distribution matching is a reasonable approach to dataset distillation.

  2. The reported CIFAR-10 results are promising. Table 1 reports improvements over the listed NCFM baseline of 3.4, 1.9, and 1.2 percentage points at IPC=1, 10, and 50, respectively. The consistent improvement across compression levels validates the superiority of SIGMS over baselines.

  3. This paper makes sufficient ablations. Tables 3–5 examine initialization, sampling, shared conditioning, patch granularity, and mixture size. In particular, Table 4 holds initialization fixed and reports improvements from shared conditioning across three datasets. Reporting runtime and memory costs in Table 8 also helps assess the practical trade-off.

Critical weaknesses
  1. Several NCFM values in this manuscript differ from its official CVPR version. NCFM discusses optional soft-label generation and dataset fine-tuning. However, the submission does not explain whether its baseline is rerun, which processing steps are retained, or why the values differ. Consequently, the apparent IPC=1 gains on CIFAR-100 and Tiny ImageNet cannot yet establish an advance over published NCFM. The authors should distinguish published results from reproductions and provide a fully matched comparison.

  2. The stated objective does not fully specify the implemented algorithm. Equation (1) describes class-conditional matching, but Eq. (10) and Appendix A.6 omit explicit class conditioning. More concerningly, Appendix A.1.2 and Table 6 introduce calibration and PLTDA weights, although the corresponding losses and their relationship to Eq. (10) are not defined.

  3. One core contribution of this paper is to use a stage variable λ\lambda to control the strength of frequency-aware matching. A previous method [1] has explored this approach, but the authors do not cite or discuss the related work.

  4. Appendix A.1.6 describes repeated evaluation-network initialization, but does not give the number of evaluations or establish that independently distilled datasets were generated. Evaluation variability and distillation variability are different sources of uncertainty. This distinction matters particularly for the ImageNet-subset IPC=10 gains of only 0.3–0.5 points, accompanied by reported standard deviations of approximately 0.5–1.5 points.

[1] Understanding Dataset Distillation via Spectral Filtering. ICLR 2026.

Questions for authors

I am not sure whether the LaTeX template meets ICLR's requirements. The TeX styles are different from other papers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.