acceptodds
Withdrawn from ICLR 2027

MaskDD: Sculpting Discriminative Features for Dataset Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dataset Distillation (DD) aims to compress large-scale datasets into a compact set of synthetic samples while preserving downstream performance. Recent diffusion-based DD methods improve scalability and visual quality by leveraging pretrained generative priors, but they suffer from a fundamental misalignment between generative realism and discriminative utility. Specifically, diffusion models optimized with maximum likelihood estimation tend to emphasize dominant generic features while overlooking the fine-grained discriminative signals essential for accurate categorization. Consequently, existing methods often generate visually realistic yet weakly discriminative synthetic datasets. To address this issue, we propose MaskDD, a diffusion-based dataset distillation framework that jointly optimizes data synthesis and model structure. Our method introduces a class-aware masking mechanism to selectively preserve task-relevant feature channels and suppress redundant representations. In addition, we develop a dynamic rollback strategy inspired by the Lottery Ticket Hypothesis to stabilize optimization and preserve favorable subnetworks during discriminative refinement. Extensive experiments across multiple datasets and architectures demonstrate that MaskDD consistently outperforms existing state-of-the-art methods in dataset distillation.

Primary area:
generative models
Keywords:
Dataset Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AdUjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MaskDD, a diffusion-based dataset distillation method. It keeps a pretrained DiT frozen and learns a separate channel mask for each class. They are trained with the standard denoising loss plus three regularizers, which push for sparsity, gates close to 0 or 1 (an entropy penalty), and less overlap between the masks of different classes. Training uses a rollback schedule inspired by lottery ticket rewinding. At a trigger epoch the method fixes the set of surviving channels, resets their logits to an earlier checkpoint, drops the sparsity penalty, and keeps refining only those gates. The final masks are thresholded to binary and used during DDIM sampling to generate the distilled set.

Critical strengths
  1. The idea of adapting a frozen generator through class-specific masks on its internal channels is novel in dataset distillation. It offers a lightweight and reusable way to bring generative models closer to the needs of downstream training.
  2. The paper goes beyond accuracy and examines what the learned masks capture. This gives some insight into how class structure is reflected inside a pretrained generator.
Critical weaknesses
  1. The training objective does not address the problem that motivates the paper. Section 1 argues that a generator optimized to produce realistic images is not necessarily optimized to produce useful classifier training data. Eq. (4) combines only the denoising loss with penalties on the masks themselves. No term depends on a classifier or on how well the generated images separate classes. Section 4.2 states that the effect on classification “is assessed experimentally rather than assumed from the loss.” The method therefore inherits the same objective it criticizes and gives no reason why a denoising-trained mask should yield more discriminative data.
  2. The paper does not show that the gains come from structural channel selection rather than from extra class-specific parameters. The mask adds class-specific parameters to a frozen model, the paper does not compare with lightweight tuning of the existing label conditioning, such as the class embeddings.
  3. The method adds several hyperparameters, and its accuracy is sensitive to them. These include the three loss weights in Eq. (4), where raising λent to 0.1 drops accuracy to 52.1% (Table 16). Across different rollback epochs E_anc and E_trg, accuracy ranges from 49.2% to 59.3% (Table 5).
  4. The paper is not reproducible. Many hyperparameters are not given, such as the learning rate, batch size, number of training epochs, mask initialization, annealing factor or the floor δ in the overlap penalty, and how many independent runs each number averages.
  5. The paper does not compare against recent diffusion-based distillation methods built for downstream utility, such as IGD, CaO2 and IGDS. These are discussed in related work but absent from the tables.
  6. The contribution of rollback and structural selection is not isolated. Section 5.3 states that its controls use different settings and an unspecified evaluator, so their gaps to the full method cannot be attributed to any one component.
  7. The training time is not reported.
Questions for authors
  1. Where exactly are the masks inserted in each DiT block? For example, after attention, inside the MLP, or on the residual stream? Did you compare different insertion points?
  2. In Table 4, MaskDD’s standard deviation with ResNet-50 and ResNet-101 is 2.0 and 1.6, versus 0.4 for MGD3, while it is 0.4 with ResNet-18. What causes this much higher variance?
Reviewer_Rww6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission adapts a frozen diffusion transformer with class-specific channel gates: these gates are trained with a denoising loss with some penalties. The rollback procedure selects a surviving channel support, rewinds its logits to an earlier checkpoint, then resets the optimizer state, and refines the active gates. The final masks generate compact synthetic datasets. Experiments include classification on ImageNet and its subsets. The mathematical analysis characterizes gate gradients and bounds soft-to-hard mask error.

Critical strengths

The class-specific gates provide a reusable way to adapt a pretrained diffusion, yielding encouraging classification results: at ImageWoof IPC 50, MaskDD improves accuracy over MGD3 across three evaluators. Reusing the ImageNet-1K synthetic dataset across resnet and vit further supports its versatility. The analysis distinguishes gate-rounding error from denoiser sensitivity and adjusts the interpretation of overlap for the high channel-retention rate, which makes it more interpretable.

Critical weaknesses

I think there are some misalignments between the method and EXPs:

  • The incomplete experiments and ablations prevent a reliable assessment of the main contribution. Appendix B (Mainly about the EXP settings) omits details on optimization, sampling, labeling, and downstream training. This limits verification of matched evaluation and statistical reliability.
  • The structural and rollback contributions are not isolated: Table 13 changes sparsity coefficients and leaves the control evaluator unverified, while its no-rollback model has approximately 85.4% sparsity versus 5.6% for the full method, making capacity loss an alternative explanation. Table 19 offers an encouraging comparison but still bundles rewinding, removal of sparsity pressure, support locking, and optimizer reset. Therefore, I encourage the authors to do matched continuous-scaling controls and separate rollback-component ablations to establish what produces the improvement.
Questions for authors
  1. How can the proposed discriminative feature-selection mechanism be distinguished from additional class-conditioning capacity, given that eq5 rewards reconstructing class-correlated backgrounds as well as foreground content?
  2. Eq8 penalizes overlap equally for all class pairs, including semantically related classes. How does the objective distinguish harmful redundancy from useful shared features, and what explains the density-adjusted semantic sharing observed in 5.4?
  3. Eq10 rewinds logits while retaining the current temperature and a fixed support. So, what makes the anchor state less saturated, and how is gradient recovery distinguished from the effects of removing sparsity pressure and resetting AdamW?
  4. Can a control matching class-conditioning capacity, final sparsity, and downstream supervision isolate the benefit of learned support selection from continuous scaling or random channel removal? This would clarify which structural operation is responsible for the reported gains.
Reviewer_JWp8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a dataset distillation approach that leverages a pretrained diffusion model. Their method, MaskDD, learns class-specific channel masks inside a frozen diffusion transformer, using a denoising objective with regularizers for sparsity, binary polarization and cross-class overlap. In addition, a rollback mechanism restores the mask logits to an earlier checkpoint and removes the sparsity constraint for further refinement. The method is evaluated on ImageNet-1K and several of its subsets at 256×256 resolution, using different classifier architectures and image budgets.

Critical strengths
  • The method learns only class-specific channel gates, while the DiT stays frozen. Unlike latent-optimization approaches such as GLaD and LD3M, there is no per-image optimization, and the same masks are reused for every sample and timestep. Sampling time and memory stay almost unchanged. This makes the approach lightweight.

  • The analysis of the learned masks is useful because it gives some indication of what information they encode. After correcting for mask density, the overlap between class masks correlates with the WordNet taxonomy, DINOv2 prototypes and ResNet-18 confusion. The authors evaluate this condition with permutation tests and Holm correction. This gives some evidence that the masks capture class structure rather than noise.

Critical weaknesses
  • Two recent generative distillation methods are neither cited nor compared: DAP (ICLR 2026) and LGD (CVPR 2026). Both report higher accuracy than MaskDD on ImageWoof. For example, with ResNet-18 at 50 IPC, DAP reports 63.2% and LGD 66.2%, compared with 59.0% for MaskDD. In addition, IGD, which is cited but not compared, reports higher or equal accuracy in the shared ImageWoof settings. The claim of improvement over the state of the art is not well supported.

  • Some parts of the paper were hard to follow because important details for reproducibility are missing. The insertion points of the masks within the DiT blocks are not specified, nor are the mask initialization and training settings. The masks are also kept fixed across all denoising timesteps, and this choice is not evaluated. Finally, each term of Eq. 4 is evaluated only one at a time on a single dataset and budget (Tables 15–17), with no joint analysis of the terms.

Questions for authors
  • Could the authors compare MaskDD with DAP (ICLR 2026), LGD (CVPR 2026), and IGD under the standard hard-label protocol on ImageWoof? IGD, IGDS, and CaO2 are cited but not evaluated. Is there a reason they were left out?

  • Where are the masks inserted inside the DiT blocks (after attention, after the MLP, or at the block output)? Are all blocks masked? Was the placement varied?

  • What motivates keeping the masks fixed across all denoising timesteps? Did the authors try timestep-dependent masks, or apply the masks only within a range of timesteps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.