- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MaskDD, a diffusion-based dataset distillation method. It keeps a pretrained DiT frozen and learns a separate channel mask for each class. They are trained with the standard denoising loss plus three regularizers, which push for sparsity, gates close to 0 or 1 (an entropy penalty), and less overlap between the masks of different classes. Training uses a rollback schedule inspired by lottery ticket rewinding. At a trigger epoch the method fixes the set of surviving channels, resets their logits to an earlier checkpoint, drops the sparsity penalty, and keeps refining only those gates. The final masks are thresholded to binary and used during DDIM sampling to generate the distilled set.
Critical strengths
- The idea of adapting a frozen generator through class-specific masks on its internal channels is novel in dataset distillation. It offers a lightweight and reusable way to bring generative models closer to the needs of downstream training.
- The paper goes beyond accuracy and examines what the learned masks capture. This gives some insight into how class structure is reflected inside a pretrained generator.
Critical weaknesses
- The training objective does not address the problem that motivates the paper. Section 1 argues that a generator optimized to produce realistic images is not necessarily optimized to produce useful classifier training data. Eq. (4) combines only the denoising loss with penalties on the masks themselves. No term depends on a classifier or on how well the generated images separate classes. Section 4.2 states that the effect on classification “is assessed experimentally rather than assumed from the loss.” The method therefore inherits the same objective it criticizes and gives no reason why a denoising-trained mask should yield more discriminative data.
- The paper does not show that the gains come from structural channel selection rather than from extra class-specific parameters. The mask adds class-specific parameters to a frozen model, the paper does not compare with lightweight tuning of the existing label conditioning, such as the class embeddings.
- The method adds several hyperparameters, and its accuracy is sensitive to them. These include the three loss weights in Eq. (4), where raising λent to 0.1 drops accuracy to 52.1% (Table 16). Across different rollback epochs E_anc and E_trg, accuracy ranges from 49.2% to 59.3% (Table 5).
- The paper is not reproducible. Many hyperparameters are not given, such as the learning rate, batch size, number of training epochs, mask initialization, annealing factor or the floor δ in the overlap penalty, and how many independent runs each number averages.
- The paper does not compare against recent diffusion-based distillation methods built for downstream utility, such as IGD, CaO2 and IGDS. These are discussed in related work but absent from the tables.
- The contribution of rollback and structural selection is not isolated. Section 5.3 states that its controls use different settings and an unspecified evaluator, so their gaps to the full method cannot be attributed to any one component.
- The training time is not reported.
Questions for authors
- Where exactly are the masks inserted in each DiT block? For example, after attention, inside the MLP, or on the residual stream? Did you compare different insertion points?
- In Table 4, MaskDD’s standard deviation with ResNet-50 and ResNet-101 is 2.0 and 1.6, versus 0.4 for MGD3, while it is 0.4 with ResNet-18. What causes this much higher variance?