acceptodds
Withdrawn from ICLR 2027

CAG: Contraction-Aware Guidance for Diffusion Dataset Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent diffusion-based dataset distillation has emerged as a promising approach for constructing compact training sets from pretrained diffusion models. Under extremely low sample budgets, the semantic fidelity and class discriminability of generated samples are important determinants of its effectiveness. Existing training-free methods therefore commonly employ classifier-free guidance (CFG) with large guidance scales. However, although strong CFG improves conditional fidelity, it also sacrifices mode coverage and intensifies late-stage within-mode contraction, causing synthetic samples to share redundant local structures and provide insufficiently complementary training signals. To resolve this tension, we propose Contraction-Aware Guidance (CAG), a training-free framework that introduces timestep-gated manifold correction during inference. CAG reconstructs clean estimates from noisy sampling states, constructs a contraction-aware objective from feature variance, and activates the resulting correction only during late-stage denoising. This design counteracts excessive trajectory concentration while preserving the class semantics established by CFG, thereby promoting richer fine-grained and intra-class variation. Extensive experiments demonstrate improved diversity and cross-architecture generalization, with state-of-the-art performance across multiple extremely low-budget dataset-distillation benchmarks.

Primary area:
generative models
Keywords:
Dataset Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KVT2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Contraction-Aware Guidance (CAG), a training-free sampling method for generative dataset distillation using diffusion models (DMs). The authors observe that although stronger classifier-free guidance (CFG) improves the class consistency of generated images, it can reduce intra-class diversity during the later stages of denoising, thereby limiting the complementary training information contained in a small synthetic dataset.

CAG preserves the early CFG sampling trajectory and modifies the predicted clean latent only during the later denoising steps. Experiments on five ImageNet-derived benchmarks show moderate performance improvements, albeit at the cost of increased generation time.

Critical strengths

The authors make an interesting observation that, although stronger classifier-free guidance (CFG) improves the class consistency of generated images, it can reduce intra-class diversity during the later stages of denoising. This finding is insightful and may inspire further research into the dynamics of guidance in diffusion-based dataset distillation. The paper also proposes a simple and intuitive approach to mitigate this issue.

Critical weaknesses
  1. The paper is highly related to Diffusion Models as Dataset Distillation Priors (DAP) [1], an ICLR 2026 paper that also uses training-free, feature-based guidance to modify the reverse diffusion process for dataset distillation. Although the two methods optimize different feature objectives, they share a closely related problem setting. However, the authors neither cite nor discuss DAP, leaving the novelty of CAG and its distinction from prior work insufficiently clarified.

  2. The paper is often difficult to follow due to its dense and overly qualified writing. For example, the fifth paragraph of the Introduction introduces several caveats and distinctions before clearly communicating the main idea, which obscures rather than clarifies the contribution.

[1] D. Su, H. Wu, H. Chen, Y. Shi, Y. Wang, X. Ye, and J. Zhu. “Diffusion Models as Dataset Distillation Priors.” ICLR 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/2c7b9949db45ccc7fa5bf857a37330da-Abstract-Conference.html

Questions for authors
  1. CAG and DAP [1] substantially overlap in their methodological framework and are evaluated on four common datasets. Could the authors provide a controlled comparison with DAP or explain why a fair comparison is infeasible?

  2. In Figure 1, the “CFG only” and “CAG” legends are placed awkwardly. Could the authors revise the legend placement and overall layout to improve readability?

[1] D. Su, H. Wu, H. Chen, Y. Shi, Y. Wang, X. Ye, and J. Zhu. “Diffusion Models as Dataset Distillation Priors.” ICLR 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/2c7b9949db45ccc7fa5bf857a37330da-Abstract-Conference.html

Reviewer_MN3oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Contraction-Aware Guidance (CAG), an inference-stage correction method for diffusion-based dataset distillation. Its core idea is to perform gradient ascent on the predicted clean latent according to the variance of single-image feature responses in a frozen visual encoder during late-stage sampling, and then feed the corrected result back into the original sampler through noise reparameterization. The generator, decoder, and encoder all remain frozen, and the method does not require explicit interactions between samples.

The paper evaluates classification performance on five ImageNet-derived benchmarks and reports diversity, class semantics, and generation cost. In particular, Table 6 reports, under the ImageWoof, IPC=10, ResNetAP-10 setting, that accuracy increases from 34.7% to 42.6%, LPIPS increases from 0.55 to 0.63, and class consistency decreases by 0.7 percentage points. The method has practical potential, but incomplete experimental configurations and records, insufficiently matched protocols for the main comparisons, and limited mechanism validation reduce the persuasiveness of the current evidence.

Critical strengths

1.The method is simple and easy to integrate into existing generation pipelines: CAG applies the correction to the predicted clean latent without updating model parameters and without backpropagating through the denoiser or the full sampling trajectory. The paper clearly explains the gradient path, temporal gate, and noise reparameterization, and the algorithmic design is relatively clear. Its combination with existing methods such as Minimax also demonstrates its compatibility as a sampling-stage component.

2.The joint evaluation focuses on the actual training value of synthetic data: Table 6 simultaneously reports diversity, semantic consistency, and downstream classification accuracy. Under the control conditions stated in the paper, CAG improves over original CFG by 7.9 percentage points and exceeds the strongest listed alternative, early-stop CFG, by 5.1 percentage points. Random noise has higher LPIPS but lower classification performance, and this contrast helps illustrate the distinction between perceptual dispersion and training utility.

3.The boundaries of the conclusions and the additional cost are explained relatively clearly: The authors explicitly distinguish single-image feature dispersion from cross-sample diversity, acknowledge that the fixed gate does not perform online contraction detection, and distinguish literature comparisons from controlled experiments. The paper also reports runtime and memory growth, allowing readers to assess the trade-off between requiring no additional training and increasing inference computation.

Critical weaknesses

1.Experimental configurations and result records are incomplete, constituting a major reproducibility limitation: Appendix A.1 explicitly states that per-dataset and per-IPC parameter adjustments, encoder preprocessing and layer-index conventions, generation seeds, downstream optimizers, augmentation methods, and training lengths are not fully specified. These settings directly affect the correction gradient and classification accuracy. In addition, the standard deviation records in Table 11 are inconsistent, and the evaluator and full configuration are unclear; Appendix E also states that some metrics lack complete computation details and that pairing claims have not been verified against generation manifests. Therefore, although the update rule is clear, the complete experimental pipeline is still insufficient for exact reproduction.

2.The main benchmark comparisons cannot yet reliably isolate CAG's contribution: Many baselines in Tables 1–3 are quoted from the literature, and the sampling and evaluation protocols are not confirmed to be consistent. In particular, in the MGD³ integration comparison on ImageNet-100, the quoted baseline uses DDPM by default, whereas the CAG integration description involves DDIM. Therefore, the reported differences may simultaneously include the effects of the sampler, checkpoint, and classifier training configuration. On ImageNet-1K, both budgets improve by 0.9 percentage points, but the pairing status of the comparison and the per-run differences are unclear, which is insufficient to judge the stability of this improvement.

3.The connection between the single-image objective and effective cross-sample diversity remains insufficient: The variance in Eq. (2) can increase through channel response magnitude, texture, background, or contrast changes, and does not necessarily produce complementary training information. The cross-sample LPIPS, feature variance, and classification results in Table 6 provide empirical support, but they still cannot sufficiently distinguish improved semantic coverage from superficial appearance changes. Figure 2 shows baseline trajectories under different CFG strengths and does not directly verify CAG's effect on the contraction phenomenon discussed. Therefore, the current evidence supports the method's utility more strongly than it supports the specific anti-contraction mechanism.

4.The representation independence and statistical support of the diversity diagnostics are limited: The default correction objective uses VGG-16, while LPIPS also uses the VGG feature family, and some other diagnostics also rely on VGG. Although the evaluation metric is not directly optimized, sharing the representation may still introduce bias. Downstream classification accuracy provides independent utility evidence, but it cannot alone prove the source of the diversity improvement. The sample size, pairing method, class mapping, aggregation process, and uncertainty of the generation metrics are also not fully reported, limiting judgment of the degree of semantic preservation.

5.Key ablations and simple alternative strategies have not been sufficiently controlled: Some default results in Tables 7, 9, and 10 are reused from Table 1, and configuration consistency has not been verified, so they cannot reliably isolate the effects of encoder choice, pretraining, or objective space. Feature activation scales are not normalized across layers, which may also cause different objectives to correspond to different effective correction strengths. Meanwhile, the parameters and tuning procedures for strategies such as lowering CFG, guidance decay, and random noise are not specified. Considering that CAG's runtime is approximately 1.85 times that of the baseline, its cost-benefit relative to sufficiently tuned cheap alternatives has not yet been clearly established.

Questions for authors
  1. Experimental Configuration and Randomness: Can you provide the complete generation and classifier training configuration for Table 6, as well as the per-run results? Did the three runs each regenerate the synthetic dataset separately, or only retrain the classifier on the same dataset? Can the stated pairing conditions be confirmed through generation manifests or logs?

  2. Comparison Protocols: Are the checkpoints, samplers, and downstream training protocols actually used for MGD³ and MGD³+CAG identical? Do Minimax and Minimax+CAG on ImageNet-1K use the same teacher checkpoint, soft-label pipeline, and paired seeds? If paired runs exist, what are the per-run accuracy differences and their uncertainty?

  3. Parameter Selection and Alternative Strategies: What are the strength of Lower CFG, the schedule for CFG decay, the stopping point for early-stop, and the magnitude of random noise, respectively? What selection criteria were used for these parameters and for CAG's η and gate position, and were they selected using the validation set on which final accuracy is reported?

  4. Ablation Validity and Record Consistency: Did the experiments with different encoders and objective spaces control for the effective correction magnitude, or perform a comparable strength search? Which evaluator and full configuration does Table 11 correspond to, what should the standard deviation be at the shared setting, and on what basis was the default η=0.09 determined?

  5. Diversity Evidence: How many samples underlie the generation metrics in Table 6, and how are they paired and aggregated across classes? Are there existing results evaluating the same batch of samples with a different feature family that support that the diversity improvement is not mainly due to texture or contrast changes under the shared VGG representation?

Reviewer_vWsFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CAG, a training-free sampling correction for diffusion-based dataset distillation. In the second half of sampling, it decodes the predicted clean latent, computes the variance of VGG-16 features within that single image, takes one gradient-ascent step on the clean latent, and feeds the result back into the solver. The goal is to offset the loss of within-class variation caused by strong CFG. Experiments on ImageWoof, ImageNette, ImageIDC, ImageNet-100 and ImageNet-1K add CAG to DiT, MGD3 and Minimax, mostly comparing against numbers quoted from prior papers. The main controlled result is on ImageWoof at IPC 10, where CAG raises ResNetAP-10 accuracy from 34.7% to 42.6% and beats lower CFG, CFG decay, early-stop CFG and random noise.

Critical strengths
  1. The method is simple and easy to plug in. It needs no training, works on top of DiT, MGD3 and Minimax, and its cost is reported plainly (1.83s vs. 0.99s per image on an RTX 4090).
  2. The controlled comparison in Table 6 is well designed. It uses the same checkpoint and seeds, compares against lower CFG, CFG decay, early-stop CFG and random noise, and reports diversity, class consistency and accuracy together. It shows that more diversity alone does not help: random noise and lower CFG have higher LPIPS but lower accuracy. If this result holds, the gain from 34.7% to 42.6% is large.
Critical weaknesses
  1. The reported results cannot be verified. The appendix says that per-setting parameter changes were made but "not enumerated", that seeds and training details are "not fully specified in the available record", that Figure 2 was "recovered from an earlier vector figure", that standard deviations in Table 11 were dropped because "the available summaries disagree", and that pairing claims "have not been checked against generation manifests." The headline number also does not match the authors' own sweep: Table 11 peaks at 38.47% at almost the default setting, while Table 6 reports 42.6%. The manuscript reads as if it was rewritten from incomplete records, possibly with heavy AI help, and the authors cannot vouch for their own numbers.
  2. Apart from Table 6, the evidence is weak, and the method is not new. Most baselines are quoted from other papers under protocols the authors call "unverified". The average gains over MGD3 and Minimax (1.2 and 1.7 points) are mostly within one standard deviation. The IPC-50, encoder and objective ablations all reuse an unpaired reference number. The method itself is standard training-free gradient guidance on the clean estimate, in the same family as Universal Guidance and FreeDoM, neither of which is cited. The name is also misleading: the gate is fixed at step 25, and the paper admits it does not detect contraction.
Questions for authors

Clarifications:

  1. Which evaluator and settings give 42.6% in Table 6, and why does Table 11 top out at 38.47% at nearly the same setting?
  2. Please list all per-dataset/IPC parameter changes, seeds, and the classifier training setup.
  3. Can you rerun DiT, Minimax and MGD3 in your own pipeline with paired seeds, instead of quoting published numbers?
  4. The objective works on a single image. Why does it increase diversity across samples? Could the gain come from sharper or higher-contrast images?
  5. How does CAG differ from existing training-free guidance on the clean estimate, such as Universal Guidance or FreeDoM?
  6. What was the "conflicting classifier-accuracy series" for the CFG sweep mentioned in Appendix A.1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.