- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CAGS, a training-free method for dataset distillation that assigns different classifier-free guidance strengths to different classes. The method estimates class complexity using cluster entropy and intra-class variance, and maps these statistics to class-specific guidance values through a sigmoid function. The authors also provide a theoretical analysis based on Gaussian mixture models to motivate the relationship between class complexity and guidance strength. Experiments show improvements over unguided generation and several published dataset distillation baselines.
Critical strengths
- The method is simple and does not require training additional models. It can be incorporated into an existing diffusion-based generation pipeline with relatively little modification.
Critical weaknesses
-
Adaptive guidance has already been explored in diffusion models, and the proposed method does not introduce a new generation or optimization framework. The novelty mainly lies in applying this idea to dataset distillation.
-
One of the main claims is that class-adaptive guidance is better than using a single guidance strength. However, the main experiments only include fixed λ=0.10, while CAGS uses guidance values within [0, 0.06]. Since the fixed λ=0.10 baseline performs worse than unguided generation, it is not clear whether the improvement comes from class adaptation or simply from choosing a smaller guidance strength.
-
The authors derive an optimal guidance strength using a manually defined utility function that balances discriminative benefit and diversity cost. However, this utility is not directly connected to the downstream classification objective used in dataset distillation.
Questions for authors
-
In Appendix D.6, the proposed complexity score has only a weak correlation with per-class accuracy gain, whereas separability shows a stronger correlation. How do the authors reconcile this observation with the claim that entropy and intra-class variance are the key factors for guidance adaptation?
-
The related work discusses several existing adaptive CFG methods, but the experiments do not directly compare with them. Could the authors include at least one representative adaptive guidance baseline under the same dataset distillation protocol?
-
The method relies on VAE features and K-means clustering to estimate class complexity. How sensitive are the results to the feature extractor, clustering method, and number of clusters? Would a simpler variance-based heuristic achieve comparable performance?
4 comments
We provide additional results on guidance and estimator choices to further address the concerns raised in your original review.
CAGS computes and caches guidance coefficients from cluster entropy and feature dispersion once, keeping CFG fixed and the generator frozen, with no added trainable parameters. The completed additional campaign (90 datasets; 270 classifier fits) adds paired replication and evaluations of the β-CFG adaptation and estimator choices.
1. Replicated gains and the β-CFG comparison (W2, Q2).
Default CAGS improves over CFG-only by +2.012 percentage points (pp), positive in all five generation replicates, with Holm p=0.044586.
Protocol: ImageNet-100, IPC=10, ConvNet-6, frozen DiT, 50 sampling steps, final classifier epoch 2000; all generation seeds 300-304 × classifier seeds 0-2, paired on A800. All rows were rerun in this cohort, separate from earlier cohorts and best-epoch results. Top-1 is mean ± sample SD across five generation means, each averaging three classifier seeds. Differences are default CAGS minus the row. Two-sided paired t-tests use Holm correction over all 34 planned contrasts.
| Condition | Top-1 (%) | CAGS minus row (pp) | Holm p |
|---|---|---|---|
| CFG-only | 22.23 ± 0.30 | +2.012 | 0.044586 |
| Default CAGS | 24.25 ± 0.60 | — | — |
| Fixed: CAGS mean | 24.55 ± 0.53 | -0.308 | 0.274678 |
| Selected fixed 0.03 | 24.34 ± 0.30 | -0.089 | 1.000000 |
| Selected CAGS | 24.15 ± 0.71 | +0.093 | 1.000000 |
| β-CFG adaptation | 12.56 ± 0.30 | +11.685 | 0.000134 |
| CAGS + β-CFG | 20.30 ± 0.37 | +3.947 | 0.001665 |
The CFG-only gain has an unadjusted 95% paired t CI of [1.292,2.732] pp (n=5, df=4). Mean-matched fixed guidance is higher in all five generations: CAGS minus fixed is -0.308 pp, with an unadjusted CI of [-0.497,-0.119] and Holm p=0.274678. This cohort replicates the complete configuration's benefit over CFG-only; an additional allocation advantage over matched fixed guidance remains unestablished.
CAGS exceeds the tested β-CFG adaptation and CAGS + β-CFG by +11.685/+3.947 pp, both significant after correction. β-CFG uses the frozen public preset (scale=7.5, a=2, b=3, gamma=1), adapted from SD/DDIM to DiT/DDPM with three guided channels, guidance-norm normalization and no beta-density peak normalization. Ordinary CAGS and fixed controls retain CFG=4. This comparison concerns that preset and adaptation; matched selection budgets and comparable CFG-scale selection remain to be evaluated.
Selected rows reuse the training-holdout choices without retuning: fixed 0.03 and CAGS weights 0.25/0.75, versus default 0.5/0.5. Default/selected CAGS gain +2.012/+1.919 pp over CFG-only (Holm p=0.044586/0.064445); their direct paired difference is +0.093 pp (Holm p=1). The original default retains its observed CFG-only benefit without adopting the selected weights.
2. Feature, clustering and K sensitivity (Q3).
The same cohort and statistical convention apply. We compare raw coefficients and coefficients matched to the default's mean.
| Complexity estimator | Raw Top-1 (%) | Mean-matched Top-1 (%) |
|---|---|---|
| Default VAE / auto K | 24.25 ± 0.60 | 24.25 ± 0.60 |
| ResNet-18 / auto K | 24.00 ± 0.29 | 24.28 ± 0.35 |
| VAE / K=5 | 24.11 ± 0.67 | 24.10 ± 0.70 |
| VAE / K=10 | 24.13 ± 0.45 | 24.16 ± 0.53 |
| VAE / PCA=16, auto K | 24.22 ± 0.67 | 24.12 ± 0.41 |
| VAE / Ward, K=10 | 24.13 ± 0.91 | 24.34 ± 0.84 |
Observed mean accuracies span 24.00-24.25% with raw coefficients and 24.10-24.34% after mean matching, ranges of 0.25/0.24 pp. Mode targets remain fixed; only complexity estimation and coefficient assignment change. Ward and K-means K=10 share K and PCA dimension; the default row is one shared reference.
All ten default-versus-estimator contrasts have Holm p=1 in the full 34-test family. The observed mean variation is limited across these tested choices. The dispersion-only control was reported earlier; this estimator table does not test a distinct covariance-trace heuristic.
3. Class-level evidence (Q1).
Across five generations, Pearson correlations with CAGS-minus-CFG-only gains are 0.133-0.312 for separability and -0.056 to 0.086 for complexity. Complexity correlations with CAGS-minus-fixed-mean gains range from -0.079 to 0.131. Each correlation uses 100 class accuracies after averaging classifier seeds within a generation. Separability has the stronger observed association with CFG-only gains; complexity correlations with the allocation contrast are small.
The additional experiments show a CFG-only gain across fresh generation seeds and limited observed mean variation across the tested estimators. These results support the practical value of CAGS as a simple, cached sampling rule in this setting. We hope its demonstrated utility and the explicitly stated comparison limits will both inform your assessment.
Thank you for your responses. I believe my original rating is fair and would prefer to keep it unchanged.
Thank you for considering our responses and clarifying your assessment. We respect your judgment and sincerely appreciate the time and effort you have devoted to reviewing our work. Your constructive comments have been valuable in guiding our revisions and improving the manuscript.
Thank you for recognizing the method's simplicity and ease of integration, and for your questions about novelty, controls and theory. We carefully checked the implementation and experiments and revised the manuscript, including the main overview figure, to clarify the method and address your concerns. We would greatly appreciate your consideration of the revised manuscript and look forward to further discussion.
1. Contribution relative to adaptive CFG (W1).
CAGS maps real-data cluster entropy and feature dispersion to class-specific mode-guidance coefficients, computed once without a learned predictor. With the generator frozen and CFG fixed at g=4, CAGS varies a separate update toward cached mode targets; adaptive CFG changes conditional/unconditional denoising. Our contribution concerns this distinct control variable for dataset distillation (Sections 2.3 and 3).
2. Is the gain due to class allocation (W2)?
CAGS improves over CFG-only across all three classifiers. Evaluation: ImageNet-100/IPC=10, final epoch 2000, generation seeds 100–104 × three classifier seeds.
Top-1 mean ± sample SD (%) across five generation replicates after averaging classifier seeds; architectures reuse datasets. Bold values mark the highest unrounded mean, not significance. pp denotes percentage points.
Paired gains over CFG-only are +2.245/+3.881/+3.041 pp, with Holm p=0.0299/0.00105/0.00666 (six comparisons per architecture). Fixed-mean differences have CIs spanning zero. Only ResNet-18 exceeds the three-permutation average (+0.260 pp, Holm p=0.00561), versus −0.231/−0.110 pp on ConvNet-6/ResNetAP-10 (Sections 5.2–5.4; Tables 1, 2, 6, 7).
Independent selection uses a 10% training holdout excluded from complexity/target estimation: 14 fixed and 13 CAGS candidates (81 fits). It selects fixed 0.03 and entropy/dispersion weights 0.25/0.75; 0.5/0.5 remains the default. The 300 reporting fits use seeds 200–204 × three classifier seeds, ImageNet-100/IPC=10/ConvNet-6, final epoch 2000. Default/selected CAGS gain +2.396/+2.368 pp over CFG-only (Holm p=0.0231/0.0372; 40-contrast family). Differences from selected fixed are −0.211/−0.239 pp (both Holm p=1); own-mean fixed and permutation-average contrasts are also inconclusive.
These controls show similar observed means to fixed guidance, without establishing an additional allocation benefit or statistical equivalence. Earlier validation-set use remains disclosed (Section 5.5; Appendix F.1; Tables 3, 8–10).
3. Would a simpler rule suffice (Q3)?
The two-factor default has the highest observed mean among these factor variants (seeds 200–204; same mean/SD convention). Mean matching controls for average guidance strength.
Bold marks the highest mean per accuracy column. Holm p compares default with each matched rule (40-contrast family). None of the six raw/matched factor contrasts is significant. We retain the compact default without claiming both factors are necessary (Table 4; Tables 9–10). Dispersion is mean Euclidean distance to the class centroid; covariance trace is untested (Section 3.2).
4. Theory and complexity interpretation (W3, Q1).
The bounded quadratic analysis characterizes the oracle allocation gap under clipped optima, with a weighted-square identity for interior optima. It provides intuition, not a classification-risk derivation or CAGS guarantee (Section 4). Appendix B includes the binary entropy counterexample and removes unsupported general entropy-monotonicity and separability-invariance claims.
The separability association remains exploratory. Complexity–gain correlations against fixed-mean guidance change sign across seeds on every architecture (Figure 3); CFG-only comparisons test a different intervention. These results do not establish unique roles for entropy/dispersion.
5. Direct adaptive-CFG and estimator tests (Q2–Q3).
The 18-condition campaign (seeds 300–304 × three classifier seeds) is incomplete (Appendix D.3). It tests beta-CFG adapted to DiT/DDPM, alone and with CAGS, plus feature/PCA/clustering/K changes with raw and mean-matched coefficients, keeping mode targets fixed. Beta-CFG uses one frozen published preset, not an equal-tuning-budget search. We will report completed, audited five-generation results.
Anonymous code: repository.