acceptodds
Under review as a conference paper at ICLR 2027

Diffusion as a Multi-Scale Discriminator: Rethinking Noise Scales for Diffusion Model Distillation

Abstract

We propose MuSD (Multi-Scale Discriminator), a simple framework for diffusion distillation that uses a frozen diffusion teacher to guide adversarial training. We train lightweight discriminator heads on the teacher's features at multiple resolutions, without training an auxiliary score network or using a separate pretrained feature extractor. Prior work that uses the teacher as the discriminator biases its noise levels toward high noise to preserve global coherence. We observe that fine-scale structure in teacher features survives mainly at low noise, whereas the coarse focus that features acquire at high noise is also available from coarse readouts at low noise. Building on this, we derive a Beta noise sampler as the maximum-entropy density under a corruption budget, which favors low noise while keeping positive density over the whole noise interval, and we show that both properties are needed. On CIFAR-10, AFHQv2, FFHQ, and ImageNet, MuSD achieves competitive or better one-step FID than the baselines with fewer training images and lower GPU memory use. It applies to both U-Net and Transformer teachers, including EDM2-S and DiT-XL/2, where the one-step student surpasses its 250-step teacher, and extends to two-step text-to-image generation with SD3.5-Medium.

Primary area:
generative models
Keywords:
Diffusion models, One/few-step generation, Sampling dynamics, Adversarial training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2C7rOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MuSD, which uses a frozen diffusion teacher for adversarial distillation. The authors further use a Beta sampler derived as the maximum-entropy density under an expected "corruption cost" budget, which favors low noise. For text-to-image, a frozen-teacher CFG direction is added to the adversarial gradient. The method is tested on CiFAR-10 and other image sets.

Critical strengths
  1. I like the finding about the low-noise preference, which not only inspired the noise sampler used in MuSD, but can also be interesting for other reasons.
  2. The derivation of Beta noise sampler is elegant.
  3. Efficiency gains are significant.
Critical weaknesses
  1. The Text-to-image results are the weak spot. Specifically, FID results are poor. MuSD's two-step COCO-30K FID (25.2) is the worst among distilled models (Table 3); There is a tradeoff between FID and preference. The paper should discuss this trade-off and the large FID gap.
  2. The mechanism evidence is weak. Probing via linear reconstruction of structure (Appendix A.1) is a reasonable proxy, but it’d be better to show a direct measure of discriminative information content.
  3. It’s not clear how the method can be sclaed up.
Questions for authors

Can the authors show the discriminator's classification AUC (real vs generated) as a function of noise level? This would be a good test of whether low-noise features carry more information.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2C7r

We thank the reviewer for the positive assessment and especially for suggesting the AUC analysis, which became one of the clearest pieces of evidence in the revision.

W1. Text-to-image FID. At the same budget of 162K images, removing the teacher CFG direction (w = 0) lowers COCO-30K FID from 24.4 to 16.15, below the 40-step teacher’s 16.3, at the cost of lower HPSv3, ImageReward, and GenEval (Table 5 vs. Table 3); the same holds after the full schedule (15.3 vs. 25.2; Table 15). On MJHQ-30K, the direction instead lowers FID from 14.76 to 12.9, since its references are curated high-aesthetic images rather than real photographs. The FID gap therefore reflects where w sets the fidelity–preference trade-off, as for SwD, which is also distilled with guidance and has COCO-30K FIDs of 23.1–23.7, rather than a limit of the adversarial objective. Sec. 5.4 discusses this trade-off.

W2 / Q. Discriminative information versus noise. We ran the analysis the reviewer suggested (Appendix A.7, Table 20). On conditional CIFAR-10, we compute held-out AUC between reference and student samples on the frozen 16×16 and 8×8 readouts at each noise level, with a closed-form linear discriminant and a k-nearest-neighbor discriminator. We use these fresh probes rather than our trained heads, for two reasons. First, the generator is trained to fool the trained heads, so their AUC stays near chance even where information remains. Second, each head sees noise levels in proportion to the sampler, so a low AUC at a rarely sampled level could reflect too little training rather than too little information. A null baseline compares two disjoint reference sets and stays within 0.01 of chance.

  • Separability is concentrated at small noise. For the EDM-trained student after 20M images, the linear AUC is 0.987 at σ = 10⁻³, falls steeply by σ ≈ 0.04, and stays between 0.52 and 0.57 at σ = 0.1 and 0.63; the k-NN AUC reaches the null at σ = 4.
  • Training reduces separability mainly at small noise, by up to 0.16, and by less than 0.04 at large noise.
  • The Beta-trained student is much less separable at small noise (0.776 vs. 0.987), and the 16×16 readout separates more than the 8×8 readout.

The k-NN discriminator gives the same pattern, so the result is not an artifact of linearity. The trend also has a theoretical basis: any discriminator satisfies ∣AUC−12∣≤TV≤W1/(2πσˉt)|AUC − \frac{1}{2}| ≤ TV ≤ W_1/(\sqrt{2π} σ̄_t), so a discrepancy of size W1W_1 is hidden once the noise is much larger than it (Lemma 3, Figure 19). The bound also matches how training changes the AUC: training shrinks W1W_1 and thus the bound, and since the bound is already close to 12\frac{1}{2} at large noise, the AUC can drop mainly at small noise, which is what Table 20 shows (by up to 0.16 at small noise and by less than 0.04 at large noise).

W3. Scalability. MuSD spans 32×32 pixel-space U-Nets to a 2.47B-parameter text-to-image Transformer at 1024×1024. Its trainable additions stay small: the heads are 6.1M parameters for EDM2-S and 35.8M for SD3.5-M, and the text-to-image student trains only LoRA adapters. MuSD needs no teacher-sized auxiliary network, so its memory advantage over score-based methods grows with teacher size (Table 6: 280.2M vs. 6.1M auxiliary parameters). At 1024×1024, both of its components improve fine detail (General Response 1), and the Limitations section notes that teachers beyond 2.47B parameters remain untested. However, as with LADD, which already uses a frozen 8B teacher as its discriminator backbone, we expect MuSD generalizes to larger teachers.

Reviewer_aUp5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MuSD distills diffusion generators using adversarial supervision from lightweight heads attached to multi-resolution features of a frozen diffusion teacher. It favors low supervision noise while retaining support over the noise interval, motivating a Beta sampler through a maximum-entropy argument under a specified corruption-cost constraint. The method avoids an auxiliary fake-score model and an external feature backbone. Experiments span several teacher architectures and image-generation settings, including one-step generation and few-step text-to-image synthesis.

Critical strengths
  1. A simple and practically useful distillation design. Reusing teacher features at multiple resolutions is a coherent way to obtain coarse and fine adversarial feedback without another large feature network or fake-score model. The reported reductions in auxiliary training resources are meaningful for practitioners.

  2. Reasonably broad validation and a concrete noise-sampling hypothesis. Results across U-Net and Transformer teachers and text-to-image generation make the study more informative than a single-backbone demonstration. The feature analysis and noise-sampling ablations give a plausible explanation for why low-noise supervision can work in this design; the maximum-entropy derivation is useful within its stated assumptions.

Critical weaknesses
  1. The reference-data description is inconsistent with the supplied preparation materials. Section 3.3 says that real images are used as references in all experiments, and the related-work discussion uses real-image training to distinguish MuSD from teacher-generated-data approaches. However, the supplementary text-to-image preparation script explicitly identifies the selected Fine-T2I sources as generated images. This does not imply that synthetic data are inappropriate, and “real” may have been intended as the positive class in the GAN objective. Nevertheless, the actual final training mixture and the intended meaning must be stated consistently, because they affect the interpretation of what distribution is distilled and the comparison with prior approaches. The generated-source proportions in the script should not be assumed to describe the entire final 500k set.

  2. The incremental contribution and small numerical advantages need tighter controls. The paper itself identifies LADD as a close predecessor using a frozen diffusion teacher as a discriminator backbone. The existing noise ablations support the proposed setting, but a matched comparison controlling teacher, data, discriminator heads, and training/checkpoint-selection budget is needed to isolate the gain from the noise strategy and multi-resolution readouts. Some leading FID results differ only slightly from competing methods, without enough cross-run evidence to establish a stable advantage. The reported noise interval also differs from the supplied text-to-image script. These are potentially fixable issues, but together they leave the current evidence just below the acceptance threshold.

Questions for authors
  1. Data clarification: What are the exact photographic/other non-generated and synthetic counts in the final text-to-image training set, and which generators produced the synthetic portion? Does “real images” mean non-generated images or simply GAN reference samples? Please reconcile the manuscript and preparation code.
  2. Assessment-changing controlled comparison: With teacher, data, discriminator architecture, and training budget fixed, how much improvement comes from the proposed low-noise sampler relative to the closest high-noise-biased alternative, and how much comes from multi-resolution readouts?
  3. Reproducibility clarification: Which noise intervals and checkpoint-selection procedures produced the reported results, and are the small FID advantages stable across independent runs under the same selection budget?
3 comments
AuthorsOpenReview ↗

Response to Reviewer aUp5

We thank the reviewer for checking our supplementary scripts so carefully; this caught a real inconsistency and improved the reproducibility of the paper.

W1 / Q1. Reference data. The reviewer is right, and our description was wrong. The 500K text-to-image set is 56.4% synthetic: 234,268 images from Z-Image-Turbo and 47,886 from FLUX.2-dev. The rest are 205,831 photographs from Pexels, Unsplash Lite, and Pixabay (all via Fine-T2I) and 12,015 curated high-resolution images from the Aesthetic-4K training split (Table 8). None was generated by SD3.5 or any model in its family, so MuSD does not train on teacher-generated data, which is the distinction from LADD we intended. “Real images” in Sec. 3.3 was imprecise: class-conditional and unconditional experiments use the real datasets only. We corrected Sec. 3.3 and Sec. 2.

This does not change the interpretation of our text-to-image results. Without the teacher CFG direction, the student trained on these references scores below the teacher on PickScore, HPSv3, and ImageReward after the full schedule (Table 15), so the references do not transfer the quality of the external generators.

W2 / Q2. Controlled comparison. As the reviewer requested, Table 5 fixes the teacher, data, discriminator heads, and budget, and changes only the readout and the sampler (configurations in Table 16; General Response 1). Against the LADD-style configuration, MuSD lowers FID on both benchmarks and patch FID at full resolution (15.45 → 13.09). At full resolution, where downsampling no longer hides fine detail, each component contributes on its own: the low-noise sampler lowers patch FID under both readouts (Native 15.45 → 14.03, MRD 14.32 → 13.09), and multi-resolution readouts lower it under both samplers (logit-normal 15.45 → 14.32, Beta 14.03 → 13.09). The sampler’s advantage is largest on CIFAR-10, where LADD’s sampler peaks at FID 11.06 and then deteriorates, versus 1.397 for Beta (Table 4).

Q3. Noise intervals, checkpoint selection, and variance.

  • Noise intervals. All reported text-to-image results use the intervals in Table 9, t ∈ [700, 1000] for the first student step and [1, 1000] for the second, and we will revise the supplementary script to match (General Response 5).
  • Checkpoint selection and variance. As is common in one-step distillation, we report one training run per setting (e.g., Uni-Instruct) and the checkpoint with the lowest FID among those evaluated during training (e.g., SiD). Neither choice drives our results. Following SiD, which re-evaluates its best checkpoint over 10 independent runs, our best conditional CIFAR-10 checkpoint (41.8M images) gives 1.347 ± 0.010, still below SiD²A (1.396) with a much smaller total budget (41.8M images, versus 23M for SiD²A on top of a SiD generator trained on about 720M; Table 1).

Thank you for the thorough response and for correcting the description of the text-to-image training data. The breakdown of the reference data and the clarification of the noise intervals address my questions about the experimental setup. The new controlled comparisons with the LADD-style configurations, including the full-range sampler, also help isolate the contributions of the low-noise sampler and the multi-resolution readout. The repeated evaluation is useful for assessing FID variability at the selected checkpoint, although variability across independent training runs remains open. I will consider the rebuttal alongside the other reviewers’ comments before deciding whether to raise my score.

Thank you for the follow-up. As is common in this literature, we report a single training run per setting because of the training cost, in diffusion training and distillation alike: EDM [1] and DMD2 [2] select checkpoints by FID within one run, SiD [3] re-evaluates its best checkpoint over 10 runs, and Uni-Instruct [4] omits error bars for this reason. As indirect evidence of stability, our runs with κ = 3, 4, and 5 reach FIDs within 0.012 of each other on conditional CIFAR-10 (Figure 4), smaller than our margins over SiD²A (0.049) and Uni-Instruct (0.033).

[1] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022.

[2] T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, and W. T. Freeman. Improved distribution matching distillation for fast image synthesis. In NeurIPS, 2024.

[3] M. Zhou, H. Zheng, Z. Wang, M. Yin, and H. Huang. Score identity Distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In ICML, 2024.

[4] Y. Wang, W. Bai, C. Zhang, D. Zhang, W. Luo, and H. Sun. Uni-Instruct: One-step diffusion model through unified diffusion divergence instruction. In NeurIPS, 2025.

Reviewer_VLU6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MuSD, an adversarial distillation framework for diffusion models. Instead of training an auxiliary fake-score network, MuSD uses a frozen teacher diffusion model as the discriminator backbone and trains lightweight discriminator heads on intermediate teacher features. These heads produce patch-level logits at multiple spatial resolutions and are aggregated by an additional head. The authors analyze how noise affects teacher features across scales and propose a Beta noise sampler derived as the maximum-entropy density under a corruption budget. Experiments cover massive diffusion models, and report few-step distillation results on CIFAR-10, ImageNet, FFHQ/AFHQv2, and text-to-image generation.

Critical strengths

(1) Simple and practically effective design. Replacing an auxiliary score network with lightweight heads on frozen teacher features reduces trainable parameters and memory. In Table 5, for EDM2-S at 512×512, auxiliary parameters drop from 280.2M to 6.1M, and peak memory drops from 52.4GB to 34.4GB. This is practically valuable.

(2) The multi-resolution readout is well motivated. The authors argue that fine-scale structure degrades first as noise increases, while coarse readouts can provide coarse focus at low noise. The ablation in Table 4 (right) shows that using only the 8×8 branch degrades FID substantially, and that 16×16 + 8×8 are complementary. This supports the fine and coarse readout combination.

(3) The paper evaluates MuSD on several different diffusion model families, including pixel-space U-Nets, latent-space U-Nets, and diffusion Transformers. This is important because multi-resolution representations arise naturally in U-Nets but require different design choices in Transformer architectures. The reported results suggest that the framework is not restricted to a particular teacher architecture.

Critical weaknesses

(1) The paper acknowledges that LADD already uses the frozen diffusion teacher itself as the discriminator backbone; Projected GAN and ADD train lightweight heads on frozen pretrained features; multi-scale discriminators are standard in GANs. The actual novelty is therefore MRD readouts and Beta noise sampling. However, there is no direct comparison to a “LADD-style single-scale frozen teacher discriminator with same Beta sampler” baseline, nor is the MRD contribution fully decoupled from Beta on Transformer settings. The current ablations do not convincingly isolate where the gains come from.

(2) The cost function and budget of the maximum-entropy sampler are largely imposed, and the link to distillation loss is weak. Eq. 7 defines C(t)=−log⁡((tmax⁡−t)/(tmax⁡−tmin⁡))C(t) = -\log((t_{\max} - t)/(t_{\max} - t_{\min})), which is the surprisal case of T0≥tT_0 \ge t under a uniform reference. It is monotone in corruption, but it is not derived from feature degradation, discriminator optimality, or FID. The Beta solution is the maximum-entropy density under this specific cost, which does not automatically imply it is the optimal noise distribution for distillation.

Questions for authors

(1) The maximum-entropy derivation seems mathematically valid, but the resulting Beta distribution depends directly on this particular cost function. Can the authors clarify whether the cost is intended to approximate information loss, feature degradation, or another property relevant to adversarial supervision? Additionally, the choice κ\kappa of is empirical in Table 4. A systematic sensitivity analysis across teachers and datasets is missing. The authors should also clarify why κ\kappa can be fixed, and what causal contribution the maximum-entropy principle makes beyond “low noise + full support.”

(2) There are inconsistencies in initialization and budget accounting. The main texts claim “Initialized from a pretrained SiD generator; the reported budget excludes SiD training,” but Appendix A.3.4 says “For EDM-family teachers and DiT-XL/2, the one-step generator is initialized from the teacher weights, without an intermediate distillation stage.” This is a direct contradiction. Which models are initialized from SiD, and which from teacher weights? Do baselines use the same initialization? If MuSD uses teacher initialization while baselines use SiD initialization, the comparison is not fully fair.

(3) Efficiency claims are incomplete: parameters and memory are reported, but actual training cost is not. Table 5 reports auxiliary parameters, optimized parameters, GFLOPs per sample, and peak memory, but not wall-clock training time, GPU hours, or throughput. More importantly, generator updates require gradients to propagate through the frozen teacher backbone, so the backward cost is not eliminated by lightweight heads. For text-to-image with a 2.47B teacher and LoRA, training cost is especially important. The efficiency claim needs more information.

(4) Limitations and failure cases are not discussed. GenEval spatial position is only 0.34 and counting is 0.70, indicating weaknesses in compositional text-to-image generation. There is no limitations section or failure analysis. The paper should discuss high-resolution behavior, complex prompts and adversarial training stability.

(5) This paper ignores some relevant works in efficient diffusion model training, including prior papers [1-2] from a data view and some studies on diffusion training acceleration [3-4].

Reference:

[1] Z. Qin, K. Wang, Z. Zheng, J. Gu, X. Peng, Z. Xu, D. Zhou, L. Shang, B. Sun, X. Xie, and Y. You, “Infobatch: Lossless training speed up by unbiased dynamic data pruning,” In ICLR, 2024.

[2] Y. Li, Y. Zhang, S. Liu, and X. Lin, “Pruning then reweighting: Towards data-efficient training of diffusion models,” In ICASSP, 2025.

[3] Z. Ding, M. Zhang, J. Wu, and Z. Tu, “Patched denoising diffusion models for high-resolution image synthesis,” In ICLR, 2024.

[4] H. Zheng, W. Nie, A. Vahdat, and A. Anandkumar, “Fast training of diffusion models with masked transformers,” In TMLR, 2024.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VLU6

We thank the reviewer for a detailed and constructive review, which led to several new analyses in the revision.

W1. Isolating the gains. The “Native + Beta” row of new Table 5 is exactly the requested baseline: a single-scale frozen-teacher discriminator with the same Beta sampler, on SD3.5-M at 1024×1024. Against it, multi-resolution readouts lower FID on both benchmarks (COCO-30K 16.86 → 16.15, MJHQ-30K 15.81 → 14.76) and patch FID at full resolution (14.03 → 13.09), so the contribution of MRD is decoupled from Beta on a Transformer teacher. Conversely, with the readout fixed, replacing LADD’s logit-normal sampler with Beta lowers patch FID under both readouts (Native 15.45 → 14.03, MRD 14.32 → 13.09), so the contribution of Beta is also decoupled from MRD. The lower GenEval of MuSD at w = 0 in Table 5 does not carry over to the full method: adding the teacher CFG direction (w = 1) at the same budget raises it from 0.631 to 0.719, above the 40-step teacher’s 0.689 (Tables 3 and 17). Its effect is larger on CIFAR-10, where LADD’s sampler peaks at FID 11.06 and then deteriorates while Beta reaches 1.397 (Table 4).

W2 / Q1. The corruption cost and the max-ent principle. We agree that CC is a modeling choice; it is not meant to approximate FID or information loss. Your question led to the analysis in Appendix A.6, and Sec. 4.2 now states what the derivation does and does not claim.

  • Why an increasing cost. The best hinge discriminator is limited by a TV distance that cannot grow with noise (Lemmas 1–2), and the AUC analysis confirms that separability lies at low noise (Appendix A.7).
  • Why not a single low level. It is harmful: a fixed level and truncation at σ = 0.5 give FIDs of 1.762 and 1.827, versus 1.397 for Beta (Table 4).
  • What max-ent adds. The budget encodes what we know, that the signal lies at low noise; the entropy guards against what we do not, which levels the current discrepancy needs, so no level gets zero density for any finite κ [1]. Any increasing cost gives the same shape; ours yields the closed-form Beta family. We do not claim FID-optimality.
  • κ sensitivity. κ = 3–5 give 1.393–1.405 on CIFAR-10, versus 1.683 and 1.514 for κ = 1 and 2 (Figure 4). We use κ ∈ {3, 4} for all teachers without per-dataset tuning (Table 9).

[1] P. D. Grünwald and A. P. Dawid. Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. The Annals of Statistics, 32(4):1367–1433, 2004.

Q2. Initialization. As summarized in General Response 2, every MuSD student, including EDM, EDM2-S, and DiT-XL/2, is initialized from the teacher weights, consistent with Appendix A.3.5, and the † marks the SiD²A baseline. SiD, SiDA, and D2O also initialize from the teacher, so the comparison is conservative for MuSD.

Q3. Training cost. Thank you for raising this; we now report actual training cost. We measured EDM2-S (ImageNet 512×512) for MuSD and SiDA on the same GPU: one RTX A6000, global batch 64 as two microbatches of 32, 20 warm-up iterations, then three windows of 30 iterations (Table 6, Appendix A.4.5).

SiDAMuSD
Time per iteration (s)3.324 ± 0.0021.942 ± 0.006
Throughput (samples/s)19.2532.95
Peak allocated memory (GiB)33.021.7
GFLOPs per sample2,3801,404
Training images to reported FID118M95M
GPU-hours to reported FID (estimated)≈1,700≈800
FID1.7561.654

For text-to-image distillation with the 2.47B teacher, MuSD costs about 1.8× fewer FLOPs per iteration than SwD’s public configuration, which updates a fake-score network five times per generator step (Table 19). The full schedule of 500K images takes 55 B200 GPU-hours and about 1.4× SwD’s total compute, but quality saturates early: after 162K images (about 18 GPU-hours), MuSD already exceeds SwD on PickScore, ImageReward, GenEval, and MJHQ-30K FID (Table 3) with about half its total compute.

Q4. Limitations and failure cases. We added a Limitations section (Sec. 6). First, unlike data-free methods such as SiD, MuSD needs reference samples. Second, the weak GenEval categories are inherited from the teacher: the 40-step teacher scores lower on position and counting (0.253 and 0.609) than our two-step student (0.34 and 0.70), and after only 162K images the student already exceeds it overall (0.719 vs. 0.689; Tables 14 and 17). Finally, although both MuSD components improve fine detail at 1024×1024 (General Response 1), we have not tested teachers beyond 2.47B parameters or 3D and video models.

Q5. Related work. Thank you for the pointers. We added the suggested works to Sec. 2. Data pruning and reweighting, patch-wise denoising, and masked Transformers reduce the cost of training a diffusion model, whereas our efficiency gains concern distilling a trained model, so the two lines are complementary.

The rebuttal improves the paper by adding the requested ablations, efficiency measurements, initialization clarification, limitations, and related work.

However, for W1, the isolation is still limited to a single Transformer teacher and a single 1024×1024 setting. The U-Net/EDM and DiT-XL/2 settings, where MuSD is also claimed to apply, are not ablated. Moreover, the reported FID gains (COCO 16.86→16.15, MJHQ 15.81→14.76) are minor.

Regrading W2/Q1, the authors now state that CC is a modeling choice and do not claim FID-optimality. This is honest but weakens the theoretical contribution. If any increasing cost yields the same Beta shape, then the maximum-entropy principle does not constitute a principled derivation of the optimal noise distribution; it mainly provides a convenient closed-form family under a specific cost.

Overall, I prefer to maintain my current score.

Thank you for the follow-up. We would like to clarify two points, since both are already addressed in the paper.

W1. The two components are also crossed on a U-Net teacher, in Table 4 (right; EDM, conditional CIFAR-10). Beta improves every readout configuration (e.g., all branches 1.645 → 1.397, 16×16 only 1.706 → 1.456), and the multi-resolution readout improves on each single branch under both samplers (e.g., with Beta, 16×16 only 1.456 → 1.397 and 8×8 only 1.904 → 1.397). On the same teacher, LADD's sampler, mapped to it by matching the signal-to-noise ratio, peaks at FID 11.06 and then deteriorates, versus 1.397 for ours (Table 4, left; Appendix A.8). DiT-XL/2 is indeed not ablated. On SD3.5-M, the gains are largest where fine detail is measured (patch FID 14.03 → 13.09 and 15.45 → 14.03) and against LADD's sampler as LADD uses it (MJHQ-30K FID 18.43 → 14.76, patch FID 18.59 → 13.09).

W2/Q1. As stated in Appendix A.6, different increasing costs give different densities, not the same Beta shape. What they share is a density that decreases with noise and keeps full support. We apologize that our response compressed this into "the same shape." Within that structure, max-ent is what makes the choice principled: given only the budget, it minimizes the worst-case log loss [1], so it adds no assumption beyond the budget. Our measurements support this preference. Appendix A.7 shows that the remaining difference between reference and student samples appears mainly at low noise. Among samplers that cover all noise levels, those that put more weight on low noise reach lower FID: ours puts the most, and LADD's almost none (Table 4).

[1] P. D. Grünwald and A. P. Dawid. Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. The Annals of Statistics, 32(4):1367–1433, 2004.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading. Reviewers found MuSD simple and practically useful (VLU6, aUp5), its efficiency gains significant (VLU6, 2C7r), and the low-noise finding interesting beyond our method, with an elegant derivation (2C7r). All three rated soundness as “minor or fixable.” And thank you all for high ratings in presentation. We address the shared concerns below. The revised PDF marks all changes in blue.

1. Controlled comparisons with LADD’s design (VLU6 W1, aUp5 Q2).

On conditional CIFAR-10, LADD’s high-noise sampler, mapped to our teacher by matching the signal-to-noise ratio (ln σ ∼ N(0.31, 1)), peaks at FID 11.06 after about 1M images and then deteriorates, whereas our Beta sampler reaches 1.397 with everything else fixed (Table 4, Table 21). On SD3.5-M at 1024×1024, we crossed the readout and the sampler with everything else fixed and w = 0 (Table 5; 162K images each):

ReadoutSamplerMJHQ pFID↓COCO FID↓MJHQ FID↓COCO HPSv3↑GenEval↑
NativeLogit-normal (LADD-style)15.4517.9815.139.240.648
MRDBeta (MuSD)13.0916.1514.769.680.631
NativeBeta14.0316.8615.819.640.645
MRDLogit-normal14.3215.4414.709.200.645

MuSD lowers FID on both benchmarks relative to the LADD-style configuration and raises PickScore and HPSv3, with ImageReward within 0.01 (full metrics in Table 5). At 256×256, the sampler’s effect on FID is mixed because the images are downsampled from 1024×1024, but patch FID (pFID), computed on 256×256 crops taken at full resolution, shows that both components (MRD and Beta) improve distribution matching. Downsampling hides the fine detail that low-noise supervision targets. The LADD-style configuration isolates LADD’s readout and sampler rather than reproducing its full system (Table 16).

2. Initialization (VLU6 Q2). We apologize for the ambiguity. All MuSD students start from the teacher. The † in Tables 1, 2, and 12 marks the SiD$^2$A baseline, which starts from a pretrained SiD generator. We rewrote these ambiguous captions.

3. Training cost (VLU6 Q3). On the same GPU, MuSD is 1.71× faster per iteration than SiDA with EDM2-S, uses 1.52× less memory, and needs about 2.1× fewer GPU-hours to reach a better FID (Table 6). For text-to-image, after 162K images it already exceeds SwD on PickScore, ImageReward, GenEval, and MJHQ-30K FID with about half SwD’s total compute (Table 19). Details are in our response to VLU6 (Q3).

4. Text-to-image FID (2C7r W1). At the same budget, removing the teacher CFG direction (w=1→0w=1 \to 0) lowers COCO-30K FID from 24.4 to 16.15, below the teacher’s 16.3, at the cost of preference scores and GenEval; on MJHQ-30K, the direction instead lowers FID. The COCO-30K gap reflects this fidelity–preference trade-off, now discussed in Sec. 5.4.

5. Reference data and noise intervals (aUp5 W1, Q3). Thank you very much for your efforts in ensuring the reproducibility of our paper. The 500K text-to-image set is 56.4% images generated by Z-Image-Turbo and FLUX.2-dev and 43.6% photographs and curated images, none from SD3.5 or its family (Table 8). We corrected Sec. 3.3 and Sec. 2; class-conditional and unconditional results use real data only. All reported text-to-image results use the intervals in Table 9, t ∈ [700, 1000] for the first student step and [1, 1000] for the second. We will revise the supplementary material.

6. Why low noise, prompted by the reviewers (VLU6 W2, 2C7r W2). Two reviewer suggestions shaped this analysis, and we are grateful for both. Following VLU6’s request to tie the corruption cost to discriminator optimality, Appendix A.6 shows that the best hinge discriminator is limited by the total variation between the corrupted distributions, which cannot grow with noise and becomes negligible once the noise greatly exceeds the remaining discrepancy (Lemmas 1–3). Following 2C7r’s suggestion to measure AUC across noise levels, Appendix A.7 confirms this directly: reference–student separability is concentrated at small noise, and training reduces it mainly there (Table 20). The theory and the measurement agree, and together they explain why MuSD favors low noise more directly than our original feature analysis did.

Summary of revisions

  • Sec. 2, Sec. 3.3, and A.3.1: reference data; related work on efficient diffusion training.
  • Sec. 4.2 and A.6: link between the corruption cost and the discriminator.
  • Sec. 5.3–5.5: MuSD after 162K images (Table 3); LADD comparisons on CIFAR-10 and SD3.5-M, with patch FID at full resolution (Tables 4–5, A.4.4, A.8); fidelity–preference trade-off; wall-clock cost (Table 6, A.4.5).
  • Sec. 6: Limitations. A.7: separability across noise levels.
  • Captions of Tables 1, 2, and 12.
AuthorsOpenReview ↗

General Response 2

LADD's sampler over the full range (VLU6 W1, aUp5 Q2). In Table 5, the LADD-style configuration rescales LADD's logit-normal(1, 1) to each student step's interval, as we do for Beta. LADD itself uses the full range at every step, so we also trained this full-range variant, which differs from the LADD-style configuration only in the first step's critic interval ([1, 1000] instead of [700, 1000]). MuSD improves on it on every metric:

MJHQ pFID↓COCO FID↓MJHQ FID↓COCO HPSv3↑COCO IR↑GenEval↑
LADD's sampler, full range18.5917.1418.439.210.8090.599
MuSD13.0916.1514.769.680.8600.631

The full-range variant is also worse than the LADD-style configuration in Table 5 on every metric except COCO-30K FID, so rescaling made that baseline stronger. In the updated PDF, we added this run to Table 5 and to Table 17 (CLIP-H and per-category GenEval), described it in Table 16 and Appendix A.4.4, and mention it in Sec. 5.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.