- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MuSD, which uses a frozen diffusion teacher for adversarial distillation. The authors further use a Beta sampler derived as the maximum-entropy density under an expected "corruption cost" budget, which favors low noise. For text-to-image, a frozen-teacher CFG direction is added to the adversarial gradient. The method is tested on CiFAR-10 and other image sets.
Critical strengths
- I like the finding about the low-noise preference, which not only inspired the noise sampler used in MuSD, but can also be interesting for other reasons.
- The derivation of Beta noise sampler is elegant.
- Efficiency gains are significant.
Critical weaknesses
- The Text-to-image results are the weak spot. Specifically, FID results are poor. MuSD's two-step COCO-30K FID (25.2) is the worst among distilled models (Table 3); There is a tradeoff between FID and preference. The paper should discuss this trade-off and the large FID gap.
- The mechanism evidence is weak. Probing via linear reconstruction of structure (Appendix A.1) is a reasonable proxy, but it’d be better to show a direct measure of discriminative information content.
- It’s not clear how the method can be sclaed up.
Questions for authors
Can the authors show the discriminator's classification AUC (real vs generated) as a function of noise level? This would be a good test of whether low-noise features carry more information.
Response to Reviewer 2C7r
We thank the reviewer for the positive assessment and especially for suggesting the AUC analysis, which became one of the clearest pieces of evidence in the revision.
W1. Text-to-image FID. At the same budget of 162K images, removing the teacher CFG direction (w = 0) lowers COCO-30K FID from 24.4 to 16.15, below the 40-step teacher’s 16.3, at the cost of lower HPSv3, ImageReward, and GenEval (Table 5 vs. Table 3); the same holds after the full schedule (15.3 vs. 25.2; Table 15). On MJHQ-30K, the direction instead lowers FID from 14.76 to 12.9, since its references are curated high-aesthetic images rather than real photographs. The FID gap therefore reflects where w sets the fidelity–preference trade-off, as for SwD, which is also distilled with guidance and has COCO-30K FIDs of 23.1–23.7, rather than a limit of the adversarial objective. Sec. 5.4 discusses this trade-off.
W2 / Q. Discriminative information versus noise. We ran the analysis the reviewer suggested (Appendix A.7, Table 20). On conditional CIFAR-10, we compute held-out AUC between reference and student samples on the frozen 16×16 and 8×8 readouts at each noise level, with a closed-form linear discriminant and a k-nearest-neighbor discriminator. We use these fresh probes rather than our trained heads, for two reasons. First, the generator is trained to fool the trained heads, so their AUC stays near chance even where information remains. Second, each head sees noise levels in proportion to the sampler, so a low AUC at a rarely sampled level could reflect too little training rather than too little information. A null baseline compares two disjoint reference sets and stays within 0.01 of chance.
The k-NN discriminator gives the same pattern, so the result is not an artifact of linearity. The trend also has a theoretical basis: any discriminator satisfies ∣AUC−21∣≤TV≤W1/(2πσˉt), so a discrepancy of size W1 is hidden once the noise is much larger than it (Lemma 3, Figure 19). The bound also matches how training changes the AUC: training shrinks W1 and thus the bound, and since the bound is already close to 21 at large noise, the AUC can drop mainly at small noise, which is what Table 20 shows (by up to 0.16 at small noise and by less than 0.04 at large noise).
W3. Scalability. MuSD spans 32×32 pixel-space U-Nets to a 2.47B-parameter text-to-image Transformer at 1024×1024. Its trainable additions stay small: the heads are 6.1M parameters for EDM2-S and 35.8M for SD3.5-M, and the text-to-image student trains only LoRA adapters. MuSD needs no teacher-sized auxiliary network, so its memory advantage over score-based methods grows with teacher size (Table 6: 280.2M vs. 6.1M auxiliary parameters). At 1024×1024, both of its components improve fine detail (General Response 1), and the Limitations section notes that teachers beyond 2.47B parameters remain untested. However, as with LADD, which already uses a frozen 8B teacher as its discriminator backbone, we expect MuSD generalizes to larger teachers.