- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies few-step distillation of pixel-space text-to-image diffusion models within the DMD framework. It makes three modifications. (i) DM-Band restricts the DMD matching timestep to a fixed high-noise range. The motivation is a C2ST diagnostic suggesting that near-clean real/generated differences in RGB are dominated by local high-frequency texture. (ii) DINO-Adv replaces DMD2’s feature GAN head on the fake-score network with a discriminator built on a frozen DINOv2 ViT-B/14 with lightweight multi-layer patch heads. (iii) AF-Loss is an auxiliary loss that moves the generated image’s DINOv2 CLS feature along an attract–repel kernel field computed from rolling FIFO banks of real and generated features.
Critical strengths
- The design is simple and practical. The DINOv2 encoder is frozen and shared by both auxiliary losses, only about 1.58M trainable parameters are added, and removing the adversarial path through the fake-score network reduces training time per step by about 2.5 times.
- All baselines are re-implemented on the same teacher, data, step budget and evaluation protocol, which makes the main comparison more controlled than copying numbers from papers.
- The proposed method achieves the best reported results among the evaluated
Critical weaknesses
- In 156-158, the authors state the synthetic image as “produces a clean RGB image” and “This native RGB output”. The terms "clean" and "native" are typically used for real/input images. In line 196, the analysis refers to the same quantity as generated images and contrasts it with real ones. The terminology is inconsistent and hard to distinguish the synthetic sample from the input or ground-truth image.
- In equation (1), some definitions are missing, such as the subscript r and f.
- AF-Loss is presented as a new objective, but as described in Eq. 3–4 and Algorithm 1, it uses the same kernel, temperatures, doubly normalized affinity, self-masking, per-temperature normalization and stop-gradient loss as Drifting (Reference: Generative Modeling via Drifting. Deng et al., 2026).
- DINO-Adv is also close to existing designs. A frozen DINOv2 feature network with trainable lightweight heads on several intermediate layers is the discriminator used in “Adversarial Diffusion Distillation” (Sauer et al., ECCV 2024; the method behind SDXL-Turbo), itself following Projected GAN (Sauer et al., in reference). The only differences seem to be skipping a VAE decoder (which follows directly from using a pixel-space teacher) and adding noise to the discriminator inputs.
- The C2ST diagnostic does not measure what DMD transfers. C2ST separates real images from student samples. The DMD gradient is the difference between the teacher score and the fake score.
- The trends in Figure 3 can largely be explained by signal-to-noise ratio. A linear classifier on heavily noised raw pixels will approach chance at high noise no matter which cue separates the clean images, so falling separability with noise is expected and is not specific to RGB.
- The effect sizes are small and no variance is reported. DM-Band changes DPG by about 0.4 and GenEval by about 0.004–0.005 (Table 2, Table 5). The paper says baselines use three seeds but gives no standard deviations.
- COCO-30K results report IS, recall and NIQE but omit FID, the standard fidelity metric.
Questions for authors
- Could you repeat the C2ST diagnostic with teacher samples instead of real images, so that it reflects what the DMD gradient actually matches?
- Is the fake-score critic trained over the full noise range or only within the band?
5 comments
Follow-Up on Rebuttal Discussion
Dear Reviewer uHZb,
We deeply appreciate your valuable feedback during the first round of review and the thoughtful discussion that has significantly helped us refine our work. Since the discussion phase ends on Nov 18, we would like to know whether we have addressed all the issues, and we would greatly welcome any additional feedback or suggestions you may have.
Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.
Best regards,
Paper 7251 Authors
Kindly Reminder for Potential Discussion
Dear Reviewer uHZb,
We sincerely thank you for your valuable feedback during the first round of review and for the thoughtful discussions that have greatly contributed to improving our work. Your insights and suggestions have been instrumental in refining our submission, and we are deeply grateful for your time and effort.
We kindly wish to confirm whether we have satisfactorily addressed all your concerns. Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.
Best regards,
Authors
Response to Reviewer uHZb (Part 1/3)
W1 Terminology for Generated RGB Images
Here, “clean” denotes the student’s predicted image before diffusion noise is added, in contrast to its noised state; it does not denote a real or ground-truth image. “Native RGB” means that the model outputs RGB directly in pixel space without VAE decoding, not that the image comes from real data. All these passages refer to the same generated sample. In the revised version, we have standardized the terminology to “student-generated RGB image” or “generated image,” denoted real images by x and generated images by x̂, and clarified that “native” refers to the output space.
W2 Definitions in Equation 1
In Eq. (1), r denotes the real-score branch, implemented by the frozen teacher, and f denotes the fake-score branch, a trainable critic that models the student-generated distribution. The quantities x0,r and x0,f are the respective denoised image estimates produced by these networks from the same noised student sample, at the same matching time and under the same text condition. The subscript r identifies the teacher branch; it does not indicate a real image or ground-truth target. In the revised version, we have added these definitions next to Eq. (1) and explicitly defined x̂ as the student-generated image, sg as the stop-gradient operator, and ε as the numerical stabilizer in the denominator.
W3 Differences Between AF-Loss and Drifting Models
In the revised version, we have added a discussion to Deng et al. (2026) and clarified the connection between the AF field and Drifting Models, including the design underlying Algorithm 1. The discussion distinguishes our adaptations and their empirical benefits, as detailed below.
AF-Loss adapts the drifting-field formulation of Deng et al. from class-to-image training from scratch to distillation-based post-training of pretrained text-to-image models. Three design choices make this adaptation effective.
Supervision and training role. In the ImageNet setting of Drifting Models, the drifting loss is the core objective for training a class-conditional generator from scratch, with support samples grouped by class. Open-domain text-to-image generation typically lacks both a fixed class partition and multiple real images for each complex prompt. We retain text-conditioned DMD for teacher and prompt supervision and introduce AF-Loss as an additional real-data objective that constrains the feature distribution across prompts. This division of responsibilities allows AF-Loss to complement distillation and adversarial supervision. Adding AF-Loss to the DINO-Adv configuration improves DPG from 82.550 to 83.427.
Support-set construction. We maintain continuously updated real and generated feature banks across batches without grouping samples by class. Current and recent generated features jointly provide the repulsive support. The updated bank-size ablation supports retaining historical features: increasing the bank size from 0 to 256 improves GenEval from 0.8132 to 0.8167 and IS from 40.13 to 40.46. These results support cross-batch support sets for improving image quality while preserving prompt-following performance.
Representation and division of responsibilities. Drifting Models apply their objective to encoder features at multiple scales and spatial locations. We use the frozen DINOv2 encoder’s final CLS representation to provide an image-level constraint. Holding the field formulation and loss weight fixed, our appendix ablation compares this choice with multi-layer patch-feature aggregation and local feature statistics. These alternatives achieve DPG scores of 81.430–81.970, whereas the final CLS representation reaches 83.194. This controlled comparison supports an image-level field constraint within our distillation framework.
W4 Contributions of DINO-Adv
ADD was an early distillation method to use a discriminator operating directly on RGB images. The subsequent LADD and DMD2 adopted a discriminator based on fake-score network features for latent text-to-image generation and demonstrated that this design outperforms direct RGB-space discrimination. Our central finding is that pixel diffusion calls for a different choice from latent diffusion: this discriminator design does not transfer unchanged to native pixel-space DMD. In a controlled replacement experiment, switching from fake-score feature discrimination to a frozen DINOv2 discriminator operating on RGB images raises four-step GenEval from 0.7709 to 0.8155 and reduces training time per step from 1.82 s to 0.72 s, a speedup of approximately 2.5×. These results demonstrate the quality and efficiency gains from reconsidering the discriminator when moving DMD from latent to pixel space. Our contribution is this experimentally validated design choice for pixel-space distillation. The frozen-feature architecture follows ADD, and we have expanded the related-work discussion about ADD.