acceptodds
Under review as a conference paper at ICLR 2027

DMA: Pixel-space Distribution Matching with Adversarial and Anchor Losses

Abstract

Few-step distribution matching distillation (DMD) accelerates latent diffusion, but its latent-tuned recipe transfers imperfectly to pixel teachers. We analyze both sides of DMD. On the teacher-matching side, our diagnostics provide evidence that low-noise RGB matching is dominated by a local-texture cue; this motivates a fixed high-noise matching band. On the real-data side, native clean-RGB outputs allow guidance from an external visual representation without traversing a decoder or sharing the heavy fake-score critic. DINO-Adv removes this critic from the adversarial gradient path and supplies local parametric patch guidance. For distribution-level guidance, we introduce AF-Loss, a parameter-free semantic distribution-field objective for DMD. It turns detached empirical real and generated feature supports into a multi-scale field and injects its stopped displacement as an auxiliary generator target. AF-Loss requires no learned field estimator, adds no learnable parameters, and introduces no inference-time computation. Together these designs form DMA. Across DPG-Bench, GenEval, VQAScore, and COCO30K, the four-step DMA student performs better than the 25-step teacher on all seven reported metrics and ranks first among the evaluated few-step distillers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Pixel Diffusion, Few-step Generation, Distribution Matching Distillation, Text-to-image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uHZbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies few-step distillation of pixel-space text-to-image diffusion models within the DMD framework. It makes three modifications. (i) DM-Band restricts the DMD matching timestep to a fixed high-noise range. The motivation is a C2ST diagnostic suggesting that near-clean real/generated differences in RGB are dominated by local high-frequency texture. (ii) DINO-Adv replaces DMD2’s feature GAN head on the fake-score network with a discriminator built on a frozen DINOv2 ViT-B/14 with lightweight multi-layer patch heads. (iii) AF-Loss is an auxiliary loss that moves the generated image’s DINOv2 CLS feature along an attract–repel kernel field computed from rolling FIFO banks of real and generated features.

Critical strengths
  1. The design is simple and practical. The DINOv2 encoder is frozen and shared by both auxiliary losses, only about 1.58M trainable parameters are added, and removing the adversarial path through the fake-score network reduces training time per step by about 2.5 times.
  2. All baselines are re-implemented on the same teacher, data, step budget and evaluation protocol, which makes the main comparison more controlled than copying numbers from papers.
  3. The proposed method achieves the best reported results among the evaluated
Critical weaknesses
  1. In 156-158, the authors state the synthetic image as “produces a clean RGB image” and “This native RGB output”. The terms "clean" and "native" are typically used for real/input images. In line 196, the analysis refers to the same quantity as generated images and contrasts it with real ones. The terminology is inconsistent and hard to distinguish the synthetic sample from the input or ground-truth image.
  2. In equation (1), some definitions are missing, such as the subscript r and f.
  3. AF-Loss is presented as a new objective, but as described in Eq. 3–4 and Algorithm 1, it uses the same kernel, temperatures, doubly normalized affinity, self-masking, per-temperature normalization and stop-gradient loss as Drifting (Reference: Generative Modeling via Drifting. Deng et al., 2026).
  4. DINO-Adv is also close to existing designs. A frozen DINOv2 feature network with trainable lightweight heads on several intermediate layers is the discriminator used in “Adversarial Diffusion Distillation” (Sauer et al., ECCV 2024; the method behind SDXL-Turbo), itself following Projected GAN (Sauer et al., in reference). The only differences seem to be skipping a VAE decoder (which follows directly from using a pixel-space teacher) and adding noise to the discriminator inputs.
  5. The C2ST diagnostic does not measure what DMD transfers. C2ST separates real images from student samples. The DMD gradient is the difference between the teacher score and the fake score.
  6. The trends in Figure 3 can largely be explained by signal-to-noise ratio. A linear classifier on heavily noised raw pixels will approach chance at high noise no matter which cue separates the clean images, so falling separability with noise is expected and is not specific to RGB.
  7. The effect sizes are small and no variance is reported. DM-Band changes DPG by about 0.4 and GenEval by about 0.004–0.005 (Table 2, Table 5). The paper says baselines use three seeds but gives no standard deviations.
  8. COCO-30K results report IS, recall and NIQE but omit FID, the standard fidelity metric.
Questions for authors
  1. Could you repeat the C2ST diagnostic with teacher samples instead of real images, so that it reflects what the DMD gradient actually matches?
  2. Is the fake-score critic trained over the full noise range or only within the band?
5 comments
AuthorsOpenReview ↗

Response to Reviewer uHZb (Part 1/3)

W1 Terminology for Generated RGB Images

Here, “clean” denotes the student’s predicted image before diffusion noise is added, in contrast to its noised state; it does not denote a real or ground-truth image. “Native RGB” means that the model outputs RGB directly in pixel space without VAE decoding, not that the image comes from real data. All these passages refer to the same generated sample. In the revised version, we have standardized the terminology to “student-generated RGB image” or “generated image,” denoted real images by x and generated images by x̂, and clarified that “native” refers to the output space.

W2 Definitions in Equation 1

In Eq. (1), r denotes the real-score branch, implemented by the frozen teacher, and f denotes the fake-score branch, a trainable critic that models the student-generated distribution. The quantities x0,r and x0,f are the respective denoised image estimates produced by these networks from the same noised student sample, at the same matching time and under the same text condition. The subscript r identifies the teacher branch; it does not indicate a real image or ground-truth target. In the revised version, we have added these definitions next to Eq. (1) and explicitly defined x̂ as the student-generated image, sg as the stop-gradient operator, and ε as the numerical stabilizer in the denominator.

W3 Differences Between AF-Loss and Drifting Models

In the revised version, we have added a discussion to Deng et al. (2026) and clarified the connection between the AF field and Drifting Models, including the design underlying Algorithm 1. The discussion distinguishes our adaptations and their empirical benefits, as detailed below.

AF-Loss adapts the drifting-field formulation of Deng et al. from class-to-image training from scratch to distillation-based post-training of pretrained text-to-image models. Three design choices make this adaptation effective.

  1. Supervision and training role. In the ImageNet setting of Drifting Models, the drifting loss is the core objective for training a class-conditional generator from scratch, with support samples grouped by class. Open-domain text-to-image generation typically lacks both a fixed class partition and multiple real images for each complex prompt. We retain text-conditioned DMD for teacher and prompt supervision and introduce AF-Loss as an additional real-data objective that constrains the feature distribution across prompts. This division of responsibilities allows AF-Loss to complement distillation and adversarial supervision. Adding AF-Loss to the DINO-Adv configuration improves DPG from 82.550 to 83.427.

  2. Support-set construction. We maintain continuously updated real and generated feature banks across batches without grouping samples by class. Current and recent generated features jointly provide the repulsive support. The updated bank-size ablation supports retaining historical features: increasing the bank size from 0 to 256 improves GenEval from 0.8132 to 0.8167 and IS from 40.13 to 40.46. These results support cross-batch support sets for improving image quality while preserving prompt-following performance.

  3. Representation and division of responsibilities. Drifting Models apply their objective to encoder features at multiple scales and spatial locations. We use the frozen DINOv2 encoder’s final CLS representation to provide an image-level constraint. Holding the field formulation and loss weight fixed, our appendix ablation compares this choice with multi-layer patch-feature aggregation and local feature statistics. These alternatives achieve DPG scores of 81.430–81.970, whereas the final CLS representation reaches 83.194. This controlled comparison supports an image-level field constraint within our distillation framework.

W4 Contributions of DINO-Adv

ADD was an early distillation method to use a discriminator operating directly on RGB images. The subsequent LADD and DMD2 adopted a discriminator based on fake-score network features for latent text-to-image generation and demonstrated that this design outperforms direct RGB-space discrimination. Our central finding is that pixel diffusion calls for a different choice from latent diffusion: this discriminator design does not transfer unchanged to native pixel-space DMD. In a controlled replacement experiment, switching from fake-score feature discrimination to a frozen DINOv2 discriminator operating on RGB images raises four-step GenEval from 0.7709 to 0.8155 and reduces training time per step from 1.82 s to 0.72 s, a speedup of approximately 2.5×. These results demonstrate the quality and efficiency gains from reconsidering the discriminator when moving DMD from latent to pixel space. Our contribution is this experimentally validated design choice for pixel-space distillation. The frozen-feature architecture follows ADD, and we have expanded the related-work discussion about ADD.

AuthorsOpenReview ↗

Follow-Up on Rebuttal Discussion

Dear Reviewer uHZb,

We deeply appreciate your valuable feedback during the first round of review and the thoughtful discussion that has significantly helped us refine our work. Since the discussion phase ends on Nov 18, we would like to know whether we have addressed all the issues, and we would greatly welcome any additional feedback or suggestions you may have.

Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.

Best regards,

Paper 7251 Authors

AuthorsOpenReview ↗

Kindly Reminder for Potential Discussion

Dear Reviewer uHZb,

We sincerely thank you for your valuable feedback during the first round of review and for the thoughtful discussions that have greatly contributed to improving our work. Your insights and suggestions have been instrumental in refining our submission, and we are deeply grateful for your time and effort.

We kindly wish to confirm whether we have satisfactorily addressed all your concerns. Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.

Best regards,

Authors

Reviewer_KTzvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies distribution matching distillation (DMD) for a pixel-space text-to-image teacher (DeCo, 1.1B, 512×512). It proposes three changes:

  1. DM-Band samples the DMD matching noise level only from a high-noise range.
  2. DINO-Adv replaces DMD2's GAN head on the fake-score network with lightweight heads on a frozen DINOv2. This reduces the training time from 1.82 to 0.72 s/step.
  3. AF-Loss is a parameter-free attract–repel loss in DINOv2 CLS space, based on rolling banks of real and generated features.

The paper compares against five distillation methods re-implemented on the same teacher. Both the 4-step and the 1-step students achieve the best results on all seven reported metrics, and the 4-step student is reported to outperform the 25-step teacher.

Critical strengths
  1. A controlled comparison. Five distillation methods are re-implemented on the same pixel-space teacher, with the same data, training budget and evaluation, using three seeds, at both 1 and 4 steps. This is much more informative than comparing numbers across different teachers, and the 1-step student compares well with the 25-step teacher (82.96 vs. 81.58 DPG).

  2. A practical efficiency gain. Moving the adversarial loss from the fake-score network to a frozen DINOv2 makes each training step 2.5× faster (0.72 vs. 1.82 s). It also gives better scores than the DMD2 GAN (e.g., 4-step GenEval 0.8155 vs. 0.7709). This is useful for distilling large pixel-space models.

Critical weaknesses
  1. Limited novelty and insufficient attribution.

    AF-Loss closely follows the drifting field of Deng et al. (2026). Algorithm 1 uses the same kernel and temperatures {0.02, 0.05, 0.2}, the same distance and per-scale normalization, self-masking, a FIFO sample queue and the same stop-gradient loss. The paper mentions Deng et al. only as related work that uses a field as a standalone objective, and never says that the field in Algorithm 1 comes from it. Since it is concurrent work, I only expect proper attribution, not an experimental comparison.

    DINO-Adv is very similar to the ADD discriminator (Sauer et al., ECCV 2024: a frozen DINOv2 with lightweight heads on several layers). For DINOv2-based discriminators, the related work only cites Projected GANs and DINOv2, and ADD is not cited.

    DM-Band is essentially a choice of timestep range, and the evidence for it is weak. Table 2 has no band-only row, and the band sweep in Table 5 keeps DINO-Adv on. The band-only point in Fig. 7 (weight 0) has a much lower IS (28.3) than DMD without the band in Table 2 (34.24). These two settings differ only in the band (one is a single run, the other a three-seed mean). Seed noise is unlikely to explain a 6-point IS gap: for band + DINO-Adv, the single run in Fig. 7 gives IS ≈ 40.0 vs. 40.21 in Table 2. This suggests that the band only helps together with the real-data losses.

    This matters because all three components are presented as contributions.

  2. The main empirical claims are not well supported.

    • No standard deviations are reported, although three seeds were used, and several margins are small (e.g., GenEval 0.8232 vs. 0.8100; CLIP 0.3203 vs. 0.3198). A single run of band + DINO-Adv reaches GenEval 0.833 (Fig. 7), while the three-seed mean is 0.8193 (Table 2), so seed variation seems comparable to these margins.
    • DMA² was tuned with sweeps on the evaluation benchmarks, while the baselines use the settings from their original papers.
    • DeCo reports GenEval 0.86 and DPG 81.4 for the same teacher, but Table 1 reports 0.8216. With 0.86, the 4-step student (0.8232) would not outperform the teacher on GenEval.
    • COCO30K FID, precision and a per-prompt diversity measure are also missing; recall only measures dataset-level coverage.

    Since the paper's case rests on these results, these issues, together with the novelty concern above, are the main reasons for my negative rating.

Questions for authors

Questions that could change my assessment

  1. Could you add band-only and band + AF-Loss rows to Table 2, and report the mean ± std over the three seeds?
  2. Why is the teacher's GenEval (0.8216) lower than the 0.86 reported by DeCo? Which sampler and CFG are used for the teacher in Table 1?
  3. The baselines keep the hyperparameters from their original papers. Did you try tuning them for this teacher? Would Decoupled DMD or DMDR, the strongest baselines, also benefit from DM-Band and DINO-Adv?
  4. Could you report COCO30K FID and precision, and a per-prompt diversity metric?
  5. What exactly is new in AF-Loss compared with the drifting field of Deng et al., and in DINO-Adv compared with ADD?

Minor clarifications and suggestions

  • The C2ST in Fig. 3 compares real images with student samples, while DMD matches the student to the teacher, so a teacher-vs-student test would be more relevant. Also, in Fig. 3(c), DINOv2 features are more separable than RGB at every noise level. This suggests that differences beyond local texture may remain even near clean images, which weakens the texture-dominance argument.
  • Table 1 and Table 4 report the same NIQE (4.064) for two different teachers; please double-check.
  • The main text only gives a simplified version of the AF field. It would help to state the actual field there.
5 comments
AuthorsOpenReview ↗

Response to Reviewer KTzv (Part 1/4)

W1.1&Q5 Differences Between AF-Loss and Drifting Models

In the revised version, we have added a discussion to Drifting Models. We also distinguish this underlying formulation from our adaptations for pixel-space text-to-image distillation, as detailed below.

AF-Loss adapts the drifting-field formulation of Drifting Models. from class-to-image training from scratch to distillation-based post-training of pretrained text-to-image models. Three design choices make this adaptation effective.

  1. Supervision and training role. In the ImageNet setting of Drifting Models, the drifting loss is the core objective for training a class-conditional generator from scratch, with support samples grouped by class. Open-domain text-to-image generation typically lacks both a fixed class partition and multiple real images for each complex prompt. We retain text-conditioned DMD for teacher and prompt supervision and introduce AF-Loss as an additional real-data objective that constrains the feature distribution across prompts. This division of responsibilities allows AF-Loss to complement distillation and adversarial supervision. Adding AF-Loss to the DINO-Adv configuration improves DPG from 82.550 to 83.427.

  2. Support-set construction. We maintain continuously updated real and generated feature banks across batches without grouping samples by class. Current and recent generated features jointly provide the repulsive support. The updated bank-size ablation supports retaining historical features: increasing the bank size from 0 to 256 improves GenEval from 0.8132 to 0.8167 and IS from 40.13 to 40.46. These results support cross-batch support sets for improving image quality while preserving prompt-following performance.

  3. Representation and division of responsibilities. Drifting Models apply their objective to encoder features at multiple scales and spatial locations. We use the frozen DINOv2 encoder’s final CLS representation to provide an image-level constraint. Holding the field formulation and loss weight fixed, our appendix ablation compares this choice with multi-layer patch-feature aggregation and local feature statistics. These alternatives achieve DPG scores of 81.430–81.970, whereas the final CLS representation reaches 83.194. This controlled comparison supports an image-level field constraint within our distillation framework.

W1.2&Q5 Contributions of DINO-Adv

ADD was an early distillation method to use a discriminator operating directly on RGB images. The subsequent LADD and DMD2 adopted a discriminator based on fake-score network features for latent text-to-image generation and demonstrated that this design outperforms direct RGB-space discrimination. Our central finding is that pixel diffusion calls for a different choice from latent diffusion: this discriminator design does not transfer unchanged to native pixel-space DMD. In a controlled replacement experiment, switching from fake-score feature discrimination to a frozen DINOv2 discriminator operating on RGB images raises four-step GenEval from 0.7709 to 0.8155 and reduces training time per step from 1.82 s to 0.72 s, a speedup of approximately 2.5×. These results demonstrate the quality and efficiency gains from reconsidering the discriminator when moving DMD from latent to pixel space. Our contribution is this experimentally validated design choice for pixel-space distillation, and we have expanded the related-work discussion about ADD.

AuthorsOpenReview ↗

Follow-Up on Rebuttal Discussion

Dear Reviewer KTzv,

We deeply appreciate your valuable feedback during the first round of review and the thoughtful discussion that has significantly helped us refine our work. Since the discussion phase ends on Nov 18, we would like to know whether we have addressed all the issues, and we would greatly welcome any additional feedback or suggestions you may have.

Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.

Best regards,

Paper 7251 Authors

AuthorsOpenReview ↗

Kindly Reminder for Potential Discussion

Dear Reviewer KTzv,

We sincerely thank you for your valuable feedback during the first round of review and for the thoughtful discussions that have greatly contributed to improving our work. Your insights and suggestions have been instrumental in refining our submission, and we are deeply grateful for your time and effort.

We kindly wish to confirm whether we have satisfactorily addressed all your concerns. Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.

Best regards,

Authors

Reviewer_cdwtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DMA² adapts DMD to few-step native pixel-space text-to-image generation. It combines a high-noise matching band, a frozen DINOv2 discriminator with lightweight heads, and an auxiliary feature-space attract-repel loss. Experiments compare one- and four-step students against several distillation baselines and test transfer to a second teacher.

Critical strengths

The method has highly motiation. The baselines use the same teacher, data, and training budget, which makes the comparisons useful. The discriminator swap improves quality and reduces training time from 1.82 to 0.72 seconds per step in the reported setting.

Critical weaknesses
  1. The contribution beyond existing methods is unclear. AF-Loss closely follows Drifting Models in its feature-space stopped target, attract-repel field, row/column softmax, and normalization across multiple temperatures. Its additions appear mainly to be integration with conditional DMD, a rolling generated-feature bank, and a shared DINO encoder. DINO-Adv also follows the frozen-feature discriminator design used in ADD. These adaptations are useful, but the paper should distinguish them more clearly from the underlying methods.

  2. The explanation for DM-Band needs stronger evidence. C2ST measures real/generated separability, while the DMD update comes from the teacher and fake-score networks. Figure 4 and the band sweeps support the chosen interval, but the limited, independently normalized visualizations do not show how consistently local texture dominates that update. Table 2 also omits the band-only and band+AF configurations, leaving unclear whether the band's benefits persist without DINO-Adv.

  3. Some gains over the teacher are very small: 0.0016 on GenEval and 0.0003 on CLIP.

Questions for authors
  1. Which parts of Algorithm 1 differ from the drifting-field implementation, and which of these changes account for the gains?
  2. Can you provide the missing DM-Band configurations and quantify the frequency content of the actual teacher/fake-score difference across multiple samples?
5 comments
AuthorsOpenReview ↗

Response to Reviewer cdwt (Part 1/2)

W1&Q1 Differences Between AF-Loss and Drifting Models

AF-Loss adapts the drifting-field formulation of Drifting Models. from class-to-image training from scratch to distillation-based post-training of pretrained text-to-image models. Three design choices make this adaptation effective.

  1. Supervision and training role. In the ImageNet setting of Drifting Models, the drifting loss is the core objective for training a class-conditional generator from scratch, with support samples grouped by class. Open-domain text-to-image generation typically lacks both a fixed class partition and multiple real images for each complex prompt. We retain text-conditioned DMD for teacher and prompt supervision and introduce AF-Loss as an additional real-data objective that constrains the feature distribution across prompts. This division of responsibilities allows AF-Loss to complement distillation and adversarial supervision. Adding AF-Loss to the DINO-Adv configuration improves DPG from 82.550 to 83.427.

  2. Support-set construction. We maintain continuously updated real and generated feature banks across batches without grouping samples by class. Current and recent generated features jointly provide the repulsive support. The updated bank-size ablation supports retaining historical features: increasing the bank size from 0 to 256 improves GenEval from 0.8132 to 0.8167 and IS from 40.13 to 40.46. These results support cross-batch support sets for improving image quality while preserving prompt-following performance.

  3. Representation and division of responsibilities. Drifting Models apply their objective to encoder features at multiple scales and spatial locations. We use the frozen DINOv2 encoder’s final CLS representation to provide an image-level constraint. Holding the field formulation and loss weight fixed, our appendix ablation compares this choice with multi-layer patch-feature aggregation and local feature statistics. These alternatives achieve DPG scores of 81.430–81.970, whereas the final CLS representation reaches 83.194. This controlled comparison supports an image-level field constraint within our distillation framework.

W1 Contributions of DINO-Adv

ADD was an early distillation method to use a discriminator operating directly on RGB images. The subsequent LADD and DMD2 adopted a discriminator based on fake-score network features for latent text-to-image generation and demonstrated that this design outperforms direct RGB-space discrimination. Our central finding is that pixel diffusion calls for a different choice from latent diffusion: this discriminator design does not transfer unchanged to native pixel-space DMD. In a controlled replacement experiment, switching from fake-score feature discrimination to a frozen DINOv2 discriminator operating on RGB images raises four-step GenEval from 0.7709 to 0.8155 and reduces training time per step from 1.82 s to 0.72 s, a speedup of approximately 2.5×. These results demonstrate the quality and efficiency gains from reconsidering the discriminator when moving DMD from latent to pixel space. Our contribution is this experimentally validated design choice for pixel-space distillation, and we have added the related-work discussion about ADD.

AuthorsOpenReview ↗

Follow-Up on Rebuttal Discussion

Dear Reviewer cdwt,

We deeply appreciate your valuable feedback during the first round of review and the thoughtful discussion that has significantly helped us refine our work. Since the discussion phase ends on Nov 18, we would like to know whether we have addressed all the issues, and we would greatly welcome any additional feedback or suggestions you may have.

Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.

Best regards,

Paper 7251 Authors

AuthorsOpenReview ↗

Kindly Reminder for Potential Discussion

Dear Reviewer cdwt,

We sincerely thank you for your valuable feedback during the first round of review and for the thoughtful discussions that have greatly contributed to improving our work. Your insights and suggestions have been instrumental in refining our submission, and we are deeply grateful for your time and effort.

We kindly wish to confirm whether we have satisfactorily addressed all your concerns. Thank you again for your devotion to the review. If all the concerns have been successfully addressed, please consider raising the scores after this discussion phase.

Best regards,

Authors

Comments on the paper

1 comment

Thank you for the author's reply. I still have some concerns: On AF-Loss, Appendix D states that “the field design follows Drifting Models,” and Algorithm 1 keeps its core components: the stopped target, the attract–repel field, the doubly normalized softmax, and multi-temperature normalization. However, the main text says only “inspired by,” and the contribution list presents the renamed AF-Loss as a new objective. The differences in the rebuttal (auxiliary use with DMD, rolling banks, CLS features) concern how the field is applied, not the field itself, and their gains are modest. The author didn't credit Drifting Models in the contributions and Section 3.3 and frame AF-Loss as an adaptation. I maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.