acceptodds
Withdrawn from ICLR 2027

Disagreement-Aware Distribution Matching Distillation for Few-Step Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Distribution Matching Distillation (DMD) enables strong few-step diffusion generation by matching the student distribution to a pretrained teacher. Existing improvements to DMD often augment its supervision or modify its optimization machinery. We ask a complementary question: can DMD diagnose its own updates using signals already produced during training? DMD inherently computes the discrepancy between a pretrained teacher and a fake model tracking the student distribution, but conventionally uses it primarily to determine the distribution matching direction. We find that the magnitude of this teacher–fake disagreement provides additional diagnostic information. Across extensive experiments, large disagreement in the high-noise regime consistently coincides with substantially higher-energy DMD updates. This reveals a dual role of disagreement: it determines not only where the student should move, but also provides an intrinsic signal of how aggressively the update should be applied. Based on this observation, we introduce Disagreement-Aware Distribution Matching Distillation (DA-DMD), which activates only in the high-noise regime and continuously modulates the update strength according to teacher–fake disagreement. DA-DMD reuses quantities already available during DMD training and requires no additional model, evaluator, or inference-time modification. We show that it exactly recovers vanilla DMD outside the selected region, reduces update energy only through selected samples, and exhibits redescending influence under extreme disagreement. Across multiple diffusion backbones, DA-DMD consistently improves few-step generation at the same inference cost. On SD3.5-L, DA-DMD improves HPSv2 from 28.23 to 30.93, while achieving a 64.4% average win rate over DMD2.

Primary area:
generative models
Keywords:
Diffusion Model, Step Distillation, Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yPh8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DA-DMD, a weighting scheme for distribution matching distillation based on disagreement between the teacher and fake score models. It starts from the observation that larger disagreement tends to coincide with higher update energy, particularly at high noise levels. The proposed rule attenuates distillation updates according to disagreement within a selected high-noise region. A distribution-mixture construction motivates the rule, and a smoothness analysis gives conditions under which attenuation improves a descent bound. Experiments across several diffusion backbones report improvements in image quality and prompt following.

Critical strengths
  1. The method makes a small, targeted change to DMD training: it uses the existing teacher–fake disagreement to control update strength in the high-noise region. This addresses unusually large updates without additional teacher evaluations or inference cost, making the reported improvements practically relevant.

  2. The experiments support the approach from several directions. Improvements are reported across five backbones, and the noise-region ablation supports restricting the intervention to high noise. The image comparisons on counting, text rendering, and attribute binding also show what the quantitative gains correspond to in generated outputs.

Critical weaknesses
  1. The experiments do not show whether disagreement-dependent weighting is better than simply using smaller updates in the same high-noise region. Since ωα(0)=α<1\omega_\alpha(0)=\alpha<1, even samples with little disagreement are attenuated. A constant weight applied in the same region and matched for average retained update strength is needed to determine what the adaptive weighting adds.

  2. The analysis explains when attenuation can help, but the evidence connecting that condition to disagreement is incomplete. Whether a smaller update improves the descent bound depends on update energy relative to useful gradient alignment, B(t,r)/A(t,r)B(t,r)/A(t,r). The reported correlation with update energy alone does not show that disagreement identifies updates whose magnitude is excessive relative to their useful contribution.

  3. The protocol for selecting α\alpha is not described precisely enough to reproduce. Appendix A provides the selected values and mentions a small candidate set, and Figure 4 examines sensitivity on SD3.5-M. However, the validation prompts and selection criterion are not specified. Since the preferred attenuation varies across backbones, these details are needed to assess the tuning required to obtain the reported gains.

Questions for authors
  1. How does DA-DMD compare with a constant attenuation factor applied in the same high-noise region, matched for average retained update strength?

  2. What evidence supports the assumption that B(t,r)/A(t,r)B(t,r)/A(t,r) increases with disagreement during training, beyond the observed correlation with update energy?

  3. Which candidate values, validation prompts, and selection criterion were used to choose α\alpha for each backbone?

  4. How is the average pairwise win rate computed? Please specify the judge, how results are aggregated across prompts, and how ties are handled.

Reviewer_jQCbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Disagreement-Aware Distribution Matching Distillation (DA-DMD), a modification to DMD for few-step diffusion generation. Motivated by the observation that large teacher–fake prediction discrepancies are associated with high-energy updates in the high-noise regime, the method uses noise-normalized disagreement to attenuate individual DMD updates. Attenuation is restricted to a predefined high-noise region, while updates elsewhere remain unchanged. The weighting rule is motivated by a teacher–fake distribution mixture and interpreted through an adaptive step-control analysis. DA-DMD reuses quantities already computed during training and introduces no additional models or inference overhead. Experiments across five text-to-image backbones and several benchmarks report improvements over the corresponding DMD2 baselines, while ablations examine the effects of the intervention region and weighting coefficient.

Critical strengths
  1. The paper is clearly written, and the method is simple and easy to implement. It reuses existing training signals without adding inference overhead and achieves improvements across multiple experimental settings, making it a simple yet effective modification.

  2. Experiments across multiple backbones provide evidence of the method’s applicability. The region ablations further demonstrate that intervention location matters and support applying attenuation in the high-noise region.

Critical weaknesses
  1. Insufficient clarification of the relationship to existing normalization. DMD2 already uses sample-wise normalization to regulate update magnitudes. The paper does not sufficiently explain why this mechanism is inadequate or what additional value the teacher–fake disagreement signal provides.
  2. The contributions of high-noise attenuation and disagreement adaptivity are not isolated. DA-DMD both attenuates updates throughout the high-noise region and applies additional attenuation as sample disagreement increases. Existing ablations do not establish whether the gains primarily result from the former.
  3. Direct evaluation of student–teacher distributional agreement is missing. Improvements in preference, visual quality, and instruction-following metrics do not directly establish that the student distribution is closer to the teacher’s. Using teacher-generated samples as the reference, the authors should compare the FID of DMD2, DA-DMD, and other distillation methods under matched prompt distributions, clearly specified sampling configurations, and sufficient sample sizes. This would provide more direct, complementary evidence of improved distribution matching.
  4. The connection between the theoretical derivation and the practical weighting rule remains insufficiently established. While the mixture formulation motivates sample-adaptive weighting, replacing the unavailable log-density ratio with −r2/2-r^2/2 is a heuristic choice rather than a consequence of the derivation. The high-noise threshold and model-specific attenuation coefficient also require empirical selection.
Questions for authors

Please refer to the weaknesses above.

Reviewer_c9fqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a mechanism that uses signals native to DMD to reweight gradient updates, guiding DMD training without additional model forward passes. It achieves improvements over the DMD2 baseline across different models and datasets.

Critical strengths
  • The method achieves reasonably consistent improvements across multiple models and evaluation benchmarks, supporting its effectiveness across the evaluated settings.
  • The method reuses signals already available during DMD training, improving performance with minimal computational overhead and no additional model forward passes.
Critical weaknesses
  • The central contribution appears to be a disagreement-dependent scalar attenuation of existing DMD updates within a predefined high-noise region. Closely related ideas have already been explored. In particular, f-distill [1] derives sample-dependent weighting of teacher–fake score differences and explicitly investigates how its weighting downweights samples with larger score discrepancies. Score-of-Mixture Training/Distillation [2] also derives the teacher–fake mixture-score construction, and introduces score-difference-based adaptive gradient weighting in its training formulation. Neither work is discussed in the current manuscript. I recognize the specific distinction of using a disagreement-only proxy without an additional discriminator and applying it selectively at high noise. However, the manuscript does not sufficiently establish why this particular combination constitutes a substantive methodological advance over these existing ideas.

  • The observation that larger teacher–fake disagreement is associated with higher-energy DMD updates is partly inherent in the update construction: Eq. (7) already makes the update depend directly on the teacher–fake prediction difference. Although the Jacobian and timestep-dependent factors prevent this relationship from being a strict identity, the observed correlation alone provides limited evidence of a distinct diagnostic insight. In particular, larger update energy does not establish that an update is overly aggressive relative to its useful descent signal. Besides, the disagreement between an estimated score and the real score is also large in diffusion training. Although may not be certain, such teacher–fake disagreement is rooted in diffusion but not in distillation only.

  • The smoothness analysis shows that attenuation can improve a one-step upper bound when update energy is sufficiently large relative to descent alignment. However, the connection to disagreement additionally assumes that the energy-to-alignment ratio increases with disagreement. The experiments principally examine update energy, rather than establish this crucial relationship.

  • Replacing the exact log-density ratio in the mixture construction with −r2/2-r^2/2 is explicitly a heuristic surrogate. Consequently, the analysis motivates conservative updates under certain conditions, but does not establish that the proposed disagreement measure identifies those conditions or that the particular exponential weighting is preferable to simpler alternatives.

The current ablations vary the intervention region and \alpha, but omit the constant attenuation within the same high-noise region, with a comparable tuning budget or matched retained update magnitude/energy. Comparisons against high-noise gradient clipping or normalization would also help establish whether the proposed mechanism offers benefits beyond ordinary magnitude control.

[1] Xu, Y., Nie, W., and Vahdat, A. One-step Diffusion Models with f-Divergence Distribution Matching. 2025. arXiv:2502.15681.

[2] Jayashankar, T., Ryu, J. J., and Wornell, G. W. Score-of-Mixture Training: One-Step Generative Model Training Made Simple via Score Estimation of Mixture Distributions. ICML, 2025. PMLR 267:27021–27049.

Questions for authors

Why do the authors believe that updates associated with greater disagreement should be attenuated? Why is the same attenuation mechanism beneficial in the high-noise region but less effective in the middle- and low-noise regions? Have the authors compared alternative attenuation designs, including replacing (r) with the existing normalized DMD update signal, using other surrogate energy or weighting functions, and applying a constant attenuation factor in the high-noise region? Do these comparisons support the advantages of the specific design adopted in the paper? How was the high-noise threshold of 0.7 selected, and what validation criterion was used? How sensitive are the results to this threshold? What are the exact boundaries of the middle- and low-noise regions in the region ablation study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.