acceptodds
Under review as a conference paper at ICLR 2027

sDMD: Sinkhorn Distribution Matching Distillation

11 views

Abstract

Distribution Matching Distillation (DMD) distills multi-step diffusion models into few-step generators by minimizing reverse KL divergence. Despite its strong performance, DMD guides samples by local density alone, which introduces two limitations: teacher scores can be unreliable in regions of low teacher density, and the mode-seeking behavior of reverse KL can collapse the student onto a subset of the teacher modes. We first connect DMD to Wasserstein gradient flow, showing that its gradient follows from a fixed-point formulation with integral KL (IKL) as the energy functional. This perspective motivates augmenting IKL with an optimal transport term that addresses these limitations by guiding outlier samples toward the target and promoting mode coverage. We realize this term using Sinkhorn divergence estimated from finite batches. In practice, however, errors in the estimated Sinkhorn velocity can perturb in-domain samples for which IKL guidance is already accurate. To this end, we introduce sDMD, which adaptively activates the Sinkhorn correction based on discriminator outputs, focusing on outlier samples where IKL may be unreliable. With 2–4 sampling steps on FLUX.1-dev, Qwen-Image-20B, and Wan2.1-14B, sDMD outperforms distillation baselines on GenEval, DPG, WISE, and VBench while improving sample diversity over DMD2.

Primary area:
generative models
Keywords:
Distillation, DMD, Wasserstein Gradient Flow, diffusion, image generation, video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7fefOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes sDMD, a diffusion distillation method that augments DMD with Sinkhorn transport guidance. It interprets DMD as fixed-point regression under the Wasserstein gradient flow of an integral reverse-KL objective, and motivates transport regularization through two limitations: unreliable learned teacher scores in low-density regions and weak guidance toward underrepresented modes. The proposed method combines the original score-based update with a Sinkhorn velocity estimated from batches of student and target samples. To limit interference from finite-sample transport errors, it reuses the DMD2 discriminator to selectively activate this correction for samples with relatively weak target support. The paper provides convergence results for idealized, ungated flows under regularity and approximation-error assumptions. Experiments on class-conditional image, text-to-image, and text-to-video generation report improved sample diversity and generally competitive or improved quality relative to distillation baselines.

Critical strengths
  1. Theory and motivation. The paper identifies relevant limitations of DMD and presents a coherent rationale for combining score-based guidance with transport-based correction. Its connection between DMD, integral KL, and Wasserstein gradient flow provides a useful interpretation of the update fields. The theoretical analysis also distinguishes exact OT from its Sinkhorn approximation and explicitly states the assumptions required for the convergence results.

  2. Empirical evaluation. The evaluation spans class-conditional image generation, text-to-image generation, and text-to-video generation, including large models such as Qwen-Image and Wan2.1. The consistent improvements across the reported diversity metrics are encouraging. Ablations on the energy functional, transport space, gating mechanism, and batch size provide useful information about the method’s practical behavior.

Critical weaknesses
  1. Practical overhead. The reviewer is concerned about the additional requirements of generating multiple student and target samples per condition and computing both cross-distribution and self-transport. Although teacher samples can be cached, the end-to-end costs of sample generation, storage, and batching remain insufficiently characterized for large image and video models.

  2. Recovery from student mode collapse. The reviewer is unsure whether Sinkhorn updates can restore multiple modes when reverse-KL-based training has already concentrated outputs from different seeds around the same mode, rather than merely move these samples together. Clarifying the student sampler’s stochasticity would help assess this concern. As a possible extension, could controlled SDE-based exploration, inspired by flow-model RL methods such as Flow-GRPO, complement the transport correction?

  3. Limited evidence for gating. The reviewer finds the gains over ungated transport in Table 4 modest, with their robustness unclear in the absence of uncertainty estimates. The current comparisons also do not distinguish effective sample selection from simply reducing the strength or frequency of transport updates.

  4. The role of the WGF analysis in motivating the method. The reviewer recognizes that the WGF formulation provides a unified interpretation of the update fields, but finds it unclear how this perspective directly motivates the choice of Sinkhorn divergence or provides additional guidance for the method’s design. The motivation for adding transport appears to follow already from the mode-seeking behavior and unreliable local guidance of IKL-based matching. The constructive insight added by the WGF derivation could therefore be articulated more clearly.

Questions for authors

Please refer to the concerns raised in the Weaknesses section, with particular attention to points 3 and 4 regarding the gating mechanism and the role of the WGF analysis. The reviewer would be strongly inclined to raise the score if these concerns are convincingly addressed through clear explanations or supporting evidence.

Reviewer_SoXDOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

sDMD: Sinkhorn Distribution Matching Distillation improves few-step diffusion distillation by combining DMD’s local score guidance with Sinkhorn optimal transport. The authors show that DMD’s update can be interpreted as a Wasserstein gradient flow of diffusion-noise-averaged reverse KL. This view motivates a transport correction to guide outlier samples toward the teacher distribution and improve mode coverage. A discriminator activates that correction where teacher support appears weak.

The theoretical development is substantial, and the experiments are broad: the paper evaluates one-step ImageNet generation, two- and four-step text-to-image generation, and four-step text-to-video generation, with baseline comparisons, diversity measurements, and ablations. The results generally support improved quality and diversity over DMD2. One qualification for a review: the convergence guarantees apply to idealized continuous flows under stated assumptions; they do not directly establish convergence of the practical gated, finite-batch training procedure.

Critical strengths
  • Clear motivation and method. The paper identifies two plausible weaknesses of DMD: unreliable score guidance for outlier samples and limited mode coverage. Sinkhorn transport addresses both through a complementary, sample based update.
  • Substantial theoretical analysis. The Wasserstein gradient flow interpretation recovers the DMD update and gives a principled basis for the added transport term. The paper also states explicit conditions under which the combined flow converges.
  • Practical design. The discriminator gate focuses transport updates on samples likely to need them, while reusing a discriminator already present in DMD2.
  • Broad experimental evidence. Results span ImageNet, FLUX, Qwen-Image, and two Wan video models. The paper reports quality, prompt alignment, and diversity metrics, alongside ablations of the transport term, gate, feature space, and batch size.
Critical weaknesses

Missing sensitivity analysis for the gate threshold. The discriminator threshold λ controls which samples receive Sinkhorn guidance, making it central to sDMD. The paper reports the selected values and compares gated with ungated training, but does not ablate λ itself. A threshold sweep would show whether the gains are robust or depend on careful tuning.

Questions for authors

Could you provide an ablation of the gate threshold λ across several values, reporting generation quality and diversity? It would be helpful to see whether sDMD’s gains are robust to this choice across the evaluated models.

Reviewer_11xhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes sDMD, a diffusion distillation method that combines DMD’s score-based updates with a Sinkhorn transport correction. It interprets DMD through Wasserstein gradient flow under an integral KL objective and reuses a discriminator to selectively activate transport. Experiments on image and video generation report improvements in several quality and diversity metrics over DMD2 and other few-step baselines.

Critical strengths

Combining score-based guidance with sample-based transport is a plausible approach to improving distribution matching. The Wasserstein gradient-flow perspective provides a useful framework for combining these updates.

The experiments cover several large image and video models, including Qwen-Image, FLUX, and Wan2.1.

The method builds on the established DMD2 training framework and reuses its discriminator, while leaving the distilled generator’s inference procedure unchanged.

Critical weaknesses
  1. My main concern is whether the improvements arise from the transport structure or from adding an auxiliary teacher-sample regression constraint. With quadratic costs, the cross-transport update locally resembles regression toward a teacher barycenter; if the matching weights concentrate, this approaches regression toward an individual teacher sample. Although the full Sinkhorn correction seems to differ from DMD1’s fixed-pair regression, the concentrated weight reduce the regression to training sample reconstruction. This alternative explanation remains insufficiently tested. Controlled comparisons with paired regression, nearest-neighbor regression, and a well-specified MMD baseline are needed, with comparable data, representations, GAN supervision, and training budgets. The so-called OT complement may be just the merit from the reconstruction. The author should explain this.

  2. The gate’s independent value and proposed mechanism are insufficiently supported. Table 4 shows relatively small improvements from gating, but lacks controls such as random gating at the same activation rate or ungated transport with a reduced weight. Consequently, it remains unclear whether the discriminator identifies samples that benefit from transport, or simply reduces the frequency and strength of a noisy auxiliary update. The discriminator estimates a density ratio, which does not directly measure teacher-score reliability. The missing latent-space ungated comparison and gate ablations on the main Qwen/Wan settings further limit the evidence.

3 The theoretical guarantees have limited applicability to the final algorithm. The convergence analysis primarily concerns idealized, ungated distributional flows, whereas the implemented method includes discriminator gating, learned scores, finite transport batches, and adversarial training.

4 The evidence for improved mode coverage remains indirect. TCE, LPIPS, and pairwise feature distances support increased sample diversity, but do not establish that previously missing teacher modes have been recovered. Teacher-relative coverage and fidelity measurements would strengthen this claim. For FLUX, the fixed cache of eight teacher images per prompt also raises the question of whether transport improves coverage of the full teacher distribution or primarily fits a small empirical reference set. The ImageNet batch-size ablation does not resolve this distinction; cache-size or refresh experiments and evaluation against held-out teacher samples would be informative.

  1. Comparisons with closely related methods are insufficient. Although the evaluation covers multiple backbones, the baseline comparisons are not sufficiently comprehensive to establish the method’s advantages over existing distribution-matching approaches. In particular, direct comparisons with JS-based f-distill (Xu et al., 2025) and AMD (Bai et al., 2026) are missing, despite their relevance to mode-seeking behavior and unreliable teacher guidance. Other new distillation methods in 2025 and 2026 are also not compared.
Questions for authors

How does Sinkhorn compare with fixed-pair or nearest-neighbor teacher regression and a well-tuned MMD baseline? Can coupling-concentration measurements and separate cross-/self-transport ablations clarify which properties of transport contribute to the gains?

Does the proposed gate outperform random gating at the same activation rate and ungated transport with a reduced regularization weight? Is there direct evidence that the discriminator output predicts teacher-score error or the benefit of applying transport?

1 comment
Reviewer_11xhOpenReview ↗

after rebuttal

Thanks for response and the added experiments.

Some concerns are partly addressed. The gating control at a matched activation rate (R6) are useful evidence. Table R1 is also helpful.

However, my main concerns remain.

  1. The coupling measurement I asked for is not provided. In Table R2, the cross-transport term alone is worse than DMD2 (FID 3.72 vs. 3.14), while the self-transport term alone already reaches 3.00. This is hard to reconcile with the claim that the benefit comes from transporting outliers toward the target. The MMD baselines in R3 are also weaker than the IKL+MMD result in Table 3 of the paper (3.01). I still believe the so-called OT complement may be just the merit from the reconstruction. THIS is my main concern. To address this concern, the authors should compare an OT alternative which is not reconstruction. Objectively, I think what the sinkhorn coupling does is just reconstruction and this is unlikely to find the OT alternative.

The new evidence is a 2D toy example with ungated sDMD-U. No teacher coverage metric is reported, and R8 does not have DMD2 reference on the held-out samples.

3 The extension to learned scores is informal and still doesn't cover gating nor adversarial training.

4 The revised PDF adds only Appendix E.6. The new tables and the changed threshold are not incorporated

I appreciate the idea to incorporate OT with DMD, but this paper does not pave a way novel enough. I encourage the author to revise and resubmit to other conference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.