acceptodds
Withdrawn from ICLR 2027

Consistent Distribution Matching for Data-Free Diffusion Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only **two** models, a frozen teacher and a trainable student, and optimizes **one** objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256256, our method attains an FID of **2.05** with a single function evaluation (1-NFE) and a 4-NFE FID of **1.37** within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data.

Primary area:
generative models
Keywords:
flow matching, distillation, diffusion models, flow maps

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yz2XOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

CDMD distills a flow teacher into a MeanFlow-style average-velocity student uθ(x,t,r)u_\theta(x,t,r) without data. Training states come from the student itself (re-noised one-step samples, or its own intermediate states, mixed with ratio mm; Eq. 6, Alg. 1). The method is presented as distribution matching in which one network acts as both generator and fake score. Validation: ImageNet-256 with three teachers (1-NFE FID 3.47 / 2.93 / 2.05, 1.37 at 4 NFE), a 2D benchmark, and a 4-step SANA-1.6B student.

Critical strengths
  1. A simple pipeline. One frozen teacher, one student and one loss, with no fake-score network or discriminator, and 20 GB peak memory vs. 33/37 GB for DMD2/FreeFlow (Fig. 1), reaching 2.05 FID at 1 NFE without data. Simplicity and computation savings are the main practical value of the paper.
  2. A useful diagnostic. The 2D experiment shows that endpoint matching does not imply transport matching: equal endpoint W2W_2, but much lower average-velocity error than FreeFlow (Table 6).
Critical weaknesses
  1. The method is not the distribution-matching objective it is presented as, and not new.
  • Eq. (7) is just the Eulerian flow-map distillation loss of Flow Map Matching (arXiv 2406.07507) and AYF-EMD (arXiv 2506.14603), evaluated on student-generated states.
  • TwinFlow (arXiv 2512.05150) and APEX (arXiv 2604.12322), which do share one network between generator and fake score, is not cited.

While it is totally possible to provide new understanding and perspective for existing loss (on-policy meanflow distillation), the authors failed to present the paper in this way and make it solid.

  1. The state-of-the-art claim does not hold against published numbers.
  • FreeFlow (arXiv 2511.19428) reports 1-NFE FID 2.24 with the same SiT-XL/2 teacher after about 20 epochs (1.45 with REPA after 300 epochs). Table 1 lists FreeFlow at 4.13 / 3.72 / 4.28 without marking these as re-runs, and CDMD at 3.47 with SiT-XL/2.
  • App. E (l. 1329–1331) also says the SiT-XL/2 settings and results "are left empty until stable training outcomes are obtained", yet Table 1 reports them, and the headline LightningDiT run alone uses the DMF architecture, post-hoc EMA and a guidance interval (Table 5).
Questions for authors
  1. Can you derive ∇θLCDMD\nabla_\theta\mathcal L_{\text{CDMD}} and explain in what sense uθ+(t−r)ddtuθu_\theta+(t-r)\tfrac{d}{dt}u_\theta along the teacher tangent estimates the velocity of qtθq^\theta_t (l. 256)?
  2. What's the main question of the paper? can we distill a diffusion or flow teacher into a few-step student without external data, teacher rollouts, or auxiliary fake score models? or Our method aligns the teacher and student marginal probability paths without data or simulation., the later one is already addressed in many previous works
Reviewer_CvVbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper models the student model as a flow map model, enabling the student model itself to approximately output the marginal velocity of its own few-step generated distribution. This removes the need to train an additional fake model in distillation process, which substantially accelerates training and reduces GPU memory consumption. The paper implements the method on ImageNet pretrained models and SANA, demonstrating its practical applicability.

Critical strengths
  • Score distillation methods, represented by DMD and SiD, require training an additional fake model, which indeed incurs considerable GPU memory and training time overhead. Studying how to perform distillation without a fake model is therefore highly meaningful.
  • The proposed method models the student model as a flow map model, which, without introducing many additional parameters, endows the model with both the ability to output an approximate marginal velocity and the ability to serve as a few-step generator. This design is very well-suited and holds potential and value for further research.
Critical weaknesses
  • The paper models the student model as a flow map model taking two timesteps as input, and then sets vfake≈uθ(xt,t,r)+(t−r)ddtuθ(xt,t,r)v_{\text{fake}} \approx u_\theta(x_t, t, r) + (t - r)\frac{d}{dt}u_\theta(x_t, t, r). Equality in this formula holds if and only if the student uθu_\theta is a flow map model that has already fitted the teacher's ODE trajectories. However, the paper imposes no constraint on the student flow map, which means the discrepancy between the two sides of this approximation could be very large, and it is questionable whether the approximation can be directly adopted. Regarding this point, the authors do not discuss an upper bound on this error, its potential impact, or any justification that motivates them to directly use this approximation.
Questions for authors

Please refer to "Critical Weaknesses".

1 comment

Thank the authors for their detailed response. I still have some questions regarding the mathematical aspects, so I will maintain my current score. I appreciate the value of this work and hope the authors will consider incorporating the points discussed into a revised version for resubmission.

Reviewer_6SEGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CDMD, which distills diffusion/flow-matching teacher models into few-step or even one-step generative student models. The core design is to parameterize the student network as an average velocity field, allowing the same network to simultaneously serve the two roles of "one-step generation" and "estimating its own marginal velocity / replacing the auxiliary fake score network," thereby removing the auxiliary network in DMD-type methods, and claiming that it does not require external training data and does not simulate teacher trajectories.

Critical strengths

Clear motivation: it performs "subtraction" targeting the problems of existing distillation methods relying on external data, teacher simulation, and auxiliary networks; the direction is concise and has practical value.
Attractive method design: using the same student network to simultaneously perform generation and velocity estimation avoids an additional fake score network and is engineering-wise simple.
Good experimental coverage: multiple teacher models and text-to-image tasks increase the persuasiveness of generalization.

Critical weaknesses

Baseline fairness and resource comparisons are not sufficiently transparent. Information such as batch size, precision, and gradient checkpointing needs to be added.
Key method ablations are missing, especially a direct comparison for the core design "student self-replaces the auxiliary network." More ablation experiments are needed to support each module proposed in the paper.
The text-to-image part lacks visual comparisons with baseline methods, making it difficult to intuitively judge the generation quality advantage.

Questions for authors

See the Strengths and Weaknesses.

Comments on the paper

1 comment
Area_Chair_1JHwOpenReview ↗

Reminder: Discussion Deadline — November 18

Dear Reviewers,

Thank you for your valuable time and contributions to ICLR. This is a friendly reminder to review the authors' responses and engage in discussions with them. The discussion deadline is November 18.

Thank you for your continued efforts!

ACs

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.