acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Regularized Distribution Distillation for One-Step Generation

Abstract

Distilling flow matching models for one-step generation generally follows two paradigms. Trajectory alignment supervises the student with the teacher's probability flow ODE, but its performance degrades under very small sampling budgets. Distribution matching directly optimizes the generated distribution, but estimating its reverse Kullback-Leibler gradient typically requires a separate network to track the evolving student score. We propose trajectory-regularized distribution distillation (TRD), which combines distribution-level and trajectory-level supervision through a momentum-mixture policy predicted in a single student forward pass. This policy serves simultaneously as a one-step generator, an estimator of the student's marginal velocity that removes the need for a second trainable network in distribution matching, and a closed-form time-dependent velocity field supervised along the teacher ODE trajectory. Experiments on SANA-1.6B and Stable Diffusion 3.5 show that TRD outperforms representative consistency and distribution matching baselines in generation quality and training efficiency for both one-step and few-step generation, with a trade-off in sample diversity.

Primary area:
generative models
Keywords:
flow matching, diffusion models, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wcPtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TRD combines ArcFlow's momentum-mixture velocity policy with distribution matching distillation. One student network does closed-form one-step generation, self-estimates the marginal velocity on re-noised samples and regresses teacher velocities on short trajectories, so no separate fake network is needed. An EMA endpoint term adds stability. Experiments on SANA-1.6B and SD3.5-Medium at 1024×1024 show clear one-step gains over retrained baselines at some cost in diversity.

Critical strengths
  1. Shared self-estimation is an identifiable contribution. Each part has precedents, but using one policy network for generation, velocity estimation and trajectory supervision removes the fake network, and the ablations show these choices matter. I don't think it is fair to call it a renamed method.
  2. I think the results is the most convincing part. One-step FD-DINOv2 is about 70 versus 97 for the next best baseline on SANA, and about 87 versus 118 on SD3.5-Medium, across many metrics and baselines. This is the main reason I see the paper as close to borderline.
  3. The notation table, algorithm box and configurations are complete, Proposition 1 shows the four losses share an ideal target, and the authors are upfront about limits such as the diversity cost, single runs and how CFG affects the theory.
Critical weaknesses

1 The gain over well-tuned distribution-matching baselines is not isolated. The main table matches inference NFE, but Fig. 1 shows different training lengths. All baselines were retrained by the authors with batch size 8, and full budgets and convergence are not reported. Removing L_fm and L_dm together cannot separate them, and removing L_tr or changing the fake estimator also shifts quality. So it is unclear how much of the advantage comes from the new components rather than budget, tuning and the EMA term. My concern is that the increment is not isolated, not that quality comes only from the distribution term. 2 The efficiency claims cannot be checked, and the trade-offs are not quantified. Table 2 gives no common FD target or timing rule, and its "Trainable params" do not match the description of a frozen backbone with LoRA. Every result is a single run, so the small two-step leads are not significant, and DINO Diversity is the lowest in three of four settings. This weakens the claim that quality and efficiency improve together. 3 The momentum-mixture parameterization, closed-form integration and teacher-velocity regression come from ArcFlow, as the paper acknowledges. Mixing trajectory and distribution objectives has precedents in DMD, rCM and SANA-Sprint, and rCM is not cited. Removing the fake network is also explored by concurrent work such as FSF-DMD and TwinFlow. The new part is shared self-estimation within this combination. I think this is a reasonable contribution only.

Questions for authors

Maybe more refer to major weakness. And what are the common FD target, the first-hit rule and the time units in Table 2, and does the 1.9B trainable count include the frozen backbone? What are the full training budgets and convergence curves of each method, and could you compare TRD with the main baselines and MeanFlow at a common time budget? Do the conclusions hold when L_fm and L_dm are removed separately, and when the baselines get the same EMA term? How is the separate fake network in Table 7 configured? Do you have repeated runs to quantify the uncertainty of the one-step and two-step results?

3 comments

Thank you for recognizing the empirical strength of TRD and the clarity of our analysis and presentation. We address the concerns about comparison fairness, efficiency, novelty, and the requested experimental details below.

W1: Fairness and isolation of the gains

Baseline training. We trained every reimplemented baseline until its performance had converged and tuned its method-specific hyperparameters to obtain its best result. All methods use the same evaluation protocol. The right panel of Figure 1 reports the resulting one-step FD-DINOv2 curves on a common normalized wall-clock axis. Its horizontal axis is logarithmic, so equal horizontal distances represent multiplicative rather than additive increases in training time and the late portions are visually compressed. The reported endpoints correspond to the converged checkpoints. The training budgets and per-iteration costs are reported in Table 2.

Isolation of the components. We agree that the original ablations do not fully separate all effects. We therefore performed two additional controlled studies. First, we conducted a full 2×22\times2 ablation that independently varies the estimator and trajectory supervision, demonstrating that their contributions are distinct and complementary. Please refer to our response to reviewer YinY, Q2, for the complete results.

Second, we separately evaluated the EMA-based distribution-preservation loss by adding Ldp\mathcal L_{\mathrm{dp}} to DMD and removing it from the separate-estimator TRD variant. The results show that Ldp\mathcal L_{\mathrm{dp}} alone does not explain TRD's gain over DMD and instead specifically stabilizes the coupled shared-estimator training. Please refer to our response to reviewer 95e6, W2, for the complete results.

W2: Scope of the quality and efficiency claims

Section 5.3, Table 2, and the right panel of Figure 1 report per-iteration time, peak memory, full training time, and time-quality curves. These results support a specific efficiency claim relative to fake-model distribution matching: TRD removes the auxiliary fake model, reduces peak memory from 75/80 GB for DMD/DMD2 to 45 GB, and reaches its converged result in 4,500 normalized time units rather than 10,000/12,000.

For quality, the main SANA results in Table 1 and the SD3.5-Medium results in Table 5 show that TRD achieves the strongest one-step results on the primary fidelity, preference, aesthetic, and text-alignment metrics. The two-step results remain strongest on most preference and quality metrics. We also explicitly report the diversity and coverage trade-off: TRD is not best on DINO Diversity, and some two-step baselines have higher Recall-DINOv2.

W3: Contribution relative to ArcFlow and related methods

We agree that TRD inherits ArcFlow's momentum-mixture parameterization and that hybrid supervision and fake-score-free estimation have precedents. TRD's contribution is the way the ArcFlow curve-valued output integrates these signals. The same predicted curve serves as the endpoint generator, supplies its start-time value as the student's marginal-velocity estimator, and supplies its values over the teacher grid for trajectory supervision. Distribution and trajectory supervision therefore act on different evaluations of one policy, with no separately optimized fake model.

Our response to reviewer 95e6 W1 gives a detailed comparison with FSF-DMD, TwinFlow, Self-Evaluation, SANA-Sprint, rCM, and TDM. We will also add the missing rCM citation.

Q1: What are the common FD target, first-hit rule, and time units in Table 2?

The FD target refers to convergence rather than a fixed numerical FD-DINOv2 threshold shared by all methods. We use a standard plateau criterion: FD-DINOv2 is considered converged at the earliest checkpoint after which the best value improves by less than 1% over three consecutive evaluation intervals. The reported time is the wall-clock time to this first converged checkpoint, normalized relative to DMD: the duration of one DMD training update is defined as one time unit.

Q2: Does the 1.9B count include the frozen backbone?

Thank you for pointing this out. The 1.9B value includes the frozen 1.6B backbone and the added policy parameters, so the column should be labeled "Total params" rather than "Trainable params". We will correct it.

Q3: Do the conclusions hold when Lfm\mathcal L_{\mathrm{fm}} and Ldm\mathcal L_{\mathrm{dm}} are removed separately?

We also tried removing Lfm\mathcal L_{\mathrm{fm}} and Ldm\mathcal L_{\mathrm{dm}} separately, and either removal caused training to collapse. These two losses form one coupled distribution-matching mechanism and are jointly required by the mathematical training objective: Lfm\mathcal L_{\mathrm{fm}} identifies the student's marginal velocity, which Ldm\mathcal L_{\mathrm{dm}} then uses together with the teacher velocity to update the generated distribution toward the teacher distribution.

Q4: What happens when a baseline receives the same EMA term?

We conducted controlled experiments that add Ldp\mathcal L_{\mathrm{dp}} to DMD and remove it from the separate-estimator TRD variant. Please refer to our response to reviewer 95e6, W2, for the complete results and analysis.

Q5: How is the separate fake network in Table 7 configured?

It is a conventional full-size DMD fake-velocity network initialized from the teacher. Only the student marginal-velocity estimate used by distribution matching is replaced by the separately trained network.

Q6: Do you have repeated runs?

We did not repeat each experiment or compute run-level confidence intervals. Each full-scale training run at 1024×10241024\times1024 is computationally expensive, and repeated training with confidence intervals is uncommon in large-scale diffusion distillation; DMD, DMD2, SANA-Sprint, and ArcFlow likewise report point estimates without run-level confidence intervals.


Thank you again for the detailed and constructive review. Your questions identified several places where the comparison protocol and the scope of our claims should be stated more precisely. Please let us know if you have any further questions and we are happy to discuss them.

I appreciate the authors’ efforts. The new ablations clarify the role of trajectory supervision and show that the EMA term alone does not explain TRD’s gain over DMD. After considering the other reviewers’ comments, my main concern remains whether TRD retains its reported advantage over well-tuned baselines at the same training budget. I will therefore maintain my original score.

Reviewer_95e6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TRD, a method for distilling flow-matching models into one-step or few-step generators. It builds on ArcFlow’s momentum-mixture policy, where a single forward pass of the student gives a velocity that changes over time. The authors reuse this one output in three ways. It generates the final image in closed form, it estimates the student’s own marginal velocity in place of DMD’s separate fake network, and it is matched to the teacher’s velocities along a short trajectory. An EMA-based preservation loss is added to keep training stable. Experiments on SANA-1.6B and SD3.5-Medium show better one-step FD-DINOv2 and preference scores than consistency and DMD-style baselines, with lower memory use than DMD, but sample diversity drops.

Critical strengths

-A single policy output that replaces DMD’s fake network. Removing the fake network reduces trainable parameters and peak memory compared with DMD and DMD2

-The paper is clear and well-written. It gives a clear analysis of how its losses interact. Proposition 4 shows that trajectory alignment only rescales the distribution-matching step and leaves its direction unchanged, which clarifies the role of each term. The authors also state the limits of their theory openly, for example that teacher guidance breaks the endpoint guarantee.

Critical weaknesses

-The novelty is limited and incremental. Having the generator estimate its own fake velocity is already done in TwinFlow [1], FSF-DMD [2] and Self-Evaluation [3]. Also, combining trajectory and distribution supervision already appeared in SANA-Sprint [4], rCM [5] and TDM [6]. The policy comes from ArcFlow [7] and the trajectory loss is adapted from it. The main new part is training the policy’s start-time velocity as the self-estimator.

-Training stability relies on the distribution preservation loss, and Table 6 shows that training collapses without it. The paper does not explain why. A likely reason is that the self-estimator moves together with the generator. If so, this term is essential to the shared-estimator idea and needs analysis like training the separate-network variant of Table 7 without L_dp. It is also not stated whether the DMD baselines use this stabilizer. If they do not, part of TRD’s gain over DMD in Table 1 may come from this term rather than the shared estimator. Adding the same loss to DMD would clarify this.

[1] Cheng, Z., Sun, P., Li, J., and Lin, T. TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flows. ICLR 2026.

[2] Kim, Y., Lee, D., and Park, J. Distribution Matching Distillation without Fake Score Network. arXiv:2605.19256, 2026.

[3] Yu, X., Qi, X., Li, Z., Zhang, K., Zhang, R., Lin, Z., Shechtman, E., Wang, T., and Nitzan, Y. Self-Evaluation Unlocks Any-Step Text-to-Image Generation. CVPR 2026.

[4] Chen, J., Xue, S., Zhao, Y., Yu, J., Paul, S., Chen, J., Cai, H., Han, S., and Xie, E. SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation. ICCV 2025.

[5] Zheng, K., Wang, Y., Ma, Q., Chen, H., Zhang, J., Balaji, Y., Chen, J., Liu, M.-Y., Zhu, J., and Zhang, Q. Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency. ICLR 2026.

[6] Luo, Y., Hu, T., Sun, J., Cai, Y., and Tang, J. Learning Few-Step Diffusion Models by Trajectory Distribution Matching. ICCV 2025

[7] Yang, Z., Tu, S., Zhang, L., Dai, Q., Jiang, Y.-G., and Wu, Z. ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation. arXiv:2602.09014, 2026.

Questions for authors

-Every baseline is a reimplementation trained with LoRA at a global batch of 8, and the authors say these are not reproductions of the original training regimes. This leaves open whether the large gaps reflect TRD or under-trained baselines.

-Why does the main paper use K = 4 when K = 2 is better on every metric in Table 8?

-How does TRD compare with the SANA-Sprint 1.6B under your protocol?

2 comments

Thank you for recognizing the clarity of the paper, the practical value of removing the fake network, and the analysis of how trajectory alignment affects the distribution update. Below, we address your concerns regarding the novelty of TRD, the role of distribution preservation, and the scope and fairness of the baseline comparisons.

W1: Novelty relative to fake-score-free and hybrid distillation

We agree that TRD builds on the ArcFlow architecture and that self-estimation and hybrid supervision have related precedents. The distinction lies in how ArcFlow's curve-valued output makes the integration of these signals natural. A single student policy simultaneously serves as few-step generator, the estimator of the student's marginal velocity, and the ODE curve supervised along the teacher trajectory. Distribution and trajectory supervision act on different evaluations of the same predicted curve, rather than being two unrelated losses attached to a conventional generator.

This construction also gives a different theoretical target. Self flow matching identifies the exact student marginal velocity at its population optimum. The distribution update is therefore unbiased under these stated ideal conditions. In contrast, FSF-DMD substitutes the endpoint pseudo-velocity of a flow-map generator for the student marginal velocity. This imposed approximation generally differs from the true marginal velocity and consequently gives a biased distribution-matching direction.

The remaining related methods integrate information in materially different ways. TwinFlow learns self-adversarial flows through real and self-generated twin trajectories and does not use one teacher-supervised curved policy for both marginal distribution matching and pointwise trajectory prediction. Self-Evaluation obtains a training signal by evaluating the generator's own predictions, instead of explicitly identifying the marginal velocity through self flow matching. SANA-Sprint is centered on continuous-time consistency supervision together with adversarial distillation, and rCM likewise builds on continuous-time consistency with score regularization. TDM treats trajectories themselves as distributions through step-aware matching, whereas TRD combines pointwise teacher-velocity supervision with reverse-KL matching of noised marginals.

Tables 1 and 5 provide empirical evidence for this design: under the matched protocol, TRD achieves substantially stronger quality than TwinFlow and FSF-DMD on both backbones. We will revise the novelty discussion accordingly. TRD contributes a unified curve-valued parameterization with an exact ideal distribution target, providing a principled integration of trajectory and distribution distillation beyond simply adding their losses.

W2: Why distribution preservation is critical

We already provide a theoretical explanation for the necessity of Ldp\mathcal L_{\mathrm{dp}} in Proposition 5. With the EMA parameters fixed, it is a proximal penalty on the generator map and directly upper-bounds the Wasserstein drift of both the endpoint distribution and its noised marginals, as well as their KL drift. This control is important because the shared estimator tracks a target defined by the evolving student distribution. Without Ldp\mathcal L_{\mathrm{dp}}, the objective contains no explicit constraint on how far this target can move between updates. Proposition 5 explains the stabilizing role of the term, while Table 6 empirically shows that removing it causes collapse in our setting.

We agree that controlled experiments are needed to determine whether this dependence is specific to the shared estimator and how much Ldp\mathcal L_{\mathrm{dp}} contributes to the comparison with DMD. We therefore add the requested ablation and apply the same preservation loss to DMD. All rows report one-step SANA-1.6B experiments under the same evaluation protocol.

MethodFD-DINOv2 ↓\downarrowPick ↑\uparrowHPS ↑\uparrowImgRwd ↑\uparrowAesth ↑\uparrowCLIP ↑\uparrowRecall ↑\uparrowDiversity ↑\uparrow
TRD with Separate fake velocity network127.860.18610.2137-0.02935.950527.76030.17720.3337
TRD with Separate fake velocity network w/o LdpL_{\mathrm{dp}}125.220.18840.21110.00795.872627.81570.17810.3215
DMD150.110.18400.2129-0.33985.848328.15670.08540.3250
DMD w/ LdpL_{\mathrm{dp}}155.190.18350.2120-0.37905.914128.00830.09670.3234

These results show that Ldp\mathcal L_{\mathrm{dp}} alone causes no clear improvement or degradation: both the separate-estimator TRD variant and DMD obtain similar, mixed results with and without it. The gain of TRD therefore does not come from adding this loss to a standard distribution-matching objective. Its role is to stabilize the coupled shared-estimator training.

Q1: Are the reimplemented baselines under-trained?

Controlled reimplementation is a standard practice for comparing distillation objectives and is adopted by many recent methods, including ArcFlow, TwinFlow, rCM, etc.

We follow the same principle. All methods use the same teacher, LoRA capacity, dataset, resolution, global batch and evaluation protocol. Within these shared constraints, we retain each method's specific objective and tune its method-specific hyperparameters to obtain its best performance, rather than applying one optimization configuration indiscriminately. The reported gaps therefore compare the methods fairly under the same resource and evaluation setting.

Q2: Why is K=4K=4 used when K=2K=2 is better?

The goal of Table 8 is to study how changing KK affects training, rather than to search for the best KK for our main results. We use K=4K=4 as a fixed setting across both backbones and vary KK only in the controlled one-step SANA ablation. The results show a clear non-monotonic trend: increasing KK from 11 to 22 improves performance, while further increasing it to 44 or 88 hurts performance despite the larger model capacity. This finding shows that a larger KK does not always lead to better training and that KK should be selected for each setting. The result that K=2K=2 works best is limited to this one-step SANA experiment and does not establish a universal best choice. We will clarify the purpose of this ablation in the paper.

Q3: Comparison with SANA-Sprint 1.6B

We agree that adding a direct SANA-Sprint comparison makes the evaluation more convincing. We therefore add a one-step SANA-Sprint 1.6B reimplementation trained with the same dataset, LoRA capacity, resolution, and global batch as the other baselines. The TRD row is copied from Table 1.

MethodFD-DINOv2 ↓\downarrowPick ↑\uparrowHPS ↑\uparrowImgRwd ↑\uparrowAesth ↑\uparrowCLIP ↑\uparrowRecall ↑\uparrowDiversity ↑\uparrow
SANA-Sprint 1.6B (our reimplementation)86.380.19030.22060.31326.075128.21490.29940.3720
TRD70.340.19150.22230.38886.116928.36470.31620.3204

Thank you again for your thoughtful and constructive comments. We sincerely appreciate the opportunity to clarify these points. Please let us know if you have any further questions. We would be very happy to discuss them.

Reviewer_YinYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRD, combining trajectory supervision with distribution matching for one-step image generation. It uses a shared student velocity estimator to avoid training the additional fake-score network required by DMD. Experiments on SANA-1.6B and Stable Diffusion 3.5 show improvements in generation quality and training efficiency.

Critical strengths
  • Removing the separate fake-score network reduces memory requirements and simplifies the training setup.
  • The experiments show good one-step generation quality, and the ablations support the effectiveness of the shared estimator.
Critical weaknesses
  • The novelty is relatively limited. The method largely combines existing ideas from ArcFlow and DMD with additional trajectory regularization. The main contribution appears incremental.
  • The improvement comes with reduced diversity and some sensitivity to the design choices. In Table 8, using two momentum modes performs substantially better than the default four modes, which raises questions about the robustness of the proposed parameterization.
Questions for authors
  • Why is K=4 selected as the default when K=2 performs considerably better?
  • Can the authors better isolate the benefits of the shared estimator from those of the additional trajectory supervision?
1 comment

Thank you for recognizing the quality of TRD results and the practical value of removing the separate fake-score network. We agree that the relationship to ArcFlow and DMD, the choice of KK, and the separate roles of the shared estimator and trajectory supervision require a more precise discussion.

W1: Novelty relative to ArcFlow and DMD

We agree that TRD uses ArcFlow's momentum-mixture architecture and DMD's reverse-KL update. However, TRD is NOT a simple combination of their losses. The key is to use the ArcFlow policy as a shared model for both distribution and trajectory supervision. One policy does three jobs: it generates samples, estimates the student's marginal velocity for distribution matching, and predicts velocities along the teacher trajectory. This 3-in-1 design applies both signals to the same student and avoids training extra networks. Results on both backbones show clear gains (Tables 1 and 5). The ablations further confirm the value of each supervision block and the shared estimator (Tables 6 and 7). We will revise the paper to make this structural contribution clearer.

W2&Q1: Why use K=4K=4 when K=2K=2 is better?

The goal of Table 8 is to study how changing KK affects training, rather than to search for the best KK for our main results. We use K=4K=4 as a fixed setting across both backbones and vary KK only in the controlled one-step SANA ablation. The results show a clear non-monotonic trend: increasing KK from 11 to 22 improves performance, while further increasing it to 44 or 88 hurts performance despite the larger model capacity. This finding shows that a larger KK does not always lead to better training and that KK should be selected for each setting. The result that K=2K=2 works best is limited to this one-step SANA experiment and does not establish a universal best choice. We will clarify the purpose of this ablation in the paper.

Q2: Isolating the shared estimator from trajectory supervision.

We agree that this separation is important. The paper already studies this question from both results and mechanisms. Section 5.4 studies the two components through controlled ablations. It removes trajectory supervision while keeping the shared estimator (Table 6), and replaces the shared estimator while keeping trajectory supervision (Table 7). Section 5.5 then studies the mechanisms. It analyzes the shared estimator's velocity accuracy and the effect of trajectory supervision on teacher-path error (Figure 2).

To make the separation clearer, we design a full 2×22\times2 ablation table for one-step SANA. All runs use the same student structure, initialization, data, optimizer, etc.

EstimatorTrajectory supervisionFD-DINOv2 ↓\downarrowPick ↑\uparrowHPS ↑\uparrowImgRwd ↑\uparrowAesth ↑\uparrowCLIP ↑\uparrowRecall ↑\uparrowDiversity ↑\uparrow
SharedYes70.340.19150.22230.38886.116928.36470.31620.3204
SharedNo107.01‾\underline{107.01}0.1880‾\underline{0.1880}0.2179‾\underline{0.2179}0.3242‾\underline{0.3242}6.0409‾\underline{6.0409}28.06890.17030.3258‾\underline{0.3258}
Separate fake networkYes127.860.18610.2137-0.02935.950527.76030.1772‾\underline{0.1772}0.3337
Separate fake networkNo138.810.18630.21340.00415.919328.1294‾\underline{28.1294}0.10470.3225

The results clearly separate the two effects. The shared estimator improves fidelity, preference, and coverage with or without trajectory supervision, showing its independent benefit. Trajectory supervision further improves fidelity and coverage under either estimator, with larger gains when both components act on the same policy. This confirms that the two components are distinct and complementary.


Thank you again for your thoughtful and constructive comments. We sincerely appreciate the opportunity to clarify these points. Please let us know if you have any further questions. We would be very happy to discuss them.

Comments on the paper

1 comment
Area_Chair_6yxzOpenReview ↗

Reminder: Please Respond to the Authors’ Rebuttal

Dear Reviewers,

Thank you for your time and effort in reviewing this submission. Please take a moment to read the authors’ responses and follow up with any remaining questions or concerns. If the responses have changed your assessment, please update your review and score accordingly.

Your engagement is greatly appreciated and will help us reach a well-informed decision.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.