acceptodds
Under review as a conference paper at ICLR 2027

StepX-OPD: Rethinking Multi-Teacher On-Policy Distillation in Diffusion Models

Abstract

Text-to-image diffusion models must combine broad capabilities, such as instruction following, text rendering, perceptual quality, and human preference, with the few-step sampling that deployment requires. On-policy distillation supervises a student on states it generates itself, consolidating capabilities from several specialized teachers into one model, while step distillation compresses a sampling trajectory into a few steps; we study combining the two. However, existing pipelines instead chain them: a full-step multi-teacher model is trained first and compressed only afterward, so compression waits on that stage, and the capabilities it consolidates are not guaranteed to survive compression. To address this, we introduce StepX-OPD, a multi-teacher on-policy distillation method that jointly consolidates specialist capabilities and compresses the sampling trajectory: after a brief self-distillation warmup, each mini-batch is drawn from one training source and routed to the frozen reward-specialized teacher for that source, which drives distribution matching on states supplied by the student's own few-step rollout. Furthermore, on low-noise batches the routed teacher also regresses the student's velocity field directly, balancing capability transfer against the fidelity of the few-step trajectory itself. Baseline evaluations span SD3.5-Medium and FLUX.1-dev, scoring GenEval, OCR, DeQA, PickScore, and their normalized average. The quantitative result reported here for StepX-OPD is on SD3.5-Medium, where it attains the highest Average among the four-step methods compared and exceeds the Average of the forty-step Flow-OPD model. Jointly optimizing capability integration and step compression on one student trajectory avoids using a consolidated full-step model as the teacher for a later compression stage.

Primary area:
generative models
Keywords:
On-Policy Distillation, Diffusion Models, Step Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TiZtOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes a diffusion distillation method StepX-OPD, which can distill multiple specialized teacher diffusion models into a 4-step student model in one stage. StepX-OPD combines several techniques from step distillation and on-policy distillation methods for diffusion models. The authors conduct experiments on two text-to-image generation benchmarks and compare the proposed method with sequential (two-stage) baselines. The paper also investigates the behavior of unintended rendering of propmpt words and explores a teacher-rollout target to mitigate this issue.

Critical strengths
  1. StepX-OPD combines capability consolidation and few-step distillation for diffusion models in one stage, which reduces the training costs and improves inference efficiency. The motivation is clear and the proposed method addresses a meaningful problem and could be practically useful for deployment.
  2. Text-to-image generation results on SD3.5-Medium and FLUX.1-dev demonstrate that StepX-OPD achieves comparable performance to specialized teacher models and improved overall performance over sequential (two-stage) OPD+TDM baselines.
  3. The investigation of unintended text generation with benchmark stratification and additional quality metrics is informative and is a nice addition to the main results.
Critical weaknesses
  1. The contribution of the proposed method is not clearly positioned in the literature. StepX-OPD combines several techniques from existing step distillation and on-policy distillation methods for diffusion models (e.g., SFT-augmented initialization, specialist teachers, source-specific training data and so on). It is unclear whether all of these techniques are necessary and how each of them affect the final performance. More ablation study is needed to isolated the core contribution of this paper from closely related methods.
  2. The presentation of the paper needs improving. Section 4 (method section) should start with a big picture of how the training recipe looks like and then describe each component in the method (e.g., why each component is necessary and how it works). Currently, the description of the proposed method is hard to follow. For example, it starts by explaining the difference between the schedule of CDM and TDM, which is very confusing.
  3. It is unclear whether the performance improvement is significant. For several quality measures (especially PickScore), StepX-OPD underperforms the best baselines on both benchmarks. Also, StepX-OPD's improved average score on FLUX.1-dev is mainly driven by the DeQA score. There is also a concrete contradiction: Appendix E says increasing OCR source’s weight improves both configurations in the text, but Table 11 shows the merged configuration’s OCR and Average score both noticeably decrease with increased OCR sampling.
Questions for authors
  1. Could the authors provide a matched comparison against routed TDM/CDM using exactly the same configurations as StepX-OPD (same initialization, teachers, prompt sources and training budget and so on)?
  2. Could the authors conduct an ablation study with wv=0w_v=0?
  3. For the two-stage sequential baseline, what if you first distill specialized teachers into few-step models and then use OPD to consolidate them into one model?
  4. Why do you only consider a 4-step student model in the experiment? How do StepX-OPD and other baselines perform for 1-step and 8-step settings?
8 comments

We thank the reviewer for reading our manuscript carefully and for these constructive comments. We respond to each point below.

W1: The position of the contribution in the literature is unclear. StepX-OPD combines several techniques from existing step distillation and on-policy distillation (SFT-augmented initialization, specialist teachers, source-based training data, and so on). It is unclear whether every component is necessary and how each one affects the final performance. More ablations are needed to separate the core contribution of this paper from related methods.

A: The revision states the origin of each component at the start of Sec. 4 and in Sec. 4.2. The distribution matching loss comes from TDM[1], the randomized grid is inspired by CDM[2], and the specialist teachers come from multi-teacher on-policy distillation. The core of this paper is that, within a single stage, source-routed specialists directly supervise the states of the deployed few-step student on its own rollout, and that a low-noise velocity regression motivated by a reverse KL decomposition is added. The table below adds the components one at a time. The three rows change only the listed component, and all other settings are the same.

ConfigurationGenEvalOCRDeQAPickScoreAverage
TDM + source routing0.92630.84884.02121.090.8476
+ randomized grid and interpolated re-noising0.94340.86564.17821.950.8722
+ low-noise velocity regression (StepX-OPD)0.94420.89864.17022.980.8901

After the randomized grid and interpolated re-noising are added, all four metrics rise, and Average rises from 0.8476 to 0.8722. After low-noise velocity regression is added, Average rises to 0.8901.

The other components also have corresponding ablations in the paper, summarized below.

ComponentTableMain result (Average)
Teacher selection (source routing / random assignment / uniform averaging)Tab. 30.8901 / 0.8410 / 0.8230
Velocity regression (off / step 0 / step 500)Tab. 60.8722 / 0.8895 / 0.8901
Target source and noise range of velocity regressionTab. 4Low-noise routed target 0.8901
Student initialization (Averaged-specialist merge / SFT-augmented merge)Tab. 90.8779 / 0.8901
Initialization from public adaptersTab. 5On the Flow-OPD adapter, TDM 0.8345 and StepX-OPD 0.8746

[1] Learning Few-Step Diffusion Models by Trajectory Distribution Matching. ICCV [2] Continuous-Time Distribution Matching for Few-Step Diffusion Distillation. arXiv

W2: The presentation needs improvement. Sec. 4 should first give the overall picture of the training recipe, and then explain one by one why each component is necessary and how it works. It now opens with the difference between the CDM and TDM schedules, which is confusing.

A: The revision has reorganized Sec. 4 accordingly. A new opening paragraph outlines one training step. The training source selects a specialist, the specialist supervises states built on the student's own few-step rollout, and it acts through both the TDM distribution matching loss and low-noise velocity regression. The three components are then introduced in turn. The subsection order is now 4.1 source routing, 4.2 supervision-point construction, and 4.3 low-noise velocity regression. Sec. 4.2 now opens by stating directly on which grid the student rolls out and why states between grid nodes need to be built. The earlier comparison of the CDM and TDM schedules has been removed, and CDM is cited only as the origin of the randomized grid. The CFG target is now given in the paragraph and is no longer listed as a separate equation.

W3: It is unclear whether the performance gain is significant. On several quality metrics (especially PickScore), StepX-OPD is worse than the best baseline on both architectures, and the Average gain on FLUX.1-dev mainly comes from DeQA. In addition, the text of App. E says that raising the OCR source weight improves both configurations, while Table 11 shows that both OCR and Average of the merged-initialization configuration drop clearly. The two contradict each other.

A: Average is an aggregate of the four metrics, and the revision reports the four metrics as separate measurement columns. The raw values of StepX-OPD and the two sequential baselines in Tab. 1 are as follows.

ArchitectureMethodGenEvalOCRDeQAPickScoreAverage
SD3.5-MediumFlow-OPD + TDM0.8750.7514.21722.600.8345
SD3.5-MediumDiffusionOPD + TDM0.8380.8333.95223.410.8403
SD3.5-MediumStepX-OPD0.9440.8994.17022.980.8901
FLUX.1-devFlow-OPD + TDM0.8640.9014.08723.300.8695
FLUX.1-devDiffusionOPD + TDM0.8820.9254.00523.320.8761
FLUX.1-devStepX-OPD0.8900.9294.23323.280.8902

On SD3.5-Medium, the GenEval and OCR of StepX-OPD are higher than those of the two sequential baselines. Its PickScore is lower than that of DiffusionOPD[3] + TDM, and its DeQA is lower than that of Flow-OPD[4] + TDM. On FLUX.1-dev, the GenEval, OCR, and DeQA of StepX-OPD are higher than those of the two sequential baselines, and its PickScore is lower than both.

On the Average gain on FLUX.1-dev, the gap in DeQA is the largest. StepX-OPD is 4.233, and the two sequential baselines are 4.087 and 4.005. At the same time, GenEval (0.890 versus 0.864 and 0.882) and OCR (0.929 versus 0.901 and 0.925) are also higher than those of the two sequential baselines.

On the contradiction in App. E, the reviewer's observation is correct. The text said that raising the OCR source weight improves both configurations, and the revision has corrected it so that it agrees with the table (Tab. 10 in the revision). In the merged-initialization configuration, the Average is 0.8901 at the source ratio 1:1:1 and 0.8427 at 1:2:1.

[3] DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models. SIGGRAPH Asia [4] Flow-OPD: On-Policy Distillation for Flow Matching Models. arXiv

Q1: Can the authors provide a comparison with routed TDM/CDM under exactly the same configuration (the same initialization, teachers, prompt sources, training budget, and so on)?

A: The cumulative ablation in W1 of this section already includes TDM + source routing under the same configuration. The table below adds CDM + source routing. The three rows have the same initialization, teachers, source ratio, and evaluation protocol.

ConfigurationGenEvalOCRDeQAPickScoreAverage
TDM + source routing0.92630.84884.02121.090.8476
CDM + source routing0.94630.82884.02122.490.8611
StepX-OPD0.94420.89864.17022.980.8901

The Average of routed TDM and routed CDM is 0.8476 and 0.8611, and that of StepX-OPD is 0.8901. All four metrics of routed TDM are lower than those of StepX-OPD. For routed CDM, only GenEval (0.9463) is slightly higher than that of StepX-OPD (0.9442), and the other three metrics are lower than those of StepX-OPD.

Q2: Can the authors give an ablation without velocity regression (the equation is not displayed in the original text, and from the context it is wv=0w_{\mathrm v}=0)?

A: Tab. 6 already gives this result, and the cumulative table in W1 of this section also contains this row. The three rows below change only velocity regression, and all other settings are the same.

Velocity regressionGenEvalOCRDeQAPickScoreAverage
Not used0.94340.86564.17821.950.8722
From step 00.94440.89504.17722.960.8895
From step 500 (StepX-OPD)0.94420.89864.17022.980.8901
Reviewer_yVeZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StepX-OPD combines multi-teacher on-policy distillation and four-step sampling compression in a single training stage. After a self-distillation warmup, it routes each source-specific batch to a frozen specialist teacher, which supervises states from the student’s own randomized few-step rollout through trajectory distribution matching; for low-noise batches, it adds guided velocity regression. The paper evaluates the method on SD3.5-Medium and FLUX.1-dev. It reports four-step Average scores of 0.8901 and 0.8902, respectively, higher than the reported four-step baselines. It also measures unintended prompt-word rendering and reports that capability integration can increase this side effect.

Critical strengths

The method addresses a meaningful bottleneck in the usual sequential pipeline. Integrating specialist capabilities and compressing the sampling trajectory in one student is a consequential goal: it could avoid waiting for a full-step multi-teacher model before beginning compression, while directly training the trajectory that will be used at inference. The randomized-grid and interpolated re-noising construction is a concrete way to align supervision with that trajectory.

Critical weaknesses
  1. 1.StepX-OPD’s reported configuration uses an SFT-augmented additive initialization, DiffusionOPD specialists, and low-noise velocity regression, whereas the sequential baselines are formed by applying TDM to existing OPD models. The paper includes useful initialization and teacher-family ablations, but they do not establish a fully matched comparison of joint versus sequential training with the same initialization, teachers, data mixture, and training budget. Consequently, it is difficult to determine how much of the improvement comes from the joint objective itself rather than these recipe differences. 2.The decomposition relies on assumptions including teacher consistency, and its exact identification with the teacher-matching term requires student self-consistency. The actual method instead uses unweighted, batch-gated velocity regression on low-noise samples, alongside an estimated distribution-matching objective. The paper acknowledges that the low-noise threshold is a design choice, but the theorem does not by itself establish that this particular truncated surrogate is a faithful or optimal approximation. This weakens the theoretical support for a central component of the method.
Questions for authors
  1. 1.Were the main StepX-OPD and sequential-baseline comparisons matched for student initialization, teacher family, training sources and source-sampling ratio, and total training compute? If not, how much of the reported gain remains in a controlled comparison that holds these factors fixed? 2.How do the assumptions used in the theorem—especially teacher consistency and the relationship between the student velocity and its induced distribution—apply to the classifier-free-guided specialist targets and the actual batch-gated, unweighted low-noise regression objective? This would affect how strongly the theoretical result supports the implemented method.
7 comments

We thank the reviewer for reading our manuscript carefully and for these constructive comments. We respond to each point below.

W1 & Q1: In the main results, StepX-OPD uses an SFT-augmented additive initialization, DiffusionOPD specialists, and low-noise velocity regression, while the sequential baselines apply TDM on top of existing OPD models. The initialization and teacher ablations in the paper are valuable, but they do not give a fully aligned comparison between joint training and sequential training under the same initialization, teachers, data ratio, and training budget. It is therefore hard to judge how much of the gain comes from the joint objective itself. The specific questions are as follows. Is the main comparison aligned in student initialization, teacher family, training sources and source ratio, and total training compute? If it is not aligned, how much gain remains after these factors are controlled?

A: The two sequential baselines also differ from each other in initialization and teachers. The Flow-OPD[1] student is first initialized by a cold start and then trained. The DiffusionOPD[2] student starts directly from the pretrained SD3.5-Medium. Their specialist teachers are also trained separately. We use the checkpoints released by both directly, and we distill them with TDM[3] using exactly the same data and training recipe. On this basis, we give the aligned results for initialization, teachers, training data, and compute one by one below.

For initialization, StepX-OPD in Tab. 5 uses the same initialization checkpoints as the sequential baselines, namely the public Flow-OPD and DiffusionOPD adapters. The TDM rows of the two are the same as the sequential baselines in Tab. 1. The teachers also match the initialization. StepX-OPD uses the Flow-OPD specialist teachers when it starts from the Flow-OPD adapter, and it uses the DiffusionOPD specialist teachers when it starts from the DiffusionOPD adapter.

InitializationMethodGenEvalOCRDeQAPickScoreAverage
Flow-OPD adapterTDM0.8750.7514.21722.600.8345
Flow-OPD adapterStepX-OPD0.9090.8974.10322.680.8746
DiffusionOPD adapterTDM0.8380.8333.95223.410.8403
DiffusionOPD adapterStepX-OPD0.9060.8934.05622.920.8727

With the same initialization checkpoint, the Average of StepX-OPD is 0.8746 and 0.8727, and the corresponding TDM rows are 0.8345 and 0.8403.

For teachers, the main recipe uses the DiffusionOPD specialist teachers. Tab. 9 in App. E adds two rows that use the Flow-OPD teachers, with all other settings unchanged.

InitializationTeachersGenEvalOCRDeQAPickScoreAverage
Averaged-specialist mergeFlow-OPD0.8850.8744.07322.610.8610
Averaged-specialist mergeDiffusionOPD0.9030.8964.16422.890.8779
SFT-augmented mergeFlow-OPD0.9320.8864.07722.750.8772
SFT-augmented mergeDiffusionOPD0.9440.8994.17022.980.8901

With the Flow-OPD teachers, the Average under the two initializations is 0.8610 and 0.8772. Both are higher than the 0.8345 and 0.8403 of the four-step sequential baselines in Tab. 1.

For training data, the data that the sequential baselines use for TDM distillation differs from the data of the main StepX-OPD experiment. The sequential baselines use JourneyDB[4], which is the training data that TDM itself uses. StepX-OPD uses the training data provided by Flow-OPD, that is, the training prompts of the three tasks GenEval, OCR, and PickScore, with the split of Flow-GRPO[5]. The reason is that source routing requires every prompt to have a clear capability source, so that the matching specialist teacher can be selected. JourneyDB prompts do not have this source, so when JourneyDB is used for training the teacher at each step can only be assigned arbitrarily. The table below gives the results of each method on the two kinds of training data.

MethodTraining dataGenEvalOCRDeQAPickScoreAverage
Flow-OPD + TDMJourneyDB0.8750.7514.21722.600.8345
Flow-OPD + TDMFlow-OPD training data0.8350.7424.13121.780.8103
DiffusionOPD + TDMJourneyDB0.8380.8333.95223.410.8403
DiffusionOPD + TDMFlow-OPD training data0.8020.8083.98622.160.8147
StepX-OPDJourneyDB0.8630.8284.03521.370.8300
StepX-OPDFlow-OPD training data0.9440.8994.17022.980.8901

The two sequential baselines have a higher Average on JourneyDB, and StepX-OPD has a higher Average on the Flow-OPD training data. Each method in Tab. 1 uses the training data that gives it the better result. When all methods use the Flow-OPD training data, the Average of StepX-OPD is 0.8901, and the two sequential baselines are 0.8103 and 0.8147.

For training compute, the GPU hours of each method are listed in the table below.

MethodCapability integration stage (GPU hours)Few-step distillation stage (GPU hours)Total (GPU hours)
Flow-OPD + TDM1600.025.91625.9
DiffusionOPD + TDM90.125.6115.7
StepX-OPD(single stage)(single stage)72.7

The values of the capability integration stage are converted from the training time that the Flow-OPD and DiffusionOPD papers each report, and they agree with Fig. 1. StepX-OPD is trained in a single stage for 72.7 GPU hours in total. DiffusionOPD + TDM and Flow-OPD + TDM take 115.7 and 1625.9 GPU hours.

[1] Flow-OPD: On-Policy Distillation for Flow Matching Models. arXiv [2] DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models. SIGGRAPH Asia [3] Learning Few-Step Diffusion Models by Trajectory Distribution Matching. ICCV [4] JourneyDB: A Benchmark for Generative Image Understanding. NeurIPS [5] Flow-GRPO: Training Flow Matching Models via Online RL. NeurIPS

W2 & Q2: The decomposition in the theorem relies on assumptions such as teacher consistency, and its strict correspondence with the teacher matching term also needs student self-consistency. The actual method uses an unweighted, batch-gated low-noise velocity regression, and uses it together with an estimated distribution matching objective. The paper admits that the low-noise threshold is a design choice, but the theorem itself does not show that this truncated surrogate objective is a faithful or optimal approximation. This weakens the theoretical support for the core component. The specific question is how the assumptions of the theorem, especially teacher consistency and the relation between the student velocity and its induced distribution, apply to the specialist targets with CFG and to the batch-gated, unweighted low-noise regression objective that is actually used.

A: We thank the reviewer for pointing this out. The assumptions of Theorem A.3 do not strictly hold under the specialist targets with CFG and under the batch-gated, unweighted implementation. We have therefore tightened the wording of this part. The revision no longer presents the theorem as a theoretical guarantee for low-noise velocity regression. It now states consistently that velocity regression is motivated by the decomposition in Theorem A.3. The corresponding sentences in the abstract, the introduction, Sec. 4.3, and the conclusion have all been changed.

The differences between the theorem and the implementation are collected in the new App. A.5. Theorem A.3 discusses the velocity fields induced by the two distributions, while training uses the velocities output by the networks. Matching the two needs teacher consistency and student self-consistency, and when student self-consistency does not hold the residual term RbR_b stays in the bound. The specialist targets carry CFG, and applying teacher consistency to them is an assumption. The velocity loss in the implementation does not carry the weight in the theorem, and the switch is decided by the mean sampled time of the batch. The threshold 0.5 and the weight wvw_{\mathrm v} are therefore empirical design choices. The actual effect of velocity regression is given by the ablation in Sec. 5.4.

Reviewer_SbDVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StepX-OPD combines multi-teacher capability transfer and four-step compression for text-to-image flow models. After self-distillation warmup, training batches are routed by their data source to frozen specialists. The method uses randomized rollout grids, interpolated Euler states followed by exact Gaussian re-noising, and a distribution-matching objective supplemented by batch-gated low-noise velocity regression. Appendix A derives a conditional reverse-KL decomposition motivating the auxiliary loss. Table 1 reports higher four-step aggregate scores than the evaluated sequential OPD-plus-TDM pipelines on SD3.5-Medium and FLUX.1-dev. The paper also measures unintended prompt-word rendering and studies an optional multi-step teacher target.

Critical strengths
  • The combined training recipe delivers useful capability retention in four steps. These results support the practical value of supervising the few-step trajectory directly.
  • It is interesting to explore how to skip the complicated stages: merge first, then distill. The work even shows improvement with a single stage.
Critical weaknesses
  • Missing baselines: 1) Conceptually, the TDM itself is an on-policy distillation at the marginal distribution level. Therefore, a direct baseline is to adopt TDM to distill multiple teachers. And such a baseline equals stepx-opd without velocity regression. 2) TDM-R1 can reinforce the few-step generator directly, therefre another direct baseline is to reinforce the few-step generator directly via multiple rewards.
  • Applying velocity regression would still, in theory, inevitably cause the few-step generator to learn the conditional mean, which in practice leads to blurry outputs. I remain concerned about this, and from the visualizations (Figs. 2 and 3), stepx-opd does appear noticeably blurrier than the other TDM variants.
Questions for authors

See Weaknesses. I am on the borderline side; if the authors could clarify the mentioned questions well, I am willing to raise my rating.

8 comments

We thank the reviewer for reading our manuscript carefully and for these constructive comments. We respond to each point below.

W1: A direct baseline is missing. TDM itself is on-policy distillation at the level of marginal distributions, so distilling multiple teachers with TDM is a direct baseline, and it is equivalent to StepX-OPD without velocity regression.

A: Distilling multiple teachers directly with TDM[1] is not equivalent to StepX-OPD without velocity regression. TDM uses a fixed anchor schedule. StepX-OPD resamples the rollout time points at every step, and it builds the supervised state by interpolating within a rollout segment and then re-noising. These two parts remain after velocity regression is removed. The table below adds the components one at a time. The three rows change only the listed component, and all other settings are the same.

ConfigurationGenEvalOCRDeQAPickScoreAverage
TDM + source routing0.92630.84884.02121.090.8476
+ randomized grid and interpolated re-noising0.94340.86564.17821.950.8722
+ low-noise velocity regression (StepX-OPD)0.94420.89864.17022.980.8901

The second and third rows are the two rows of Tab. 6 with velocity regression turned off and with velocity regression starting from step 500. After the randomized grid and interpolated re-noising are added, all four metrics rise, and Average rises from 0.8476 to 0.8722. After low-noise velocity regression is added, Average rises to 0.8901.

[1] Learning Few-Step Diffusion Models by Trajectory Distribution Matching. ICCV

W2: Another direct baseline is missing. TDM-R1 can apply reinforcement learning to a few-step generator directly, so reinforcing the few-step generator directly with multiple rewards is also a direct baseline.

A: TDM-R1[2] applies reinforcement learning to a few-step generator directly with non-differentiable rewards. In its experiments each model is trained with only one reward. Our paper needs one model to have several capabilities at the same time, so our setting differs from the single-reward setting of TDM-R1. We therefore follow the way multi-reward GRPO[3] handles multiple rewards, and we train an SD3.5-Medium four-step student with the GenEval, OCR, and PickScore rewards. The evaluation protocol is the same as in Tab. 1, and the results are below. The revision now cites TDM-R1 in the Step distillation paragraph of Sec. 2.

MethodTraining signalGenEvalOCRDeQAPickScoreAverage
Multi-reward few-step reinforcement learning (TDM-R1 style)GenEval, OCR, and PickScore rewards0.7410.7944.23021.940.8063
StepX-OPDSource-routed specialist teachers0.9440.8994.17022.980.8901

The Average of multi-reward few-step reinforcement learning is 0.8063, and that of StepX-OPD is 0.8901. The DeQA of the former is 4.230, which is higher than the 4.170 of StepX-OPD. Its GenEval, OCR, and PickScore are all lower than those of StepX-OPD.

This result is related to the starting point of reinforcement learning. Few-step reinforcement learning has to start from a distilled four-step model, and SD3.5-Medium itself becomes worse after it is distilled to four steps. As the table below shows (taken from Tab. 1), Average drops from 0.7105 to 0.6573.

ModelNFEGenEvalOCRDeQAPickScoreAverage
Base800.6660.5294.04521.790.7105
Base + TDM40.6530.3354.00221.870.6573

Prior work also reports that it is hard to continue training on a few-step model. ShortFT[4] points out that few-step models lose performance and capacity during distillation, and that fine-tuning on them works worse than fine-tuning on the base model. D-OPSD[5] reports a GenEval of 0.1588 after directly fine-tuning the few-step Z-Image-Turbo.

[2] TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward. ICML [3] Flow-OPD: On-Policy Distillation for Flow Matching Models. arXiv [4] ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuning. ICCV [5] D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models. arXiv

W3: In theory, velocity regression still makes the few-step generator learn the conditional mean, and in practice this shows up as blurry images. From Fig. 2 and 3, StepX-OPD is clearly blurrier than the other TDM variants, so this concern remains.

A: Whether the conditional mean is blurry depends on the noise level. A high-noise state corresponds to many possible images, and their mean is blurry. In a low-noise state the image content is mostly determined, and the conditional mean is close to a single image. So the velocity regression of StepX-OPD is turned on only on low-noise batches, and its target is the guided velocity of the teacher on the same state, vcfgtgt(zτ,τ,c)v_{\mathrm{cfg}}^{\mathrm{tgt}}(z_\tau,\tau,c). The regression therefore does not pull the student toward a blurry mean. At the same time, the distribution matching loss is the main objective, and velocity regression is an auxiliary term with weight wv=0.5w_{\mathrm v}=0.5. In Tab. 6, the DeQA with and without velocity regression is 4.170 and 4.178.

Results on multi-step models also show that regression to the teacher velocity can keep image quality. DiffusionOPD[4] minimizes the squared L2L_2 distance between the one-step transition means of the student and the teacher under deterministic sampling. In the loss of Flow-OPD[3], the DeQA regularization term that keeps image quality is also a regression to the velocity of the DeQA teacher. In Tab. 1 their DeQA is 4.113 and 4.268, which is higher than the 4.045 of the base model.

To quantify sharpness, we use two common measures, the variance of the Laplacian[5] and the high-frequency energy ratio in the frequency domain[6]. The blurrier an image is, the fewer high-frequency components it has, and the lower both measures are. The table below gives the results of the SD3.5-Medium four-step models. Each model uses the first 200 prompts of the PickScore test set, with 4 random seeds per prompt.

ModelSharpness (Laplacian variance)High-frequency energy ratio
Flow-OPD + TDM1461.30.0246
DiffusionOPD + TDM2117.40.0330
StepX-OPD without velocity regression2438.90.0313
StepX-OPD2578.90.0342

Both measures of StepX-OPD are the highest among the four models, and both are higher with velocity regression than without it.

[3] Flow-OPD: On-Policy Distillation for Flow Matching Models. arXiv

[4] DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models. SIGGRAPH Asia

[5] Diatom Autofocusing in Brightfield Microscopy: A Comparative Study. ICPR

[6] Sharpness Measure: Towards Automatic Image Enhancement. ICIP

Thank you for the authors' response and the new experiments. However, I still have a few questions:

  • Is the GPU cost of the multi-reward RL based on TDM-R1 aligned with OPD + distillation, or with RL + OPD + distillation?

  • Is the proposed velocity regression filtered according to noise level at the sample level, or at the batch level?

  • (minor) If I understand correctly, as far as t I know, randomized grid was adopted in DIDI-instruct [1], and interpolated re-noising was used as a regularization during training in the public repo of TDM; the authors need to discuss these works reasonably.

[1] Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct

Comments on the paper

1 comment
Area_Chair_2tFVOpenReview ↗

Discussion Window Coming to Close

Dear Reviewers,

The authors have made some effort to address your concerns and questions. As the discussion phase is coming to a close, I encourage you to keep engaging with them until the discussion reaches a clear outcome. If any of your questions or concerns remain unaddressed, now is the time to raise them.

Many thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.