acceptodds
Under review as a conference paper at ICLR 2027

DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

18 views

Abstract

Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose , a egraded-eference xtrapolative method for flow-matching models that bridges these two paradigms. Our converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.

Primary area:
generative models
Keywords:
On-policy distillation, flow matching, post-training, reward extrapolation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F4K6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DreOPD, a post-training method for Flow Matching models. The authors draw on the idea of reward extrapolation from G-OPD on autoregressive language models, enabling a student model not only to consolidate the capabilities of multiple specialized teachers but also to surpass the teacher models on some metrics. However, since flow models predict continuous velocity fields, trajectory-level reward extrapolation requires teacher-reference likelihood ratio estimation and credit assignment across denoising steps. Under the assumption of shared-covariance Gaussian transitions, the authors derive a closed-form velocity target. To strengthen this extrapolation signal, they further propose using a mildly degraded reference to enlarge the contrast between the teacher and the reference while preserving the generative structure as much as possible. In experiments, the closed-form velocity target achieves favorable gains in both single-teacher and multi-teacher distillation.

Critical strengths
  1. The problem is clearly proposed, and the method design is concise. Motivated by the core tension between RL and OPD, the authors optimize and implement a reward extrapolation framework on flow matching models. This framework retains the dense and stable supervision of OPD while achieving the ability to surpass teacher models.
  2. The theoretical derivation is relatively complete. The main text and appendix provide proofs ranging from trajectory-objective decomposition and conditional Gaussian objectives to the closed-form velocity solution and reward interpretation. In particular, the teacher is interpreted as a reward-tilted reference, and the expected reward is proven to be monotonically increasing with lambda, providing relatively clear theoretical support.
  3. The experimental coverage is relatively comprehensive. The authors conduct single-/multi-teacher, multi-task RL, multiple OPD baselines, and related ablation experiments. The results show that the method outperforms teacher models and baselines on most metrics and achieves good overall performance.
Critical weaknesses
  1. The theoretical assumptions are strong. The closed-form velocity target relies on assumptions such as shared-covariance Gaussian transitions and "linear" transition means. In practical scenarios, the predictive variances of the teacher, degraded reference, and student model may be inconsistent, and the dependence of the velocity field on the state may also cause the true mean to be nonlinear (resulting in the imprecise derivation of Eq. 28).
  2. The theory assumes that the teacher model is the KL-regularized reward optimum (i.e., a reward-tilted reference), but the actual situation may not strictly satisfy this assumption. Multi-task conflicts are not fundamentally resolved, and the method may still produce trade-offs among different tasks. In single-teacher distillation, some off-task metrics decrease; in the multi-teacher setting, ClipScore does not surpass the teacher, and the improvements are more reflected in most metrics or the average metric. Moreover, the degraded reference and lambda are also sensitive to tuning and may require relatively fine-grained selection to obtain good results.
Questions for authors
  1. The derivation of the closed-form target is based on SDE Gaussian transitions, whereas actual training is conducted under a deterministic ODE, so the correspondence between the two scenarios is not strict. The authors need to discuss what the error of this assumption is when deterministic ODE and finite-step sampling are actually used. Can an error bound or approximate guarantee under finite-step ODE be provided?
  2. Does lambda need to be task-adaptive or scheduled during training? Experiments show that lambda = 1.25 gives the best overall balance, while lambda = 1.5 improves OCR but harms GenEval and perceptual quality. Can different lambdas be set for different tasks? Can lambda be dynamically adjusted during training, and can over-extrapolation be detected using validation-set reward variance or distribution shift?
  3. Does multi-task extrapolation truly alleviate interference among tasks? The paper claims that a multi-task OPD student will interpolate, whereas DreOPD can extrapolate. Can the similarity between gradients of different tasks (or other evaluation methods) be measured to demonstrate that DreOPD produces fewer conflicts than standard OPD, or does it merely amplify certain task directions (e.g., as shown in the experimental results of Table 2, DreOPD’s ClipScore drops by 0.14%)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer F4K6 (1/3)

We sincerely appreciate your insightful review and support of our work.

W1&Q1. Strong theoretical assumptions:

The theoretical assumptions are strong. The closed-form velocity target relies on assumptions such as shared-covariance Gaussian transitions and "linear" transition means. In practical scenarios, the predictive variances of the teacher, degraded reference, and student model may be inconsistent, and the dependence of the velocity field on the state may also cause the true mean to be nonlinear (resulting in the imprecise derivation of Eq. 28).

Thank you for raising this concern. Our original wording may have made the covariance and linearity assumptions unclear. We would like to clarify what the covariance and linearity assumptions refer to.

Shared covariance. The transition covariance is determined by the noise injected by the sampler. At the same state and timestep, the teacher, reference, and student use the same noise schedule and noise level, so their transition covariances are identical by construction. Their velocity predictions and therefore their transition means can still differ. We have made this distinction explicit in the manuscript (Line 199-202).

Dependence on the state. The required property is that, at a fixed visited state, the transition mean is affine in the velocity prediction. For the Euler update, μa(x,t)=x+Δtva(x,t)\mu_a(x,t)=x+\Delta t v_a(x,t), so μS−μT=Δt(vS−vT)\mu_S-\mu_T=\Delta t (v_S-v_T). This relation holds even when the velocity networks are highly nonlinear in xx, because all networks are evaluated at the same student-visited state. Thus, state dependence does not invalidate this step of the derivation.

We have clarified these assumptions in the manuscript to avoid ambiguity (Line 199-202, Line 820-821).

Q1. SDE and ODE gap:

The derivation of the closed-form target is based on SDE Gaussian transitions, whereas actual training is conducted under a deterministic ODE, so the correspondence between the two scenarios is not strict. The authors need to discuss what the error of this assumption is when deterministic ODE and finite-step sampling are actually used. Can an error bound or approximate guarantee under finite-step ODE be provided?

Thank you for highlighting the SDE–ODE connection. We would like to clarify that Sec. 5.4 and Fig. 4 already include both SDE and ODE training. We use ODE in the main experiments because it performs best in our ablation, consistent with DiffusionOPD.

At a fixed visited state, the Euler updates satisfy μθ−μλ⋆=Δtj(vθ−vλ⋆)\mu_\theta-\mu_\lambda^\star=\Delta t_j(v_\theta-v_\lambda^\star), where vλ⋆=vT+(λ−1)(vT−vref)v_\lambda^\star=v_T+(\lambda-1)(v_T-v_{\mathrm{ref}}). Consequently, 12∥μθ−μλ⋆∥2=(Δtj)22∥vθ−vλ⋆∥2\frac12\|\mu_\theta-\mu_\lambda^\star\|^2=\frac{(\Delta t_j)^2}{2}\|v_\theta-v_\lambda^\star\|^2 holds exactly for finite step sizes, giving the same pointwise minimizer in velocity space as the conditional SDE objective. To avoid ambiguity, we have explicitly described the ODE loss as a transition-matching surrogate in Sec. 4.1 (see Line 243-245) and Appendix B.2 (see Line 864-875).

Our current analysis does not establish a quantitative bound on the terminal distribution or expected reward after finite ODE sampling. Such a bound would require additional assumptions on the smoothness of the velocity fields, control of regression errors along the trajectory, and a link between the ODE sampler and the ideal distributional optimum. These are not established by our conditional target derivation.

The table below shows that DreOPD improves over DiffusionOPD under both ODE and matched SDE settings, providing empirical support for the practical surrogate.

MethodSamplerGenEval↑OCR↑PickScore↑
DiffusionOPDODE0.96070.924223.980
DreOPDODE0.96810.936224.035
DiffusionOPDSDE, $ \sigma = 0.3$0.94930.923923.994
DreOPDSDE, $ \sigma = 0.3$0.95820.930124.012
AuthorsOpenReview ↗

Response to Reviewer F4K6 (2/3)

W2. Teacher optimality:

The theory assumes that the teacher model is the KL-regularized reward optimum (i.e., a reward-tilted reference), but the actual situation may not strictly satisfy this assumption.

We agree that practical RL teachers may not attain the exact KL-regularized optimum. The reward-tilt assumption provides a sufficient condition for the reward interpretation and monotonicity of the ideal distributional optimum. The conditional target derivation does not require teacher optimality and still yields vλ⋆=vT+(λ−1)(vT−vref)v_\lambda^\star=v_T+(\lambda-1)(v_T-v_{\mathrm{ref}}). Thus, the geometric extrapolation direction remains vT−vrefv_T-v_{\mathrm{ref}}, although its alignment with the task reward is not guaranteed without the assumption. We have clarified the scope of this assumption and the resulting guarantees in Sec. 4.2 (Line 274-277).

W2&Q2. Reference, lambda and over-extrapolation:

Moreover, the degraded reference and lambda are also sensitive to tuning and may require relatively fine-grained selection to obtain good results.

Does lambda need to be task-adaptive or scheduled during training? Experiments show that lambda = 1.25 gives the best overall balance, while lambda = 1.5 improves OCR but harms GenEval and perceptual quality. Can different lambdas be set for different tasks? Can lambda be dynamically adjusted during training, and can over-extrapolation be detected using validation-set reward variance or distribution shift?

We agree that reference construction and λ\lambda can affect performance and the balance across tasks. Our ablations provide practical guidance for selecting them.

Reference degradation. Table 3 shows higher normalized average student scores with several degraded references, covering velocity quantization, weight quantization, and Gaussian perturbation.

Reference constructionNormalized Avg.↑
Original reference0.558
8-bit velocity quantization0.926
8-bit weight quantization0.739
Gaussian velocity perturbation (σ=0.1\sigma=0.1)0.594

Extrapolation strength. Both λ=1.25\lambda=1.25 and 1.51.5 improve all three metrics below (i.e., GenEval, OCR and PickScore) over teacher matching (λ=1\lambda=1). Increasing λ\lambda from 1.25 to 1.5 further improves OCR while reducing GenEval and PickScore. Thus, λ\lambda can be selected according to the desired task trade-off using validation results. We use 1.25 for its better overall balance.

Task-specific lambda. Yes, each task can use its own λ\lambda. We are testing λ=1.25\lambda=1.25 for GenEval and aesthetics and λ=1.5\lambda=1.5 for OCR. These experiment results show that the task-specific settings require further tuning.

λ\lambdaGenEval↑OCR↑PickScore↑
1.000.9580.92623.988
1.250.9680.93624.034
1.500.9610.93824.019
Task-specific0.9630.94023.537

Dynamic lambda. Thank you for this insight. Yes, λ\lambda can also change across training updates. We leave the design of an effective adjustment schedule for future work.

Over-extrapolation. We evaluate validation prompts every 60 steps and have reported Aesthetic and ImageReward (never used in optimization) to monitor the reward hacking or over-extrapolation. We can also inspect validation images for quality degradation that may indicate over-extrapolation. We totally agree with you that reward variance and distribution shift could provide additional warning signals.

AuthorsOpenReview ↗

Response to Reviewer F4K6 (3/3)

W2&Q3. Task interference:

Does multi-task extrapolation truly alleviate interference among tasks? The paper claims that a multi-task OPD student will interpolate, whereas DreOPD can extrapolate.

Thank you for raising this concern. DreOPD retains alternating task updates. It separates reward specialization in the teachers from capability consolidation in the shared student. Task interference can still occur during student training, and DreOPD does not explicitly resolve gradient conflicts. Our contribution lies in the extrapolated velocity target and reference construction for improved consolidation.

We agree that our original wording could imply stronger conflict mitigation than intended. The interpolation/extrapolation distinction refers specifically to the pointwise distillation targets. Standard DiffusionOPD uses the teacher velocity as its target. With λ>1\lambda>1, DreOPD moves this target beyond the teacher along the teacher–reference direction.

Can the similarity between gradients of different tasks (or other evaluation methods) be measured to demonstrate that DreOPD produces fewer conflicts than standard OPD, or does it merely amplify certain task directions (e.g., as shown in the experimental results of Table 2, DreOPD’s ClipScore drops by 0.14%)?

Following your suggestion, we compared task-gradient cosine similarity and gradient norms at the same training step. DreOPD has a less negative mean pairwise cosine than DiffusionOPD (−0.0875 vs. −0.1273) and larger gradient norms for all three tasks. These results show modestly better average directional alignment alongside increased gradient magnitudes, but do not establish fewer gradient-conflict occurrences.

MethodMean pairwise cosine ↑GenEval norm (×10−5\times 10^{-5})OCR norm (×10−5\times 10^{-5})Aes norm (×10−5\times 10^{-5})
DiffusionOPD−0.12731.1000.7911.233
DreOPD−0.08751.2291.3051.465

Task trade-offs. We acknowledge that trade-offs remain during multi-teacher consolidation. In Table 2, the 0.14% ClipScore decrease is relative to the aesthetics teacher (0.2963 to 0.2959). DreOPD improves GenEval, OCR, PickScore, and HPSv2.1 over DiffusionOPD, while ClipScore decreases slightly. Improved overall consolidation therefore does not imply gains in every metric.

We have clarified the manuscript to distinguish improved capability consolidation from explicit mitigation of gradient conflicts (Line 430-431).


We hope our explanation could address your concerns and increase your confidence in our work, and we are willing to offer more clarification if needed.

Reviewer_7mFSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DreOPD, which extrapolates the velocity difference between the teacher and the degraded reference as the implicit reward direction, deriving a closed-form velocity regression objective. This allows the student model to outperform the teacher model while preserving the stability of OPD.

Critical strengths

The paper is very well written, with a complete and coherent overall narrative. The authors also provide extensive detailed proofs and derivations, which substantially strengthen the technical presentation.

Critical weaknesses
  1. The method first requires training a teacher model to satisfy the experimental setup (RL). In practice, a more meaningful setting would be to use a stronger teacher to train a weaker student, e.g., using Wan2.1 14B to train Wan2.1 1.3B. In such a setting, does the teacher still satisfy the assumption in the proposition that it is the optimizer of a KL-regularized reward objective? If not, is the extrapolation direction still the one stated in the paper?

  2. SDE derivation versus ODE training. The theoretical derivation is carried out under an SDE formulation, while the actual training uses an ODE. The connection between the two is explained in a rather unconvincing manner and does not constitute a rigorous justification.

  3. Degraded reference is intuitive but empirically chosen.

  4. Reward hacking may be more severe than for the teacher. This should be analyzed.

  5. Computational cost is doubled relative to standard OPD (teacher and the degraded reference).The paper should report training time and memory usage.

  6. Multi-task training still relies on alternating updates. The multi-task setting still uses alternating task-wise training and does not explicitly address gradient conflicts among different tasks.

Questions for authors

Please see the Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7mFS (1/3)

We deeply appreciate your thorough and insightful review.

Q1. Teacher setting:

The method first requires training a teacher model to satisfy the experimental setup (RL).

Thank you for raising this practical concern. Our work (DreOPD) can reuse existing teachers without additional RL training. Our main experiments focus on capability consolidation from independently trained specialists.

Teacher preparation and training time. If teachers must be prepared from scratch, the three RL teachers require 46.92, 33.17, and 85.75 hours, followed by 13.85 hours of DreOPD distillation. Training the teachers concurrently on separate resources gives an elapsed time of 99.60 hours. Both Multi-Task GRPO-Guard and CascadeNFT are multi-task RL baselines trained from the same pretrained base model. To compare the full training pipelines, we report their complete RL training time and include both teacher preparation and distillation in DreOPD’s elapsed time.

MethodElapsed time (h)↓GenEval↑OCR↑PickScore↑
Multi-Task GRPO-Guard129.860.9000.92823.197
Cascade NFT148.490.9380.88323.803
DreOPD99.600.9680.93624.035

In practice, a more meaningful setting would be to use a stronger teacher to train a weaker student, e.g., using Wan2.1 14B to train Wan2.1 1.3B. In such a setting, does the teacher still satisfy the assumption in the proposition that it is the optimizer of a KL-regularized reward objective? If not, is the extrapolation direction still the one stated in the paper?

Cross-scale applicability. Equal teacher and student model sizes are not required. The target can be applied to a larger teacher and a smaller student when their velocity predictions and transition parameterizations are compatible.

Evaluating the larger Wan2.1-13B model was infeasible within our rebuttal time and compute budget, so we instead tested distillation from SD3.5-Large to SD3.5-Medium. When distilling SD3.5-Large into SD3.5-Medium under matched training settings, increasing λ\lambda from 1 (DiffusionOPD) to 1.25 improves OCR from 0.606 to 0.636, exceeding the teacher's score of 0.609. This supports the empirical usefulness of extrapolation in this setting.

SettingOCR↑
SD3.5-L (teacher)0.609
SD3.5-M (student)0.508
λ=1.0\lambda=1.0 0.606
λ=1.25\lambda=1.250.636

Teacher optimality and extrapolation. The teacher optimality assumption provides a sufficient condition for the reward interpretation. An existing teacher may not satisfy this assumption. The target derivation in Sec. 4.1 does not require teacher optimality and still yields vλ⋆=vT+(λ−1)(vT−vref)v_\lambda^\star=v_T+(\lambda-1)(v_T-v_{\mathrm{ref}}). Thus, the geometric extrapolation direction remains vT−vrefv_T-v_{\mathrm{ref}}, although its alignment with the task reward is not guaranteed without the assumption. We have made this distinction explicit in Sec. 4.2 (Line 274-277).

Q2. SDE and ODE gap:

The theoretical derivation is carried out under an SDE formulation, while the actual training uses an ODE.

Thank you very much for highlighting the SDE–ODE connection. We would like to clarify that Sec. 5.4 and Fig. 4 already include both SDE and ODE training. We use ODE results for its better performance in our ablation, consistent with DiffusionOPD.

The connection between the two is explained in a rather unconvincing manner and does not constitute a rigorous justification.

We agree that the theoretical connection needs clearer qualification. At a fixed visited state, the conditional SDE objective yields vλ⋆=vT+(λ−1)(vT−vref)v_\lambda^\star=v_T+(\lambda-1)(v_T-v_{\mathrm{ref}}). The ODE transition-matching surrogate has the same pointwise minimizer, since 12∥μθ−μλ⋆∥2=(Δtj)22∥vθ−vλ⋆∥2\frac12\|\mu_\theta-\mu_\lambda^\star\|^2=\frac{(\Delta t_j)^2}{2}\|v_\theta-v_\lambda^\star\|^2. We have explicitly described the ODE loss as a transition-matching surrogate in Sec. 4.1 (see Line 243-245) and Appendix B.2 (see Line 864-875).

The table below shows that DreOPD improves over DiffusionOPD under both ODE and matched SDE settings.

MethodSamplerGenEval↑OCR↑PickScore↑
DiffusionOPDODE0.96070.924223.980
DreOPDODE0.96810.936224.035
DiffusionOPDSDE, $ \sigma = 0.3$0.94930.923923.994
DreOPDSDE, $ \sigma = 0.3$0.95820.930124.012
AuthorsOpenReview ↗

Response to Reviewer 7mFS (2/3)

Q3. Reference construction:

Degraded reference is intuitive but empirically chosen.

Thank you for raising this point. We select references to increase the contrast with the teacher while preserving generative structure, as discussed in Line 311 to Line 316. Proposition 3 supports this principle under an idealized reward-tilt assumption.

Table 3 in the paper shows that degrading the reference through velocity quantization, 8-bit weight quantization, or Gaussian perturbation (σ=0.1\sigma=0.1) yields higher average student scores than using the original reference. The poorer results with 4-bit weight quantization further highlight the importance of preserving generative structure.

We provide the following practical insights and have included this discussion in Appendix D.4 (Line 1237-1241):

  • Industrial training: Saved intermediate checkpoints are often undertrained and perform below the final model, providing candidate references with essentially no additional training cost.
  • Open-source models: When only one checkpoint is available, quantization, as evaluated in our paper, offers a controlled way to construct a degraded reference without retraining.

We will explore finer control of reference degradation in future work.

Q4. Reward hacking:

Reward hacking may be more severe than for the teacher. This should be analyzed.

Thank you for raising this important concern. To assess whether gains extend beyond the optimized rewards, we already report Aesthetic and ImageReward in Tables 2, 3, and 4 of the main paper. Neither metric is used in optimization, and both improve from the base student to the teacher and further to DreOPD, as shown below.

ModelAesthetic↑ImageReward↑
Student5.3960.8324
Aes teacher6.2321.5071
DreOPD6.2921.5245

We also evaluate validation prompts every 60 training steps. The generated images and scores allow us to monitor discrepancies between reward scores and visual quality. We acknowledge the risk of reward hacking. If the teacher already exhibits reward hacking, we neither claim nor provide a theoretical guarantee that DreOPD reduces it. We have added a new discussion of this risk and practical monitoring in Appendix G (Discussion, Line 1263-1274).

Q5. Training cost:

Computational cost is doubled relative to standard OPD (teacher and the degraded reference). The paper should report training time and memory usage.

Thank you for asking us to quantify the training cost. Our measurements show that DreOPD does not double the training time. Under the same setup, the additional reference forward pass increases the time per training step by 2.9 s, from 27.3 s to 30.2 s (10.6%).

MethodTraining time (s/step)↓Total time (h)↓Peak GPU memory (GiB)↓Time to GenEval=0.95 (h)↓
DiffusionOPD27.313.1353.55.25
DreOPD30.213.8553.53.69

Note: total time contains training and validation time.

Over 900 training steps, training alone takes ~10% longer; including the unchanged validation time, the total wall-clock time increases by 5.5%. Moreover, DreOPD reaches GenEval=0.95 in 3.69 hours, compared with 5.25 hours for DiffusionOPD, demonstrating faster convergence. We have added these results in the appendix (Sec. E.2, Line 1285-1290).

AuthorsOpenReview ↗

Response to Reviewer 7mFS (3/3)

Q6. Task interference:

The multi-task setting still uses alternating task-wise training and does not explicitly address gradient conflicts among different tasks.

Thank you for raising this concern. DreOPD improves task-specific distillation targets within an alternating-update framework. It does not introduce a new gradient-conflict resolution algorithm. Thank you for this insight, we will explore integrating gradient-conflict mitigation into this framework in future work.

Following the Multi-teacher OPD (M-OPD) setup, reward specialization is performed independently for each teacher, while capability consolidation occurs in the shared student. This separates teacher-side reward optimization from student-side consolidation, but does not eliminate possible interference during student training. Our contribution lies in the extrapolated velocity target and reference construction for improved consolidation.

Under the same settings, DreOPD achieves better performance compared with DiffusionOPD (Table 2 in the paper). These results show that the proposed targets improve capability consolidation.

PCGrad[1] and GradNorm[2] in Multi-Task Learning (MTL) address a complementary optimization problem by modifying the task gradients. We will explore this combination in future work. Moreover, we have clarified the manuscript to distinguish improved capability consolidation from explicit mitigation of gradient conflicts (Line 430-431).

[1] Yu, Tianhe, et al. Gradient surgery for multi-task learning. NeurIPS 2020.

[2] Chen, Zhao, et al. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. ICML 2018.


We hope these additional experiments and clarifications address your concerns. If you find these responses convincing, we would be grateful if you would consider increasing your score.

Reviewer_7mFSOpenReview ↗

Official Comment by Reviewer 7mFS

Thank you for the response. I think these results strengthen the paper and raise the score to 3.

Reviewer_o3GZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses post-training of flow-matching text-to-image models for multiple conflicting objectives. The authors observe a tradeoff between multi-task RL, which can exceed existing models but suffers from high-variance trajectory-level gradients and cross-task interference, and on-policy distillation, which is stable but imitation-based and therefore caps a shared student at interpolation among its teachers. Their method, DreOPD, adapts implicit reward extrapolation from the autoregressive LM setting to flow matching. The central technical move is a derivation showing that, under shared-covariance Gaussian transitions, the trajectory-level extrapolation objective has a per-state closed-form optimum, v*_λ = v_T + (λ−1)(v_T − v_ref). Training therefore reduces to weighted L2 velocity regression on student rollouts, with λ=1 recovering ordinary teacher matching. Since the step past the teacher scales with v_T − v_ref, the authors deliberately degrade the reference (8-bit quantization of the base model) to widen that gap.

The contributions are the closed-form target itself, which removes likelihood-ratio estimation and credit assignment from the objective; a reward interpretation of the teacher–reference contrast, with a monotonicity result dJ/dλ = Var[r]/β ≥ 0 at the distribution level; and the degraded-reference construction with an accompanying amplification argument.

Validation is on SD3.5-M at 512×512 across GenEval, OCR, and aesthetic objectives, reusing DiffusionOPD's teachers so that comparison is like-for-like. The method beats its teachers on 4/5 metrics in the single-teacher setting and attains the best average in the multi-teacher setting, with ablations over λ, reference degradation type and strength, and rollout noise level.

Critical strengths

A clean controlled demonstration that reward extrapolation transfers to continuous flow models. The λ sweep ({0.5, 0.75, 1.0, 1.25}) holds everything else fixed, and at λ=1 the method reduces exactly to standard teacher matching within the same codebase and teachers, so the ordering (GenEval .921→.950→.958→.968; OCR .899→.918→.926→.936) isolates the extrapolation term itself. The underlying ExOPD effect was established for autoregressive LMs by G-OPD, and the closed-form per-step reduction for diffusion was given by DiffusionOPD; this paper's role is to compose them and show the composition holds in a continuous-velocity setting, where the transfer was not obvious. That is a worthwhile port, cleanly executed, and it is the main reason I do not consider the paper empty.

Critical weaknesses

The closed-form reduction that the method section presents as its central derivation — shared-covariance Gaussian transitions, trajectory-level objective collapsed to a per-step closed-form KL, Gaussian mean-matching converted to velocity regression, SDE and ODE unified under one principle, and the argument that this analytic gradient has lower variance than a PPO-style estimator — is all present in DiffusionOPD (2605.15055), which the paper uses as its primary baseline and whose teachers, data splits, and ODE rollout choice it adopts. The λ-extrapolation idea, the finding that λ>1 lets a student exceed its teacher, the multi-domain-expert consolidation setting, and even the observation that the choice of reference model affects reward-signal quality all come from G-OPD (2602.12125). What remains is the composition of the two plus one additional completing-of-the-square step, the observation that the net coefficient of ‖v_θ‖² is 1 independent of λ (one line of algebra, though correct and worth stating), and the degraded reference itself.

Questions for authors

Precisely which components of the Section 4.1 derivation are new relative to DiffusionOPD? DiffusionOPD derives a closed-form per-step KL objective under shared-covariance Gaussian transitions, converts it to mean-matching, unifies SDE and ODE regimes, and argues that the resulting analytic gradient has lower variance than a PPO-style estimator including the explicit score-function term. Each of these appears in your Section 4.1, Appendix B.2, and the noise-level ablation respectively. I take the genuinely new elements to be the λ-weighted negative term, the resulting second completion of the square, and the observation that the net coefficient of ‖v_θ‖² is 1 independent of λ. If I am missing a further technical obstacle that your derivation resolves, please identify it. My novelty assessment turns on this.

2 comments
AuthorsOpenReview ↗

Response to Reviewer o3GZ

We deeply appreciate your valuable feedback and recognition of our work.

W1&Q1. Relation to prior work:

Our work builds on DiffusionOPD’s flow-model OPD framework and G-OPD’s reward-extrapolation principle in autogressive language model. The key extension for the community is to make distribution-level extrapolation directly usable as dense velocity supervision in flow-model OPD.

DiffusionOPD regresses toward teacher velocities, we derive a close-form target toward extrapolation. While G-OPD formulates reward extrapolation for autoregressive LMs, we bridge the gap between distribution-level reward extrapolation and local velocity regression for flow models. Furthermore, we introduce controlled reference degradation to shape the extrapolation contrast.

Our starting point was to retain the dense regression supervision of flow-model OPD while enabling extrapolation beyond teacher-matching targets. This led us to derive the extrapolated velocity target and investigate how reference construction affects extrapolation. Our contributions span three sections.

  • In Section 4.1, we derive a closed-form extrapolated velocity target and turn trajectory-level reward extrapolation into an on-policy regression recipe for flow models.
  • In Section 4.2, we characterize when the teacher–reference contrast is reward-aligned and prove reward monotonicity for the ideal distributional optimum under the reward-tilt assumption. This provides conditional theoretical support for extrapolation.
  • In Section 4.3, we introduce controlled reference degradation and prove contrast amplification within an idealized reward-aligned family, motivating the practical reference construction.

Precisely which components of the Section 4.1 derivation are new relative to DiffusionOPD?

Your characterization of the new algebraic components in Sec. 4.1 is accurate. The trajectory factorization and shared-covariance Gaussian calculations in Eq. (3), (6)–(7) follow DiffusionOPD, while Eq. (4)–(5) instantiate G-OPD’s extrapolation objective over flow trajectories. We have made these attributions explicit in the revised version (Line 155-156, 194-195).

The additional derivation in Eq. (8)–(13) solves the teacher–reference extrapolation objective in velocity space. The negative reference term changes ordinary teacher matching, yet the net quadratic coefficient remains λ−(λ−1)=1\lambda-(\lambda-1)=1. Completing the square therefore gives a unique finite conditional optimum

vλ⋆=vT+(λ−1)(vT−vref).v_\lambda^\star=v_T+(\lambda-1)(v_T-v_{\mathrm{ref}}).

This result reveals how the reference controls the extrapolation displacement in Eq. (12) and yields the regression objective in Eq. (13). The derivation is simple, but makes reward extrapolation directly usable in flow-model OPD.


We hope our explanation could address your concerns and increase your confidence in our work, and we are willing to offer more clarification if needed.

Dear Reviewer,

Thank you for your time and effort in reviewing the submissions. We kindly ask you to check the authors' responses and engage in further discussion if necessary. Please let us know promptly whether your concerns have been addressed or if any issues remain unresolved.

Your timely feedback would be greatly appreciated. Thank you for your cooperation!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.