- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DreOPD, a post-training method for Flow Matching models. The authors draw on the idea of reward extrapolation from G-OPD on autoregressive language models, enabling a student model not only to consolidate the capabilities of multiple specialized teachers but also to surpass the teacher models on some metrics. However, since flow models predict continuous velocity fields, trajectory-level reward extrapolation requires teacher-reference likelihood ratio estimation and credit assignment across denoising steps. Under the assumption of shared-covariance Gaussian transitions, the authors derive a closed-form velocity target. To strengthen this extrapolation signal, they further propose using a mildly degraded reference to enlarge the contrast between the teacher and the reference while preserving the generative structure as much as possible. In experiments, the closed-form velocity target achieves favorable gains in both single-teacher and multi-teacher distillation.
Critical strengths
- The problem is clearly proposed, and the method design is concise. Motivated by the core tension between RL and OPD, the authors optimize and implement a reward extrapolation framework on flow matching models. This framework retains the dense and stable supervision of OPD while achieving the ability to surpass teacher models.
- The theoretical derivation is relatively complete. The main text and appendix provide proofs ranging from trajectory-objective decomposition and conditional Gaussian objectives to the closed-form velocity solution and reward interpretation. In particular, the teacher is interpreted as a reward-tilted reference, and the expected reward is proven to be monotonically increasing with lambda, providing relatively clear theoretical support.
- The experimental coverage is relatively comprehensive. The authors conduct single-/multi-teacher, multi-task RL, multiple OPD baselines, and related ablation experiments. The results show that the method outperforms teacher models and baselines on most metrics and achieves good overall performance.
Critical weaknesses
- The theoretical assumptions are strong. The closed-form velocity target relies on assumptions such as shared-covariance Gaussian transitions and "linear" transition means. In practical scenarios, the predictive variances of the teacher, degraded reference, and student model may be inconsistent, and the dependence of the velocity field on the state may also cause the true mean to be nonlinear (resulting in the imprecise derivation of Eq. 28).
- The theory assumes that the teacher model is the KL-regularized reward optimum (i.e., a reward-tilted reference), but the actual situation may not strictly satisfy this assumption. Multi-task conflicts are not fundamentally resolved, and the method may still produce trade-offs among different tasks. In single-teacher distillation, some off-task metrics decrease; in the multi-teacher setting, ClipScore does not surpass the teacher, and the improvements are more reflected in most metrics or the average metric. Moreover, the degraded reference and lambda are also sensitive to tuning and may require relatively fine-grained selection to obtain good results.
Questions for authors
- The derivation of the closed-form target is based on SDE Gaussian transitions, whereas actual training is conducted under a deterministic ODE, so the correspondence between the two scenarios is not strict. The authors need to discuss what the error of this assumption is when deterministic ODE and finite-step sampling are actually used. Can an error bound or approximate guarantee under finite-step ODE be provided?
- Does lambda need to be task-adaptive or scheduled during training? Experiments show that lambda = 1.25 gives the best overall balance, while lambda = 1.5 improves OCR but harms GenEval and perceptual quality. Can different lambdas be set for different tasks? Can lambda be dynamically adjusted during training, and can over-extrapolation be detected using validation-set reward variance or distribution shift?
- Does multi-task extrapolation truly alleviate interference among tasks? The paper claims that a multi-task OPD student will interpolate, whereas DreOPD can extrapolate. Can the similarity between gradients of different tasks (or other evaluation methods) be measured to demonstrate that DreOPD produces fewer conflicts than standard OPD, or does it merely amplify certain task directions (e.g., as shown in the experimental results of Table 2, DreOPD’s ClipScore drops by 0.14%)?
3 comments
Response to Reviewer F4K6 (2/3)
W2. Teacher optimality:
The theory assumes that the teacher model is the KL-regularized reward optimum (i.e., a reward-tilted reference), but the actual situation may not strictly satisfy this assumption.
We agree that practical RL teachers may not attain the exact KL-regularized optimum. The reward-tilt assumption provides a sufficient condition for the reward interpretation and monotonicity of the ideal distributional optimum. The conditional target derivation does not require teacher optimality and still yields . Thus, the geometric extrapolation direction remains , although its alignment with the task reward is not guaranteed without the assumption. We have clarified the scope of this assumption and the resulting guarantees in Sec. 4.2 (Line 274-277).
W2&Q2. Reference, lambda and over-extrapolation:
Moreover, the degraded reference and lambda are also sensitive to tuning and may require relatively fine-grained selection to obtain good results.
Does lambda need to be task-adaptive or scheduled during training? Experiments show that lambda = 1.25 gives the best overall balance, while lambda = 1.5 improves OCR but harms GenEval and perceptual quality. Can different lambdas be set for different tasks? Can lambda be dynamically adjusted during training, and can over-extrapolation be detected using validation-set reward variance or distribution shift?
We agree that reference construction and can affect performance and the balance across tasks. Our ablations provide practical guidance for selecting them.
Reference degradation. Table 3 shows higher normalized average student scores with several degraded references, covering velocity quantization, weight quantization, and Gaussian perturbation.
| Reference construction | Normalized Avg.↑ |
|---|---|
| Original reference | 0.558 |
| 8-bit velocity quantization | 0.926 |
| 8-bit weight quantization | 0.739 |
| Gaussian velocity perturbation () | 0.594 |
Extrapolation strength. Both and improve all three metrics below (i.e., GenEval, OCR and PickScore) over teacher matching (). Increasing from 1.25 to 1.5 further improves OCR while reducing GenEval and PickScore. Thus, can be selected according to the desired task trade-off using validation results. We use 1.25 for its better overall balance.
Task-specific lambda. Yes, each task can use its own . We are testing for GenEval and aesthetics and for OCR. These experiment results show that the task-specific settings require further tuning.
| GenEval↑ | OCR↑ | PickScore↑ | |
|---|---|---|---|
| 1.00 | 0.958 | 0.926 | 23.988 |
| 1.25 | 0.968 | 0.936 | 24.034 |
| 1.50 | 0.961 | 0.938 | 24.019 |
| Task-specific | 0.963 | 0.940 | 23.537 |
Dynamic lambda. Thank you for this insight. Yes, can also change across training updates. We leave the design of an effective adjustment schedule for future work.
Over-extrapolation. We evaluate validation prompts every 60 steps and have reported Aesthetic and ImageReward (never used in optimization) to monitor the reward hacking or over-extrapolation. We can also inspect validation images for quality degradation that may indicate over-extrapolation. We totally agree with you that reward variance and distribution shift could provide additional warning signals.
Response to Reviewer F4K6 (3/3)
W2&Q3. Task interference:
Does multi-task extrapolation truly alleviate interference among tasks? The paper claims that a multi-task OPD student will interpolate, whereas DreOPD can extrapolate.
Thank you for raising this concern. DreOPD retains alternating task updates. It separates reward specialization in the teachers from capability consolidation in the shared student. Task interference can still occur during student training, and DreOPD does not explicitly resolve gradient conflicts. Our contribution lies in the extrapolated velocity target and reference construction for improved consolidation.
We agree that our original wording could imply stronger conflict mitigation than intended. The interpolation/extrapolation distinction refers specifically to the pointwise distillation targets. Standard DiffusionOPD uses the teacher velocity as its target. With , DreOPD moves this target beyond the teacher along the teacher–reference direction.
Can the similarity between gradients of different tasks (or other evaluation methods) be measured to demonstrate that DreOPD produces fewer conflicts than standard OPD, or does it merely amplify certain task directions (e.g., as shown in the experimental results of Table 2, DreOPD’s ClipScore drops by 0.14%)?
Following your suggestion, we compared task-gradient cosine similarity and gradient norms at the same training step. DreOPD has a less negative mean pairwise cosine than DiffusionOPD (−0.0875 vs. −0.1273) and larger gradient norms for all three tasks. These results show modestly better average directional alignment alongside increased gradient magnitudes, but do not establish fewer gradient-conflict occurrences.
| Method | Mean pairwise cosine ↑ | GenEval norm () | OCR norm () | Aes norm () |
|---|---|---|---|---|
| DiffusionOPD | −0.1273 | 1.100 | 0.791 | 1.233 |
| DreOPD | −0.0875 | 1.229 | 1.305 | 1.465 |
Task trade-offs. We acknowledge that trade-offs remain during multi-teacher consolidation. In Table 2, the 0.14% ClipScore decrease is relative to the aesthetics teacher (0.2963 to 0.2959). DreOPD improves GenEval, OCR, PickScore, and HPSv2.1 over DiffusionOPD, while ClipScore decreases slightly. Improved overall consolidation therefore does not imply gains in every metric.
We have clarified the manuscript to distinguish improved capability consolidation from explicit mitigation of gradient conflicts (Line 430-431).
We hope our explanation could address your concerns and increase your confidence in our work, and we are willing to offer more clarification if needed.
Response to Reviewer F4K6 (1/3)
We sincerely appreciate your insightful review and support of our work.
W1&Q1. Strong theoretical assumptions:
Thank you for raising this concern. Our original wording may have made the covariance and linearity assumptions unclear. We would like to clarify what the covariance and linearity assumptions refer to.
Shared covariance. The transition covariance is determined by the noise injected by the sampler. At the same state and timestep, the teacher, reference, and student use the same noise schedule and noise level, so their transition covariances are identical by construction. Their velocity predictions and therefore their transition means can still differ. We have made this distinction explicit in the manuscript (Line 199-202).
Dependence on the state. The required property is that, at a fixed visited state, the transition mean is affine in the velocity prediction. For the Euler update, μa(x,t)=x+Δtva(x,t), so μS−μT=Δt(vS−vT). This relation holds even when the velocity networks are highly nonlinear in x, because all networks are evaluated at the same student-visited state. Thus, state dependence does not invalidate this step of the derivation.
We have clarified these assumptions in the manuscript to avoid ambiguity (Line 199-202, Line 820-821).
Q1. SDE and ODE gap:
Thank you for highlighting the SDE–ODE connection. We would like to clarify that Sec. 5.4 and Fig. 4 already include both SDE and ODE training. We use ODE in the main experiments because it performs best in our ablation, consistent with DiffusionOPD.
At a fixed visited state, the Euler updates satisfy μθ−μλ⋆=Δtj(vθ−vλ⋆), where vλ⋆=vT+(λ−1)(vT−vref). Consequently, 21∥μθ−μλ⋆∥2=2(Δtj)2∥vθ−vλ⋆∥2 holds exactly for finite step sizes, giving the same pointwise minimizer in velocity space as the conditional SDE objective. To avoid ambiguity, we have explicitly described the ODE loss as a transition-matching surrogate in Sec. 4.1 (see Line 243-245) and Appendix B.2 (see Line 864-875).
Our current analysis does not establish a quantitative bound on the terminal distribution or expected reward after finite ODE sampling. Such a bound would require additional assumptions on the smoothness of the velocity fields, control of regression errors along the trajectory, and a link between the ODE sampler and the ideal distributional optimum. These are not established by our conditional target derivation.
The table below shows that DreOPD improves over DiffusionOPD under both ODE and matched SDE settings, providing empirical support for the practical surrogate.