DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose $\mathtt{DreOPD}$, a $\mathtt{D}$egraded-$\mathtt{r}$eference $\mathtt{e}$xtrapolative $\mathtt{OPD}$ method for flow-matching models that bridges these two paradigms. Our $\mathtt{DreOPD}$ converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that $\mathtt{DreOPD}$ outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.