acceptodds
Withdrawn from ICLR 2027

Flow-OPD: On-Policy Distillation for Flow Matching Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

43 views

Abstract

Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a “seesaw effect" of competing metrics and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in the large language model community, we propose Flow-OPD, the first unified post-training framework that integrates on-policy distillation into Flow Matching models. Flow-OPD adopts a two-stage alignment strategy: it first cultivates domain-specialized teacher models via single-reward GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; it then establishes a robust initial policy through a Flow-based Cold-Start scheme and seamlessly consolidates heterogeneous expertise into a single student via a three-step orchestration of on-policy sampling, task-routing labeling, and dense trajectory-level supervision. We further introduce Manifold Anchor Regularization (MAR), which leverages a task-agnostic teacher to provide full-data supervision that anchors generation to a high-quality manifold, effectively mitigating the aesthetic degradation commonly observed in purely RL-driven alignment. Built upon Stable Diffusion 3.5 Medium, Flow-OPD raises the GenEval score from 63 to 93 and the OCR accuracy from 59 to 93, yielding an overall improvement of roughly 10 points over vanilla GRPO, while preserving image fidelity and human-preference alignment and exhibiting an emergent "teacher-surpassing" effect. These results establish Flow-OPD as a scalable alignment paradigm for building generalist text-to-image models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a8HDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Flow-OPD consolidates specialized flow-matching teachers into a single text-to-image student. The student is initialized through filtered teacher-trajectory SFT or LoRA merging, samples its own states, and receives velocity supervision from one teacher selected by prompt metadata. A separate aesthetic teacher supplies manifold-anchor regularization. Experiments evaluate composition, OCR, preference, and image quality against an interleaved GRPO-Mix baseline, with additional controls for direct distillation, initialization, and the aesthetic reference.

Critical strengths
  1. A practically relevant consolidation problem with promising reported results. Combining task-specialized teachers into one generator can retain useful capabilities without deploying several experts. The reported composition and OCR improvements over the evaluated GRPO-Mix recipe suggest that dense teacher supervision is a productive direction for this setting.

  2. Useful attribution controls for the complete system. The manuscript includes direct velocity distillation, MAR removal and reference replacement, GRPO-Mix with MAR, and merging controls. In particular, the aesthetic-anchor comparisons support a separate contribution to image quality, and the direct-distillation result helps show that much of the overall gain comes from dense supervision. These are consequential controls for assessing the system’s value.

Critical weaknesses
  1. The actual optimization objective and its incremental contribution are unclear. Section 5 describes differentiating a weighted velocity loss on detached student states and explicitly avoiding score-function estimation. Appendix B.1 instead discusses a detached transition-KL reward, and Table 5 labels Flow-OPD as a “PG-style update.” The distinction from direct velocity distillation is therefore not specified consistently. This is central because the reported independent aggregate gain over that control is 0.0155; without the exact two losses, it is difficult to identify what the proposed objective contributes beyond existing online velocity supervision.

  2. Inconsistent results and an underspecified merge prevent reliable reproduction of the key comparison. Table 2 gives merge-model scores of 0.93/0.93/4.31/23.05 with average 0.9021, whereas Section 6.2 gives 0.92/0.94/4.35/23.08 with average 0.9044. SFT+GRPO-Mix is also 0.8515 in Table 2 but 0.8165 in Table 5. These may represent different runs, but their identity and aggregation need reconciliation. Moreover, Eq. 17 averages three rank-32 weight updates, whose sum need not remain rank 32, without explaining the representation subsequently optimized. Both issues affect whether the claimed incremental gains and parameter comparison can be verified.

Questions for authors
  1. Please provide exact Flow-OPD and direct-distillation losses or pseudocode, including weighting, any reward/log-probability terms, and all stop-gradient operations. What precisely differs between the two controls?
  2. Which checkpoint, evaluation setup, and seed aggregation produce the canonical main results? Please reconcile the Table 2, Section 6.2, and Table 5 values.
  3. How is the average of the LoRA weight updates represented and subsequently trained? What is its resulting rank, and is compression or another approximation applied?
Reviewer_ybhVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Flow-OPD consolidates single-reward Flow-GRPO teachers (GenEval, OCR, PickScore, DeQA) into one SD-3.5-M student. After a cold start, the student rolls out its own SDE trajectories and regresses the velocity of the teacher its prompt is routed to. The loss uses a weight w(t) taken from the closed-form Gaussian KL, plus an extra term toward a frozen aesthetic teacher.

Critical strengths
  1. The derivation is correct and gives a practical objective. With a shared covariance the per-step KL is exactly a weighted velocity MSE, so no log-probability bookkeeping or clipping is needed, and public Flow-GRPO checkpoints work as teachers.
  2. The paper is candid about its limits. It admits that GRPO-Mix is interleaved rather than joint, that within-batch interference is untested, and that direct distillation explains most of the gain.
  3. The end product is useful. One student roughly matches each specialist on its own task It also beats GRPO-Mix on metrics the teachers were not trained on, such as HPS-v2.1, and plain LoRA averaging turns out to be a good cold start.
Critical weaknesses
  1. The genuinely new part is thin, and the paper’s own attribution says it adds little. Eq. (12) is the closed-form KL that Flow-GRPO [1] uses as its reference penalty, with the teacher swapped in. Dropping the gradient through the state distribution (Eq. 15) then makes the update DAgger [2]. Velocity matching along the student’s own trajectory already appears in π-Flow [3], and consolidating RL-trained single-task teachers is policy distillation [4].

  2. The RL comparison is too weak to show that distillation beats multi-reward RL. GRPO-Mix optimizes one reward at a time on a 3:1:1 epoch schedule with no DeQA partition, while Flow-OPD gets four task teachers plus an aesthetic one. The obvious missing baseline is within-batch multi-reward Flow-GRPO. DiffusionNFT [5] (ICLR 2026) trains one SD3.5-M jointly on GenEval, OCR, PickScore and other rewards and reports 0.94/0.91/23.80 without CFG. Protocols differ, but this undercuts the premise that joint RL must trade capabilities off..

  3. Much of the gain seems to come from the cold start, which the main table never isolates. In Table 3, 62–73% of the improvement over GRPO-Mix on Color, Shape, Complex and 3D-spatial is already there before distillation. Fig. 4 shows only 0.81 GenEval and 0.85 OCR without a cold start. PickScore is the clearest case. The SFT-initialized student stays at 21.83 (base 21.64), although, as far as I can tell, it is distilled from a teacher above 23; the merged one reaches 23.05. That looks inherited from the merged adapter. Table 2 needs SFT-only and Merge-only rows.

[1] Liu et al. Flow-GRPO: Training Flow Matching Models via Online RL. NeurIPS 2025.

[2] Ross, Gordon, Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS 2011.

[3] Chen et al. pi-Flow: Policy-Based Few-Step Generation via Imitation Distillation. ICLR 2026.

[4] Rusu et al. Policy Distillation. ICLR 2016.

[5] Zheng et al. DiffusionNFT: Online Diffusion Reinforcement with Forward Process. ICLR 2026. DOI: 10.48550/arXiv.2509.16117

Questions for authors

Are teacher targets computed with CFG, and is the student also sampled with CFG? That would apply guidance twice.

Reviewer_T6fvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Flow-OPD, a multi-teacher on-policy distillation framework for consolidating specialized text-to-image capabilities into a single flow-matching model. The method obtains task-specific teachers through reinforcement learning, initializes the student through supervised fine-tuning or LoRA merging, and provides task-routed velocity supervision on student-generated states. Experiments on Stable Diffusion 3.5 Medium report improvements in compositional generation and text rendering while maintaining competitive image quality.

Critical strengths

(1) This paper proposes Flow-OPD that combines specialized image-generation checkpoints into a single deployable model is valuable for industrial model development. The paper addresses the balance between compositional generation, text rendering, and visual quality. (2) The main table reports substantial gains over the base model and GRPO-Mix on GenEval and OCR, while maintaining competitive image-quality scores. These results suggest that task-routed teacher supervision is useful for consolidating specialized capabilities. (3) The paper includes controls for initialization, direct velocity distillation, model merging, and aesthetic regularization. In particular, the MAR ablations provide evidence that a separate aesthetic teacher can help preserve visual quality during capability transfer.

Critical weaknesses

(1) The implemented optimization procedure is unclear. Equation (15) describes directly differentiated, time-weighted velocity regression and explicitly avoids policy-gradient estimation and PPO clipping. However, Appendix B.1 and Table 5 describe detached transition-KL rewards and PG-style updates. The mathematical presentation introduces further ambiguity: Eqs. (6) and (11) use inconsistent transition-time conventions, and Eq. (15) does not clearly express the stated stop-gradient treatment of sampled states. These issues prevent a precise reconstruction of the method and its distinction from the direct-distillation control. (2) Direct on-policy velocity distillation achieves an aggregate score of 0.8889, compared with 0.9044 for the full method, explaining most of the improvement over GRPO-Mix. The remaining gain requires clearer attribution under matched configurations. Furthermore, GRPO-Mix alternates task partitions by epoch, and the paper does not directly measure gradient interference. The distinction from Flow-OPD could be clarified more precisely. Flow-OPD already employs task-based teacher routing and teacher velocity-field supervision on student-generated states. DanceOPD’s more specific contributions therefore appear to lie in its low-noise single-query strategy, simplified training recipe, and extension to editing and field absorption. These are useful developments, but the paper would benefit from a clearer separation between the shared framework and its new contributions. (3) Table 2 reports GenEval/OCR scores of 0.93/0.93 and an aggregate of 0.9021, whereas the accompanying text reports 0.92/0.94 and 0.9044. The SFT+GRPO-Mix aggregate also differs between Tables 2 and 5. These discrepancies make it unclear which results support the comparisons, while the reported seed means provide insufficient information about the uncertainty of the smaller gains. (4) All reported Flow-OPD experiments use Stable Diffusion 3.5 Medium and its specialized variants. Multiple benchmarks demonstrate task coverage but do not establish transfer across backbones or teacher collections. This limits the evidence for the proposed framework’s broader applicability.

Questions for authors

(1) What objective was actually implemented: direct velocity regression or policy-gradient optimization with detached KL rewards? Could the authors provide consistent pseudocode, time conventions, and explicit stop-gradient operations? (2) Could the authors double-check the conflicting numerical results, identify their checkpoints and seeds, and report variability for both the full method and the direct-distillation control? Could the authors clarify the methodological differences from Flow-OPD and isolate the effects of query location, query count, and optimization objective under matched backbones, teachers, initialization, and training budgets? (3) Could the authors validate the method on an additional flow-matching backbone with compatible teachers, including merge-only, direct on-policy distillation, and full Flow-OPD comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.