acceptodds
Under review as a conference paper at ICLR 2027

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Abstract

On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduce Self-OPD, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision. At each timestep, Self-OPD branches the deterministic next-state prediction into stochastic SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline to obtain normalized advantages. The velocity field is optimized with an all-branch pull-push objective, where high-advantage branches attract the student and low-advantage branches repel it under direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
flow matching, On-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PdxZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Self-OPD, a teacher-free OPD framework for flow matching models. At each timestep, it branches the next state into K SDE candidates, rolls each one out with the ODE sampler, and computes advantages against a deterministic self-reference baseline. The velocity field is trained with an all-branch pull-push objective, and multiple rewards are combined by reward-level fusion. Experiments use SD3.5-M on GenEval, OCR, PickScore and HPSv2. In the multi-reward setting, Self-OPD reports GenEval 0.952 and OCR 0.960, against Flow-OPD, DiffusionOPD and DiffusionNFT.

Critical strengths

It needs no task-specific teachers and works with black-box rewards, which makes it simple to use. A single model reports strong GenEval (0.952) and OCR (0.960) in Tab. 2.

Critical weaknesses
  1. Contributions have limited novelty.The self-reference baseline is similar to DenseGRPO's "step-wise dense reward". DenseGRPO is not cited or compared. Pulling toward the best sample and pushing away from the worst is also close to TAGRPO, and TAGRPO is not cited.
  2. DiffusionOPD's training times are taken from its paper, but its metrics are not; the results also look like reward hacking. Sec. 4.3 uses "the training times … reported in DiffusionOPD", but Tab. 2 reports GenEval 0.915 for DiffusionOPD versus 0.96 in its paper. Where do these numbers come from? With 0.96, Self-OPD (0.952) does not lead. OPD is meant to "distill their capabilities into a unified student", not to hack one reward. Fig. 3 claims "superior performance in both accuracy and aesthetics", but in "No Liquids Allowed" Self-OPD's people are clearly more distorted than those of the other methods.
Questions for authors
  1. Where do the DiffusionOPD numbers in Tab. 2 come from? If they were reproduced, please give the setup and the training time of this reproduction. And please report the training cost on the same hardware.
  2. Please provide evidence beyond the training rewards (e.g., human evaluation of artifacts) to rule out reward hacking.
5 comments
AuthorsOpenReview ↗

Response to Reviewer PdxZ

Thank you for your detailed review. We address your concerns about novelty, baseline provenance, cost comparisons, and visual artifacts below.

Contributions have limited novelty.The self-reference baseline is similar to DenseGRPO's "step-wise dense reward"... is also close to TAGRPO...

Self-OPD differs substantially from DenseGRPO and TAGRPO in the following respects:

First, the advantage reference differs. Self-OPD labels branches positive or negative relative to the deterministic ODE rollout reward from the same state, rather than a group mean. Standard-deviation normalization rescales these advantages without changing their signs. DenseGRPO normalizes adjacent-state reward gains within trajectory groups, while TAGRPO also uses group-relative advantages.

Second, the update objective differs. Self-OPD performs signed, step-wise regression toward or away from all K branch targets, normalized by the transition variance. DenseGRPO and TAGRPO instead use likelihood-ratio-based, clipped GRPO updates.

Third, the targets and repulsion mechanism differ. Self-OPD uses freshly sampled local branches from one parent state and attenuates repulsion when it would oppose the best branch's direction. TAGRPO uses group-best/group-worst trajectory anchors, cross-sample likelihood ratios, and a memory bank.

We will cite both papers. We could not obtain publicly released official DenseGRPO LoRA weights, so we could not reevaluate DenseGRPO under our evaluation protocol.

Sec. 4.3 uses "the training times … reported in DiffusionOPD", but Tab. 2 reports GenEval 0.915 for DiffusionOPD versus 0.96 in its paper... Where do the DiffusionOPD numbers in Tab. 2 come from...

Please note that our evaluation uses two GenEval scoring modes, detailed in Appendix D and Fig. 5. Strict scoring requires all prompt sub-requirements to be satisfied, while continuous scoring awards partial credit. Under the same evaluation protocol, DiffusionOPD scores 0.9150 (stri.) and 0.9479 (cont.). Self-OPD's 0.9521 in Table 2 is a strict score, not directly comparable to the 0.96 reported in DiffusionOPD's paper under continuous scoring.

The remaining gap between our continuous score of 0.9479 for DiffusionOPD and its reported 0.96 may reflect differences in scoring settings or evaluation parameters. For example, StepX-OPD (arXiv:2611.04386v1, Table 1 and Sec. 5.1) reports 0.945 for DiffusionOPD using SD3.5-Medium at 512×512, with 40 sampling steps and CFG 4.5, close to our 0.9479 under similar settings. All scores in Table 2 come from our own evaluations under a common protocol.

Fig. 3 claims "superior performance in both accuracy and aesthetics", but in "No Liquids Allowed" Self-OPD's people are clearly more distorted than those of the other methods.Please provide evidence beyond the training rewards (e.g., human evaluation of artifacts) to rule out reward hacking.

Table S1 and S2 provide independent evidence against reward hacking as the sole explanation for the observed gains. Table S1 provides evidence beyond the training rewards: Self-OPD leads on ImageReward, Improved Aesthetic Predictor, NIMA, and PaQ-2-PiQ, ranks second on CLIP-IQA, and maintains competitive feature diversity. These results support broader perceptual-quality gains rather than improvements confined to the optimized rewards. In Table S2, Self-OPD achieves the highest mean ratings among all compared methods for both task accuracy and aesthetics.

Table S1. Evaluation of Self-OPD and other methods on additional metrics using the GenEval/OCR test sets. Best scores are bold; second-best are underlined.

MetricSD3.5-MediumFlow-GRPOGRPO-GuardFlow-OPDDiffusionOPDDiffusionNFTSelf-OPD
ImageReward1.13701.19471.14111.37971.07841.4829‾\underline{1.4829}1.6094
Improved Aesthetic Predictor5.39585.18845.13355.46075.10535.7747‾\underline{5.7747}5.8055
NIMA5.47485.43975.34795.78705.34055.8169‾\underline{5.8169}5.9131
PaQ-2-PiQ75.303574.925374.554477.070774.766777.5123‾\underline{77.5123}77.9858
CLIP-IQA0.70450.69760.67290.73990.70710.72370.7262‾\underline{0.7262}
CLIP cosine distance0.41910.41570.41540.40700.40430.40370.4163‾\underline{0.4163}
DINOv2 cosine distance0.95300.9423‾\underline{0.9423}0.93920.93190.90800.91540.9396

Table S2. User study. Ten volunteers rated anonymized, randomly ordered images from four methods on 100 held-out prompts with one image per method per prompt. Task accuracy and aesthetics use 1–5 scales.

MethodTask accuracyAesthetics
Flow-OPD4.123.91
DiffusionOPD4.283.74
DiffusionNFT4.204.02
Self-OPD4.434.14
Reviewer_PdxZOpenReview ↗

Follow-up on Author Response: Unanswered Same-Hardware Cost, Human Distortion, and Novelty

Thank you for clarifying strict vs. continuous GenEval, which resolves my question on DiffusionOPD's scores. Three concerns remain:

  1. Unanswered hardware specs and unfair DiffusionOPD time comparison. Your reply truncated and skipped my original Question 1 request for same-hardware training cost. Across the paper and all threads, GPU model, GPU count, and batch size remain undisclosed, while DiffusionOPD's 97.01h (max(46.92h GenEval, 33.17h OCR, 85.75h aesthetics) + 11.26h) is copied from its paper. Moreover, comparing this 97.01h against Self-OPD's 48h warm-start (max(27h GenEval, 37h OCR) + 11h) is unfair because the pretraining objectives do not match: DiffusionOPD's 97.01h is bottlenecked by its 85.75h aesthetics teacher, whereas Self-OPD's warm-start omits aesthetics specialist pretraining entirely (excluding the aesthetics teacher, DiffusionOPD requires max(46.92h, 33.17h) + 11.26h = 58.18h, shorter than Self-OPD's 90h from-scratch).

    • Requested: Please report the exact GPU model, GPU count, and batch size, along with an objective-matched DiffusionOPD training cost measured on the same hardware.
  2. Human/limb distortion. Your added automatic metrics and user study only measure global aesthetics and task accuracy, missing the severe human distortion I flagged in "No Liquids Allowed."

    • Requested: Direct evaluation (e.g., human study on existing outputs) of human body / limb distortion on person-containing prompts across methods.
  3. Limited novelty and practicality. Self-OPD is simply an online RL method that ports step-wise branching and pull-push ideas (as in DenseGRPO and TAGRPO) from denoising-MDP RL (DDPO / Flow-GRPO / DanceGRPO) to forward based RL (like DiffusionNFT). Framing it as "teacher-free OPD" is also unpractical: once the teacher is removed and training relies on online reward models, it loses the two key strengths of OPD, namely fast merging (~11h) and merging non-RL specialists (such as SFT models without reward models).

I maintain my rating pending the objective-matched same-hardware DiffusionOPD cost and human distortion evaluation.

AuthorsOpenReview ↗

Response to Follow-up: Hardware Specs & Cost Analysis, Human Distortion Study, and Method Positioning(1/2)

Thank you for following up and confirming that our clarification on strict vs. continuous GenEval resolves the score-provenance question. Below, we address your remaining concerns directly.

Q1. DiffusionOPD's 97.01h is bottlenecked by its 85.75h aesthetics teacher, whereas Self-OPD's warm-start omits aesthetics specialist pretraining entirely (excluding the aesthetics teacher, DiffusionOPD requires max(46.92h, 33.17h) + 11.26h = 58.18h, shorter than Self-OPD's 90h from-scratch).

A1: First, warm-start checkpoints in Self-OPD serve as student initializers, not teachers, which changes how much pretraining they need. In DiffusionOPD, teachers provide the fixed supervision targets for distillation, so they must be fully trained to convergence. In Self-OPD, warm-start LoRAs only give the student a rough starting point. Our actual OCR and GenEval initializers only score around 0.87 and 0.88, well below DiffusionOPD's teachers. After initialization, the student improves on its own through online rewards and never queries those LoRAs again. If DiffusionOPD used such under-trained checkpoints as teachers, its distillation targets would directly degrade.

Second, Self-OPD drops the external aesthetics teacher, not the aesthetics objective. During joint training, every output in Self-OPD is scored at the same time by its task reward (OCR or GenEval) and two preference rewards (PickScore and HPSv2). By contrast, if we drop the aesthetics teacher from DiffusionOPD to get the 58.18 h figure, DiffusionOPD loses aesthetic supervision completely and only distills OCR and GenEval. Thus, 58.18 h corresponds to a two-objective setup rather than an objective-matched comparison to Self-OPD's full training.

Q2. Unanswered hardware specs and unfair DiffusionOPD time comparison. Please report the exact GPU model, GPU count, and batch size, along with an objective-matched DiffusionOPD training cost measured on the same hardware.

A2: Self-OPD is trained on 8 NVIDIA A800 GPUs with a per-GPU microbatch size of 2. We accumulate one OCR and one GenEval microbatch per step, resulting in an effective global batch size of 32.

For DiffusionOPD, we cited the wall-clock times reported in their paper. Comparing their student-stage GPU-hours (Fig. 3) with wall-clock hours (Table 1) shows they also used 8 GPUs, though their paper does not disclose the exact GPU model or batch size. Because time constraints prevent us from re-running their full teacher-plus-student pipeline during the rebuttal window, we will run and report an objective-matched benchmark on the exact same 8 × A800 machine in the final manuscript.

More importantly, our main goal with Self-OPD is not just to train faster than teacher-based OPD, but to optimize task accuracy and aesthetics on the exact same output image. As noted in Sec. 4.2 of DiffusionOPD, single-task teachers generalize poorly outside their own target domains. Because DiffusionOPD's round-robin recipe (Sec. 3.4) pairs GenEval prompts only with the GenEval teacher and OCR prompts only with the OCR teacher, nothing explicitly forces an OCR or GenEval image to also have high aesthetic quality. Self-OPD addresses this through reward-level fusion, where every GenEval or OCR sample is jointly guided by its task reward plus PickScore and HPSv2. This is why Table 2 reports “Pref. (same test images)”, which measures aesthetic preference directly on the GenEval and OCR test outputs rather than on a separate prompt set.

Q3. Human/limb distortion. Please provide direct evaluation of human body / limb distortion on person-containing prompts across methods.

A3: To directly evaluate this, we added a human-body/limb distortion metric to our 10-volunteer user study (Table S2). Using the same blind, randomized setup across Flow-OPD, DiffusionOPD, DiffusionNFT, and Self-OPD, evaluators rated all person-containing prompts from the 100 held-out test prompts.

Distortion is rated on a 1 to 5 scale (1 means no visible distortion and 5 means severe distortion, so lower is better), checking for extra, missing, or fused limbs, broken joints, and deformed body proportions. Prompts where required people failed to appear were penalized under task accuracy rather than counted as distortion-free. As shown in Table S2, all methods show similarly low average distortion with very small gaps. Self-OPD scores 1.23, which is practically tied with DiffusionOPD at 1.22. While Self-OPD does not systematically worsen human anatomy, occasional background artifacts (such as in Fig. 3's "No Liquids Allowed") can still happen at 512×512 resolution when most spatial capacity goes to rendering large foreground text.

Reviewer_czWvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Teacher-based OPD for flow matching models needs a separately trained teacher for each objective, and multi-teacher variants do not require one image to satisfy all objectives at once. The paper proposes Self-OPD, which branches each on-policy step into KK SDE candidates, rolls each out with the ODE sampler, scores them against a deterministic ODE self-reference to get advantages AkA_k, and trains the velocity field with a pull-push loss (gated by a direction term dkd_k and scaled by the SDE variance), with multiple rewards fused at the reward level.

Critical strengths
  1. The idea is simple and removes the teacher. Using a deterministic ODE rollout from the same parent state as the baseline is a clean way to turn terminal rewards into per-step advantages without a critic or a teacher, and black-box rewards can be used since they are never differentiated.
  2. The multi-reward setup is practical. A single model trained with z-normalized, weighted rewards gets the best GenEval and OCR among the compared methods, including the two teacher-based OPD baselines, while also keeping the preference scores on the same test images high.
  3. The ablations target the main design choices. The paper compares Best-of-K, all-branch, and all-branch with ∣Δt∣|\Delta t| normalization, and shows that letting dkd_k go up to 2 trains faster early but then collapses, which supports the bounded gate. It also measures preference gaps across prompt families instead of only reporting averages.
Critical weaknesses
  1. PickScore and HPSv2 are both used as training rewards and as evaluation metrics, and no held-out metric (for example ImageReward or an aesthetic predictor) or user study is given, so reward hacking cannot be ruled out. On the separate test set in Table 2, Self-OPD is behind DiffusionOPD on both PickScore (23.39 vs 23.95) and HPSv2 (0.3415 vs 0.3729), and behind DiffusionNFT on HPSv2 (0.3482). The paper then names the "same test images" protocol as primary in the appendix. The gap Δs\Delta s in Fig. 4(e,f) can also shrink when the aesthetic-prompt score drops, and Self-OPD's aesthetic-prompt PickScore in Fig. 4(e) looks lower than DiffusionOPD's. In Table 1, the Self-OPD HPSv2 model (0.4099) has no baseline trained on HPSv2, and the OCR model drops GenEval strict to 0.4186, below the base model (0.5222).
  2. Each training step rolls out K=8K = 8 branches plus one reference trajectory to a clean image and decodes and scores all of them, yet per-iteration cost, NFEs, GPU type, and GPU count are not reported. The efficiency claim compares Self-OPD's own runs against the hours reported in the DiffusionOPD paper, which may use different hardware. Even so, the from-scratch run needs about 90 h against 97.01 h for DiffusionOPD, which does not match the Fig. 4 caption "far lower wall-clock cost". The warm-start run uses single-reward GenEval and OCR specialists, which play a role close to teachers. Baseline numbers also look low: Flow-GRPO reaches GenEval strict 0.9005 here, while the Flow-GRPO paper reports 95% for SD3.5-M (from 63%, while the base here is 0.5222). All results come from one run with no variance.
  3. By Eq. 15, v(k)−vθv^{(k)} - v_\theta is a scaled noise zkz_k, so the gradient of Eq. 5 is an advantage-weighted sum of the injected noise. This is close to an evolution-strategies or reward-weighted regression estimator, and Proposition 1 is the standard gradient of a Gaussian log-likelihood. The push term is not derived from the KL objective and makes the loss unbounded below, which matches the collapse in Fig. 4(b). Branching rollouts for per-step credit assignment also appear in recent GRPO work, which is not discussed, and "reward-weighted regression" is mentioned without a citation. Key settings (αj\alpha_j, number of sampling steps SS, LoRA rank, batch size, τ\tau) are not given. Suggested references: Li et al., "BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models" (arXiv 2025); Li et al., "MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE" (arXiv 2025); Peng et al., "Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning" (arXiv 2019); Salimans et al., "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" (arXiv 2017).
Questions for authors
  • What is the per-iteration cost (NFEs, decodes, reward calls) and the hardware used, and were DiffusionOPD's hours measured on the same hardware?
  • Why is Flow-GRPO's GenEval (0.9005) lower than the 95% in the Flow-GRPO paper, and why is the base 0.5222 here versus 63% there?
3 comments
AuthorsOpenReview ↗

Response to Reviewer czWv(1/3)

Thank you for your review. We address your concerns about evaluation reliability, efficiency comparisons, and theoretical justification below.

PickScore and HPSv2 are both used as training rewards and as evaluation metrics, and no held-out metric ... or user study...

To ensure our model's improvements are genuine and free from reward hacking, we provide rigorous evaluations on unseen, held-out metrics alongside a human user study.

Table S1. Evaluation of Self-OPD and other methods on additional metrics using the GenEval/OCR test sets. Best scores are bold; second-best are underlined.

MetricSD3.5-MediumFlow-GRPOGRPO-GuardFlow-OPDDiffusionOPDDiffusionNFTSelf-OPD
ImageReward1.13701.19471.14111.37971.07841.4829‾\underline{1.4829}1.6094
Improved Aesthetic Predictor5.39585.18845.13355.46075.10535.7747‾\underline{5.7747}5.8055
NIMA5.47485.43975.34795.78705.34055.8169‾\underline{5.8169}5.9131
PaQ-2-PiQ75.303574.925374.554477.070774.766777.5123‾\underline{77.5123}77.9858
CLIP-IQA0.70450.69760.67290.73990.70710.72370.7262‾\underline{0.7262}
CLIP cosine distance0.41910.41570.41540.40700.40430.40370.4163‾\underline{0.4163}
DINOv2 cosine distance0.95300.9423‾\underline{0.9423}0.93920.93190.90800.91540.9396

Table S2. User study. Ten volunteers rated anonymized, randomly ordered images from four methods on 100 held-out prompts with one image per method per prompt. Task accuracy and aesthetics use 1–5 scales.

MethodTask accuracyAesthetics
Flow-OPD4.123.91
DiffusionOPD4.283.74
DiffusionNFT4.204.02
Self-OPD4.434.14

Tables S1–S2 support quality gains beyond the training rewards. Self-OPD leads on ImageReward (1.6094) and Improved Aesthetic Predictor (5.8055), and receives the highest blinded human ratings for task accuracy (4.43) and aesthetics (4.14), supporting human-aligned improvements.

On the separate test set in Table 2... The paper then names the "same test images" protocol as primary in the appendix.

Section 4.2 and Appendix D.2, “Preference Protocols: Same Test Images vs. Separate Test Set,” explain this choice. Our goal is for the same generated image to satisfy task accuracy and preference quality simultaneously. Evaluating both on the same outputs avoids conflating model quality with changes in the prompt distribution. The separate-set results remain in Table 2 as a complementary evaluation on another prompt distribution.

The gap in Fig. 4(e,f) can also shrink when the aesthetic-prompt score drops, and Self-OPD's aesthetic-prompt PickScore in Fig. 4(e) looks lower than DiffusionOPD's.

Compared with DiffusionOPD, Self-OPD achieves a smaller cross-prompt gap while improving the equal-weight average across aesthetic and GenEval/OCR prompts: PickScore rises from 23.335 to 23.630 and HPSv2 from 0.32025 to 0.33145 (Table S3). Our interpretation therefore rests on a smaller gap together with higher average preference scores, not on gap reduction alone. Appendix Figs. 6–7 further illustrate Self-OPD's aesthetic quality on GenEval/OCR prompts.

Table S3. Equal-weight mean preference scores derived from manuscript Table 2: mean = (aesthetic-prompt score + GenEval/OCR-prompt score) / 2.

MetricMethodAesthetic promptsGenEval/OCR promptsEqual-weight mean
PickScoreDiffusionOPD23.9522.7223.335
PickScoreSelf-OPD23.3923.8723.630
HPSv2DiffusionOPD0.37290.26760.32025
HPSv2Self-OPD0.34150.32140.33145

In Table 1, the Self-OPD HPSv2 model (0.4099) has no baseline trained on HPSv2.

PickScore provides an objective-matched comparison for another human-preference reward: Self-OPD achieves 24.47, versus 23.57 for Flow-GRPO and 23.98 for GRPO-Guard. This supports the effectiveness of Self-OPD for preference alignment.

The OCR model drops GenEval strict to 0.4186, below the base model (0.5222).

Single-reward training prioritizes the target reward and can trade off performance on other tasks. For example, PickScore-only Flow-GRPO and GRPO-Guard achieve GenEval strict scores of 0.5081 and 0.4602, respectively, both below the base model's 0.5222. Optimizing an aesthetic/preference scorer can shift the generation distribution toward the preferred visual characteristics rather than compositional accuracy. Similarly, OCR-only Self-OPD prioritizes text rendering over compositional generation. Its lower GenEval score is consistent with this cross-task trade-off under single-reward specialization, rather than a phenomenon unique to Self-OPD.

AuthorsOpenReview ↗

Response to Reviewer czWv(2/3)

What is the per-iteration cost (NFEs, decodes, reward calls) and the hardware used, and were DiffusionOPD's hours measured on the same hardware? Even so, the from-scratch run needs about 90 h against 97.01 h for DiffusionOPD, which does not match the Fig. 4 caption "far lower wall-clock cost".

DiffusionOPD states that its experimental setup follows DiffusionNFT, while DiffusionNFT states that its setup follows Flow-GRPO. Our experiments also follow Flow-GRPO, so the experimental settings are aligned. DiffusionOPD's training times are taken directly from its paper.

Two points clarify this comparison: DiffusionOPD's teacher-training cost and Self-OPD's warm-start setting.

First, the approximately 90 h from-scratch run is only modestly shorter than 97.01 h. However, DiffusionOPD reports the maximum of three teacher-training times (GenEval: 46.92 h; OCR: 33.17 h; aesthetics: 85.75 h), plus 11.26 h of student distillation. Its 97.01 h therefore reflects parallel teacher training, not the sum of all training jobs. Counting those jobs separately gives 177.10 h of summed job durations, versus approximately 90 h for our single from-scratch training pipeline without separate teacher-training jobs. This distinction exposes the parallel resource requirement; 177.10 h is neither wall-clock time nor GPU-hours without GPU counts.

Second, our warm-start setting further shortens training: GenEval and OCR specialists take 27 h and 37 h in parallel, followed by up to approximately 11 h of multi-reward training. Including specialist pretraining, the total is approximately 48 h, about 50.5% shorter than the reported 97.01 h. Notably, this shorter reported duration is achieved while jointly optimizing three rewards per task image—OCR or GenEval, plus PickScore and HPSv2—rather than task accuracy alone. Thus, the shorter duration is obtained with additional preference rewards, not by reducing supervision to a single task reward. We will clarify that “far lower wall-clock cost” refers to warm-start, rather than the modest from-scratch reduction; these reported-time comparisons are not controlled same-hardware speedups.

The warm-start run uses single-reward GenEval and OCR specialists, which play a role close to teachers.

Warm-start is an optional initialization strategy to reduce training time, not teacher-based supervision. It can use intermediate specialist checkpoints rather than the best-performing ones, allowing shorter single-reward pretraining. These checkpoints only initialize the student and are not queried as separate teachers during subsequent training; updates are driven by reward-model feedback on the student's own local branches. In contrast, DiffusionOPD and Flow-OPD require generative teacher checkpoints and use their transformer predictions on noisy latents as distillation supervision.

Why is Flow-GRPO's GenEval (0.9005) lower than the 95% in the Flow-GRPO paper, and why is the base 0.5222 here versus 63% there?

Appendix D and Fig. 5 explain our two GenEval scoring modes: strict scoring requires all prompt sub-requirements to be satisfied, while continuous scoring gives partial credit. The cited 0.9005 and 0.5222 are our strict scores for Flow-GRPO and the base model, respectively. Their continuous scores are 0.9470 and 0.6219, close to the reported 95% and 63%. These two values come from our own evaluation, not from the Flow-GRPO paper. Under the same stricter criterion, Self-OPD achieves 0.9536 versus Flow-GRPO's 0.9005, an improvement of 5.31 percentage points.

All results come from one run with no variance.

We conducted multiple independent training runs using Self-OPD's Table 2 multi-reward configuration across three different random seeds (N=3). In Table S5 below, we report the original single-run scores and the mean and standard deviation (std) across the three runs.

As shown in the table, the relative standard deviations (std divided by mean) are at most 1.1% for all six metrics. These small variations suggest that Self-OPD's performance is consistent across the tested seeds and support the stability of the reported scores.

Table S5. Self-OPD multi-reward independent-training-run statistics.

BenchmarkOriginalMultiple Runs
GenEval0.96910.9689 ± 0.0015
OCR0.95970.9601 ± 0.0018
PickScore23.3923.41 ± 0.05
HPSv20.34150.3414 ± 0.0011
AuthorsOpenReview ↗

Response to Reviewer czWv(3/3)

By Eq. 15, (v(k)−vθ)(v^{(k)} - v_\theta) is a scaled noise (zkz_k), so the gradient of Eq. 5 is an advantage-weighted sum of the injected noise. This is close to an evolution-strategies or reward-weighted regression estimator, and Proposition 1 is the standard gradient of a Gaussian log-likelihood.The push term is not derived from the KL objective and makes the loss unbounded below, which matches the collapse in Fig. 4(b).

Eq. 15 expresses the branch displacement (v(k)−vθ)(v^{(k)} - v_\theta) as a scaled noise zkz_k. This relation alone does not describe the full learning procedure. Self-OPD scores the branches' ODE completions against its own ODE baseline and updates all branches with direction-aware weighting. Proposition 1 provides the transition-variance normalization used in this update. Considering only the noise-weighted gradient therefore overlooks the local target construction, self-reference baseline, and branch update design.

Appendix B.2 already distinguishes the two roles: the KL analysis motivates transition-variance normalization, while Eq. 5 is a practical signed regression surrogate. The paper does not claim that the push term is an exact KL gradient. The collapse in Fig. 4(b) occurs in the stronger repulsion variant with dkd_k in [0,2][0, 2], while our proposed [0,1][0, 1] setting remains stable in that experiment. Here, “bounded” describes the coefficient range, not a theorem about the global lower bound of the loss. The ablation supports controlling repulsion strength rather than showing collapse of the proposed setting.

Branching rollouts for per-step credit assignment also appear in recent GRPO work, which is not discussed, and "reward-weighted regression" is mentioned without a citation.

We will add the missing citations and discuss the following related work:

  • BranchGRPO: BranchGRPO constructs a rollout tree and propagates leaf rewards across depths to assign step-wise advantages.
  • MixGRPO: MixGRPO performs GRPO updates within a sliding window of SDE steps and uses deterministic ODE sampling outside that window.
  • DenseGRPO: DenseGRPO uses reward gains between adjacent denoising states to construct step-wise dense advantages.
  • TAGRPO: TAGRPO uses high- and low-reward trajectory anchors and a memory bank to guide group-relative policy updates.
  • Reward-weighted regression: Peters and Schaal (2007) fit a policy using samples weighted by their rewards.
  • Advantage-weighted regression: Peng et al.'s AWR (2019) learns a value-function baseline from replay data and uses the resulting advantages to weight policy regression.
  • Evolution strategies: Salimans et al.'s ES (2017) weights random parameter perturbations by their rewards to estimate an update direction.

Self-OPD differs in its reference and update rule, using the reward of a deterministic ODE rollout from the same state as the baseline for signed velocity regression over all local branches. The loss is normalized by transition variance, and repulsion is reduced when it would oppose the best branch.

Reviewer_qtHLOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission presents Self-OPD, that removes the teacher from on-policy distillation (OPD) for flow-matching models. At each training timestep, the student's next state is perturbed into eight SDE branches, and each branch is rolled out with the ODE sampler and scored by a reward model. The advantage of each branch is its reward minus that of a deterministic ODE self-reference, normalized by the spread of branch rewards. A pull-push loss pulls the velocity field toward positive-advantage branches and pushes it away from negative ones, with direction-aware damping and SDE-variance scaling that Proposition 1 ties to a per-step KL. For multiple objectives, rewards are z-normalized and summed before advantages are computed. Experiments fine-tune SD3.5-Medium (512px, LoRA) on GenEval, OCR, PickScore and HPSv2 against RL and teacher-based OPD baselines.

Critical strengths
  1. Clear practical motivation. Removing the per-objective teacher-training cost of Flow-OPD and DiffusionOPD addresses a real limitation.
  2. Simple, sensible design. Local branching gives credit assignment at each step instead of relying on one terminal reward. The ODE self-baseline is cheap and needs no critic.
  3. Useful observation about multi-objective training. DiffusionOPD's task-conditioned matching lets preference quality depend on the prompt family. The cross-dataset gap analysis (Fig. 4e, f), including the aesthetic co-teacher control, is a good experiment.
  4. Strong single-model multi-reward results. In Tab. 2, Self-OPD has the best GenEval (0.9521 strict) and OCR (0.9597), and the best preference scores on the same test images.
  5. Informative ablations. Best-of-K vs all-branch (Fig. 4a) and bounded vs unbounded repulsion (Fig. 4b) are useful; the unbounded variant collapses.
Critical weaknesses
  1. Experiments use a single base model (SD3.5-M) at a single resolution (512px). There are no results on FLUX or at higher resolution, where the cost of K + 1 rollouts matters most.
  2. Section 4.2 says single-reward Self-OPD reaches PickScore 24.79 and HPSv2 0.3665. Table 1 shows 24.47 and 0.4099.
Questions for authors

What are the NFEs and GPU-hours per update compared with Flow-GRPO and DiffusionNFT, on the same hardware? What are the results on held-out reward models and on diversity metrics?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qtHL

Thank you for the positive feedback. We address your concerns about model coverage, numerical consistency, computational cost, and independent evaluation below.

Experiments use a single base model (SD3.5-M) at a single resolution (512px). There are no results on FLUX or at higher resolution...

We provide separate additional comparisons on FLUX.1-dev (Table S1) and SD3.5-Medium at 1024×1024 (Table S2).

Table S1. FLUX-base comparison (FLUX.1-dev, 512×512).

MethodGenEval stri.GenEval cont.OCRPickScore
FLUX-base0.53930.65330.619022.15
Flow-GRPO0.90510.94740.945623.74
Self-OPD0.95420.96710.970424.25

Table S2. SD3.5-Medium-base comparison at 1024×1024: single- and multi-reward.

MethodTraining rewardGenEval stri.GenEval cont.OCRPickScoreHPSv2
SD3.5-Medium—0.51130.63030.532122.130.3032
Flow-GRPO+ RM GenEval0.89880.9405———
-+ RM OCR——0.9116——
-+ RM PickScore———23.51—
GRPO-Guard+ RM GenEval0.90100.9464———
-+ RM OCR——0.9205——
-+ RM PickScore———23.84—
Self-OPD+ RM GenEval0.93290.9548———
-+ RM OCR——0.9672——
-+ RM PickScore———24.21—
-+ RM HPSv2————0.3934
MethodGenEval stri.GenEval cont.OCRPickScore (same test images)HPSv2 (same test images)PickScore (separate test set)HPSv2 (separate test set)
SD3.5-Medium0.51130.63030.532122.710.282922.130.3032
Flow-OPD0.84450.91680.928523.430.300622.970.3300
DiffusionOPD0.90400.93740.935422.930.279423.620.3658
DiffusionNFT0.87450.91190.924423.620.318323.370.3446
Self-OPD0.94480.96890.950523.840.322923.560.3452

Section 4.2 says single-reward Self-OPD reaches PickScore 24.79 and HPSv2 0.3665. Table 1 shows 24.47 and 0.4099.

Thank you for pointing this out. The inconsistency is our mistake; the correct values are those in Table 1 (PickScore 24.47 and HPSv2 0.4099). We will correct Section 4.2 accordingly.

What are the NFEs and GPU-hours per update compared with Flow-GRPO and DiffusionNFT, on the same hardware?

We benchmark Flow-GRPO, DiffusionNFT, and Self-OPD under the same hardware configuration.

Table S3. Single-reward time-to-target comparison, measured in GPU-hours (NFEs = 10).

Training rewardScore cutoffSelf-OPDDiffusionNFTFlow-GRPO
GenEval0.947086922150
OCR0.92533235731
PickScore23.571531821525
HPSv20.37392405241538

Table S3 compares the training cost to reach target scores under 10 NFEs. Self-OPD achieves the fastest convergence across all benchmarks, outperforming Flow-GRPO by 6.4×6.4\times to 25×25\times. It also consistently surpasses DiffusionNFT, notably reducing the training time by over half on HPSv2.

What are the results on held-out reward models and on diversity metrics?

Held-out reward/image-quality and feature-diversity results are reported in Table S4.

Table S4. Evaluation of Self-OPD and other methods on additional metrics using the GenEval/OCR test sets. Best scores are bold; second-best are underlined.

MetricSD3.5-MediumFlow-GRPOGRPO-GuardFlow-OPDDiffusionOPDDiffusionNFTSelf-OPD
ImageReward1.13701.19471.14111.37971.07841.4829‾\underline{1.4829}1.6094
Improved Aesthetic Predictor5.39585.18845.13355.46075.10535.7747‾\underline{5.7747}5.8055
NIMA5.47485.43975.34795.78705.34055.8169‾\underline{5.8169}5.9131
PaQ-2-PiQ75.303574.925374.554477.070774.766777.5123‾\underline{77.5123}77.9858
CLIP-IQA0.70450.69760.67290.73990.70710.72370.7262‾\underline{0.7262}
CLIP cosine distance0.41910.41570.41540.40700.40430.40370.4163‾\underline{0.4163}
DINOv2 cosine distance0.95300.9423‾\underline{0.9423}0.93920.93190.90800.91540.9396

As shown in Table S4, Self-OPD achieves the best results on ImageReward, Improved Aesthetic Predictor, NIMA, and PaQ-2-PiQ, demonstrating its superior aesthetic and perceptual quality. Meanwhile, on the CLIP and DINOv2 diversity metrics, Self-OPD also ranks at the forefront among all fine-tuned methods, confirming that it improves image quality while preserving strong diversity.

We hope these additional results and clarifications address your concerns, and we would be grateful if you could reconsider your score in light of this additional evidence.

Reviewer_qtHLOpenReview ↗

reply to the response

Thank you very much for your additional experiments, although it's not required, I will raise my score. By reading at the replies here, I feel it is very solid.

Thank you sincerely for raising the score and for your encouraging feedback! We are glad that our response addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.