acceptodds
Withdrawn from ICLR 2027

Generalizing On-Policy Self-Distillation for Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scalar image rewards can guide diffusion model training, but translating them into effective parameter updates remains challenging. DiffusionOPSD offers an explicit target-fitting route, yet reward-gradient-based target construction still requires differentiable evaluators. Thus, we present GenOPSD, an on-policy self-distillation method for score-only feedback. It converts scores of decoded images from completed original trajectories and projected self-SDE branches into detached query-level targets, then fits the clean-latent targets with uniform positive clean-prediction MSE and refreshes the policy. Rewards affect target construction only, without gradients through the evaluator or decoder or a reward-weighted policy-gradient loss. Matched single-reward SD3.5-M results support the generalization: GenOPSD approaches DiffusionOPSD on four differentiable reward metrics, staying within 5.6% across all four and gaining 1.5% on ImageReward, while also supporting score-only GenEval and OCR evaluation. Under multi-reward training, its scores remain within 7.8% of DiffusionNFT across measures. On Z-Image-Turbo, it improves over the base by 2.5% on GenEval and 1.2% on OCR. On Qwen-Image-Bench, it improves over the SD3.5-M base by 34.2% overall and by 8.9% to 62.5% across five categories. Ablations support the roles of both state sources, prompt-population-relative signed preferences, coverage over native noise levels, and policy refresh. Guidance tests favor the CFG 1 actor, and qualitative examples show stronger composition and text rendering but residual color and season errors. We hope GenOPSD offers the community an effective method for training diffusion models with scalar reward feedback.

Primary area:
generative models
Keywords:
image generation, on-policy self-distillationimage generation, on-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4a6KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GenOPSD, a score-only variant of on-policy self-distillation for flow models. For each fresh ODE trajectory, it also completes a projected self-SDE branch (as in Self-OPD) and scores both decoded images with a black-box evaluator. Rewards are normalized within each prompt group into clipped signed preferences α. Each final latent is forward-noised at nine noise levels, and a detached target is built. The model is then fitted with one AdamW step of clean-prediction MSE, and the sampler is refreshed. Experiments on SD3.5-M (single- and multi-reward) and Z-Image-Turbo compare GenOPSD with FlowGRPO, DiffusionNFT, Self-OPD and the reward-gradient method DiffusionOPSD at matched trajectory budgets, with additional Qwen-Image-Bench evaluation and training-reward ablations.

Critical strengths
  1. Practical and competitive with score-only feedback. On single-reward SD3.5-M, GenOPSD stays within 2.9–5.6% of the reward-gradient method DiffusionOPSD on PickScore, CLIPScore and HPSv2.1 and matches it on ImageReward (1.35 vs. 1.33). At the same trajectory budget, it also outperforms DiffusionNFT on these metrics and on GenEval and OCR. Because many useful evaluators (GenEval, OCR, API judges) are non-differentiable, this matters in practice.
  2. Careful accounting and useful ablations. Trajectories, scored images, query forwards and optimizer steps are reported separately. The ablations show clearly that negative preferences and sampler refresh are both necessary.
Critical weaknesses
  1. The novelty is overstated. GenOPSD takes a single step from the point where the target is built (θ_old = θ_i, Prop. A.2). At that point, the gradient of Eq. (10) equals the gradient of a signed, reward-weighted flow-matching loss Σ(ησ²α/2ω)‖y_θ − x‖². In other words, rewards do act as per-sample loss weights, and the "uniform positive MSE" is a reparameterization. Appendix A.3 builds the target from DiffusionNFT's components: the implicit positive and negative policies p± (Eq. 18), the optimality weight q = (1 + α/κ)/2 (Eq. 17), and ω, which corresponds to DiffusionNFT's self-normalized x₀-error weighting. The result is DiffusionNFT's gradient in its fully on-policy limit (η = 0) with one step per batch. What is new is the extra self-SDE branch sample and the nine-level query coverage, and the paper never tests these against DiffusionNFT directly. Advantage-weighted flow-matching methods such as AWM (Xue et al., 2025), which make the same equivalence explicit, are not discussed. The paper presents itself as a "generalization of OPSD," but by its own admission, the link is only "operational."

  2. Some headline gains rest on weak references, and the evidence is thin. The Qwen-Image-Bench gains (+34.2% overall, +62.5% on Quality) are measured against the SD3.5-M base sampled without CFG. That model is weak: its GenEval macro score is 0.297, versus 0.667 with CFG. Against the base with CFG 4.5, Fig. 7c shows a GenEval gain of only 8.8%. Part of the improvement may therefore come from distilling guidance into the model rather than from reward optimization.

  3. In addition, on Z-Image-Turbo, GenOPSD trails DiffusionOPSD by 11.7% on HPSv2.1 and 29.5% on ImageReward. Neither the abstract nor the text mentions this; the Z-Image-Turbo results are compared only against the base model. Every result is a single run with no seeds or confidence intervals, even though several margins are about one point (e.g., Z-Image-Turbo GenEval +1.8 and OCR +1.0). The ablations report relative changes in training reward after 20 updates, not held-out metrics.

  • Xue. et al. Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models. 2025
Questions for authors
  • At your one-step synchronized setting, does the GenOPSD gradient match that of DiffusionNFT with η = 0 (up to the weighting scale)? If so, please run a direct ablation of "DiffusionNFT + self-SDE branch sample + nine noise levels" against GenOPSD.

  • Please report Qwen-Image-Bench and GenEval results for the SD3.5-M base with CFG 4.5. Please also state which training CFG each matched baseline used.

  • Please report mean ± std over at least three seeds for the main SD3.5-M and Z-Image-Turbo results, and explain why GenOPSD trails DiffusionOPSD on Z-Image-Turbo.

  • GenOPSD scores twice as many images per trajectory as DiffusionNFT and adds SDE-suffix sampling. Please report wall-clock time or GPU-hours for all methods.

Reviewer_hUPtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GenOPSD, an on-policy self-distillation method for diffusion models using scalar rewards without reward gradients. It combines completed original trajectories and projected self-SDE branches to construct reward-informed, detached clean-prediction targets, which are fitted using MSE.

Critical strengths
  1. GenOPSD provides an interesting way to construct explicit training targets from scalar rewards. It converts prompt-relative signed preferences into detached clean-prediction targets, allowing reward-based optimisation through standard MSE without requiring reward gradients.
  2. The combination of original trajectories and projected self-SDE branches provides additional on-policy outcomes. Using both sources across nine distinct noise levels allows the model to learn from diverse states while requiring only two reward-scored images per original trajectory.
  3. The ablation studies provide meaningful support for the proposed design. In particular, removing negative preferences, reducing noise-level coverage, or freezing the sampling policy substantially degrades performance.
  4. The experimental results are promising. GenOPSD achieves competitive performance with gradient-based DiffusionOPSD and outperforms DiffusionNFT on GenEval and OCR in the matched single-reward setting, despite using only scalar rewards.
Critical weaknesses
  1. Limited novelty over DiffusionNFT. The underlying optimisation principle remains similar: encouraging predictions towards high-reward samples and away from low-reward samples. GenOPSD reformulates this supervision as explicit detached-target fitting, but the paper does not clearly demonstrate the specific advantage of this formulation over DiffusionNFT's negative-aware objective.
  2. Limited and inconsistent improvements over DiffusionNFT. GenOPSD achieves modest improvements on GenEval and OCR in the matched single-reward setting, but performs worse on both under multi-reward training. The results do not convincingly establish a consistent advantage over DiffusionNFT.
Questions for authors

Under the synchronized behavior-model setting in Appendix A.3, GenOPSD's target construction reduces to a reward-signed correction that pulls predictions towards high-reward samples and pushes them away from low-reward samples. The resulting generator gradient appears closely related to DiffusionNFT's implicit positive/negative optimisation, although the scaling and regularization differ.

Could the authors clarify the fundamental difference between these two optimisation approaches? A controlled comparison using identical training samples, noise levels, and other settings would help establish the specific contribution of GenOPSD's explicit target-fitting formulation.

Reviewer_jDXMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

While diffusion/flow matching-based generative models have shown remarkable progress in content generation, their outputs do not always align with human preferences or values. Aligning pretrained models with these preferences is therefore important for real-world deployment. Since asking humans to evaluate every generated sample is too costly, we often use automated reward functions, including other models, to evaluate the outputs. When the reward model is differentiable and we have access to its gradients, we can use them directly for alignment. However, many reward functions are black-box or inherently non-differentiable. This paper proposes a practical method for aligning diffusion/flow matching-based generative models using such rewards.

Critical strengths
  1. I agree that developing an algorithm that works with non-differentiable rewards is important. Many reward functions are inherently non-differentiable, not just black-box, such as outcomes from simulations in scientific applications.

  2. The authors compared the proposed method with several relevant competing methods through extensive experiments.

  3. The authors provided useful ablations to examine the role of individual components.

Critical weaknesses
  1. Some components are not clearly explained. For example, I am not sure what exactly “projected self-SDE” means. Up to my understanding, it adds stochasticity to the flow model’s ODE sampling by adding a diffusion term and a corresponding drift correction. But this is still my interpretation, and the paper does not clearly explain whether this is correct or what is being “projected.” Giving the actual update equation and explaining how it works with the solver would make this part much easier to understand and reproduce.

  2. It is unclear why using current-policy samples and self-SDE samples together is better. The matched ablations seem to generate and score the same number of images and use the same number of training queries. But when both sources are used, training uses more distinct final images. So, is the improvement really from self-SDE branching, or just from using more different images for training? What happens if we use more independent initial noises instead of self-SDE branches? I think this comparison is needed, with the reward-query and sampling costs also taken into account.

  3. Also, the reason for using self-SDE branching is not very clear beyond the experimental results. Intuitively, samples sharing part of the trajectory could give useful local variations, but they are also more correlated. Whether this helps would likely depend on where the branching happens and how much noise is added. Also, the method re-noises the final images for training rather than directly using the paired branches at their shared intermediate state. This does not remove all the effects of branching, but it makes me wonder how much the shared trajectory actually matters. It would help to explain what property of branching is useful here and test how sensitive the method is to these choices.

  4. Although DiffusionOPSD requires reward gradients, we could potentially approximate them using a zeroth-order method for black-box rewards. Of course, this could be noisy and expensive in high-dimensional spaces, so I am not saying it would necessarily work well. But it still seems like a relevant baseline. What happens if we apply DiffusionOPSD with zeroth-order gradient estimates, and how does it compare with GenOPSD in performance and cost?

  5. My biggest concern is that most of the main components already exist. Stochastic branching was used in Self-OPD, and evaluating final images, re-noising them, and using positive/negative reward-based updates were used in DiffusionNFT. The target construction and update procedure here could still be useful, but it is not clear enough what is new and why this combination is better than simpler alternatives. Right now, the paper feels mainly like a good combination of existing components. Also, the results in Table 1 are mixed rather than consistently better than strong baselines. So I think the paper needs stronger controlled comparisons and a clearer explanation of what this particular method adds.

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.