acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Reinforcement Learning via Centered Reward Distillation

Abstract

Diffusion and flow models achieve State-Of-The-Art (SOTA) generative performance, yet many practically important behaviors such as fine-grained prompt fidelity, compositional correctness, and text rendering are weakly specified by score or flow matching pretraining objectives. Reinforcement Learning (RL) fine-tuning with external, black-box rewards is a natural remedy, but diffusion RL is often brittle. Trajectory-based methods incur high memory cost and high-variance gradient estimates; forward-process approaches converge faster but can suffer from distribution drift, and hence reward hacking. In this work, we present Centered Reward Distillation (CRD), a diffusion RL framework derived from KL-regularized reward maximization built on forward-process-based fine-tuning. The key insight is that the intractable normalizing constant cancels under within-prompt centering, yielding a well-posed reward-matching objective. To enable reliable text-to-image fine-tuning, we introduce techniques that explicitly control distribution drift: (i) decoupling the sampler from the moving reference to prevent ratio-signal collapse, (ii) KL anchoring to a CFG-guided pretrained model to control long-run drift and align with the inference-time semantics of the pre-trained model, and (iii) reward-adaptive KL strength to accelerate early learning under large KL regularization while reducing late-stage exploitation of reward-model loopholes. Experiments on text-to-image post-training with GenEval and OCR rewards show that CRD achieves competitive reward optimization results with fast convergence and reduced reward hacking, as validated on unseen preference metrics.

Primary area:
generative models
Keywords:
diffusion models, diffusion reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZaTuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose diffusion fine-tuning technique that combines multiple previous works, such as GRPO, DPO, and Diffusion-NFT. The authors propose two main contributions, CRD which aims to minimize the explicit reward with implicit reward with ELBO (though not exactly) approximation, and CFG-based KL regularization term to penalize the model from deviating too far from the CFG distribution of the base model. The paper compares with different diffusion fine-tuning on image generation tasks, improving quality and compositionality in some cases.

Critical strengths

S1. Reward-adaptive KL anchoring. Scaling the initial-reference KL coefficient by the sample’s raw reward is a simple, practical idea. The ablation study shows promising results, though it is questionable whether this alone can solve the reward hacking problems for different models or applications. However, I assume that this can also suffer when the group size (K) is small, where the reward can be biased.

Critical weaknesses

W1. The derivations are loose, and the key approximations are not validated. The paper presents derivations from the stated objective to the final training loss relies on a series of approximations that are neither justified theoretically nor checked empirically. In particular, the paper frames the intractable normalization constant as the central bottleneck. However, for diffusion models, the real difficulty is the intractable log-likelihood itself. In fact, the vanishing normalization constant is hardly can be seen as a novel breakthrough. The paper sidesteps this by substituting a surrogate for the log likelihood with w=1w=1, which is not a proper ELBO, so it is unclear what quantity is actually being optimized. There seems to be a lack of analysis or experiment that quantifies how tight this surrogate or how sensitive the resulting policy is.

W2. The CFG-based KL term and the per-sample beta are inconsistent with the stated objective. Replacing the KL to the reference model with a CFG-based KL changes the regularizer and therefore no longer corresponds to the entropy-regularized reward objective or its derived optimal distribution in Eq. 1. Additionally, introducing a per-sample beta coefficient further deviates the method from that objective. These deviations should be stated explicitly or presented with more theoretical insights.

W3. The overall justification for the proposed design is weak. Taken together, the derivation reads as a post-hoc rationalization of a set of heuristics rather than a principled method. After reading the paper, I am still not convinced why, or under what conditions, diffusion RL fine-tuning should benefit from this particular combination of ingredients.

W4. The empirical gains are marginal. The quantitative results are comparable to existing baselines, which does not provide strong evidence for the effectiveness of the proposed choices and raises concerns on their justification. The paper would be more strengthened by providing more evidence why each componenet has to be designed in the proposed specific manner, and how each compoenent affects the final performances.

Questions for authors

Can the authors present more evident on why their proposed method still holds theoretically, or at least empirically plausible? Additionally, how does the proposed method behave with and without each component shown in the method section? What is more critical choice and what is the intuition behind this? There are some ablations, but seems like internal study, rather than full comparison. More extensive ablation study would clearly show the effectiveness of the method.

Reviewer_BJ98OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies efficient reinforcement learning for diffusion/flow models using black-box rewards while limiting distribution drift and reward hacking. It proposes Centered Reward Distillation (CRD), which matches within-prompt differences between external rewards and implicit rewards constructed from a diffusion likelihood surrogate. Centering eliminates the prompt-dependent partition function in the ideal KL-regularized formulation. The method combines this objective with separate sampling and reference EMA models, anchoring to a fixed CFG-guided pretrained model, and reward-dependent anchoring strength. Experiments fine-tune SD3.5-Medium on GenEval and OCR rewards, with additional multi-reward experiments and evaluation using held-out preference scorers. The results suggest that the combined recipe can improve task performance while preserving visual quality. The central contribution to assess is how much this recipe advances beyond existing centered reward regression and efficient forward-process diffusion RL.

Critical strengths
  1. The paper addresses a consequential trade-off beyond maximizing the training reward. Efficient reward optimization is already possible, but maintaining scene quality while improving OCR or compositional correctness remains practically important. Table 1 evaluates several scorers beyond the optimized reward, and Figures 3 and 5 illustrate concrete failures such as replacing a requested scene with text on a plain background. These results make the quality-preservation problem tangible.

  2. The stabilization recipe is useful and potentially transferable. Separating the sampling model from the log-ratio reference and anchoring to the CFG-guided pretrained behavior are sensible design choices for this setting. The ablations examine reference update speed, anchoring strength, and adaptive regularization. Appendix H.3 also applies the regularization to DiffusionNFT, suggesting that this component may benefit an existing algorithm independently of CRD.

Critical weaknesses
  1. The incremental contribution beyond the closest existing methods is not sufficiently isolated.

    Removing a prompt-dependent normalizer through relative or centered reward regression is already established in REBEL and GVPO. More specifically, Appendix H.4 states that all other experiments use uniform centering, precisely the setting identified in Appendix F as recovering GVPO. With ei=ri−R^θ,ie_i = r_i - \widehat{R}_{\theta,i}, the implemented centered loss has the form

    1K∑i(ei−eˉ)2.\frac{1}{K}\sum_i(e_i-\bar e)^2.

    Thus, the principal evaluated objective has an existing centered-regression structure, instantiated with a diffusion surrogate. Efficient forward-process training is also established by AWM and DiffusionNFT. These connections are acknowledged, but they substantially narrow the remaining contribution.

    A useful diffusion-specific adaptation can still be a meaningful contribution. However, the paper needs a controlled comparison that changes the objective while holding the sampler, reference updates, likelihood surrogate, CFG anchoring, and adaptive regularization fixed. Figure 7 already shows that regularization improves DiffusionNFT, making this comparison especially important. Without it, the results do not establish which benefits require CRD rather than the accompanying stabilization recipe. The relationship to VMPO, which already develops variance-based diffusion alignment, also deserves a more precise technical comparison than the discussion of differing experimental scope.

  2. The evidence does not yet establish a clear improvement in the joint reward–quality–compute trade-off.

    On OCR, Table 1 reports 0.92 for CRD, matching Flow-GRPO with KL, while CRD has lower ImageReward, PickScore, CLIPScore, and HPSv2.1 at its default inference setting. AWM with CFG reaches 0.96 OCR with broadly comparable quality scores. CRD has a more favorable result on GenEval, particularly for aesthetics, but this supports a task-dependent trade-off rather than a consistent advantage. In the multi-reward setting, Table 4 also places CRD below DiffusionNFT and Choi et al. on several metrics.

    Efficiency could justify these trade-offs, but the paper does not provide a direct comparison of reward and held-out quality against GPU-hours, including sampling, reward evaluation, and the additional reference computations. Training-step ablations and the statement that runs finish within 36 hours do not establish the comparative speedup. Reported point estimates also lack uncertainty estimates, making small preference-score differences difficult to interpret. Finally, evaluating alternative scorers on DrawBench provides useful evidence of generalization, but does not directly establish reduced reward exploitation on the same OCR/GenEval prompts where optimization occurs.

Questions for authors
  1. Contribution attribution: Under an otherwise identical training pipeline, how does CRD compare with DiffusionNFT and an alternative forward-process objective when each receives the same CFG-reference anchoring, adaptive regularization, and sampling/reference EMA configuration? In particular, does CRD retain a measurable advantage over the regularized DiffusionNFT variant already explored in Figure 7?

  2. Practical trade-off: Can you provide matched comparisons of task reward and held-out quality against total GPU-hours, together with the checkpoint-selection rule and uncertainty estimates? For the reward-hacking claim, can you also evaluate independently judged scene fidelity on the same prompts used for OCR/GenEval evaluation? These results would help distinguish improved robustness from stronger regularization or earlier stopping.

  3. Scope of the theoretical interpretation: The centering identity is exact for fixed rewards and exact log-density ratios. In the actual implementation, rewards are standardized using the sampled group, and Eq. (13) uses model-dependent adaptive normalization of denoising errors. What connection to a fixed KL-regularized population objective survives these choices? In particular, centering removes prompt-level constants but does not generally remove sample-dependent likelihood-surrogate errors. Please distinguish the exact result from the heuristic interpretation of the implemented update.

Reviewer_FuA9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies reward based RL finetuning for diffusion/flow-based models, and proposes a new algorithm named centered reward distillation. The key RL loss function regresses the group-centered implicit model rewards (estimated by ELBO differences under current and pretrained models) to the group-centered external rewards using L^2 loss, plus a KL anchoring loss w.r.t. the CFG-guided pretrained model prediction, whose strength depends on raw rewards. The method is tested on SD3.5-M and shows competitive performance compared to baselines

Critical strengths
  • Reward based RL finetuning for diffusion is an important and timely topic. The paper is well-written and very easy to follow
  • Matching implicit model rewards and using centering trick to avoid the intractable log normalization constant is a clever design
  • The paper provides a comprehensive ablation study of the design choices for the proposed method
  • The paper provides evaluation metrics on non-training rewards to measure the generalization capability, giving some evidence that the finetuned model doesn't suffer from reward hacking
Critical weaknesses
  • The experiments are limited to SD3.5-M, which is relatively old for a paper in late 2026. I suggest the authors to strengthen the empirical results by testing on other models such as Flux or Qwen Image
  • Like DiffusionNFT, the proposed method still requires subtle tuning of the EMA; further, two sets of parameters θold\theta_{old} and θsample\theta_{sample} need to be maintained to stabilized training and serve in different parts of the loss, which makes the implementation more complex
Questions for authors
  • Can the authors provide some theoretical guarantees for the CRD loss? For example, suppose ELBO estimation is accurate, we have infinitely many generations per prompt, the training is always on policy, and no anchoring loss, can we guarantee that the pθ∗p _ {\theta ^ *} is the unique fixed point of the RL loss?
  • The KL anchoring loss regresses the current (no-CFG) model prediction to the pretrained (with CFG) model prediction. Can the authors justify the performance improvement mostly comes from RL (with external reward model) instead of simply distilling the pretrained model's CFG prediction into the current model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.