acceptodds
Under review as a conference paper at ICLR 2027

Data-regularized Reinforcement Learning for Diffusion Models at Scale

Abstract

Aligning generative diffusion models with human preferences via reinforcement learning (RL) is critical yet challenging. Most existing algorithms are often vulnerable to reward hacking, such as quality degradation, over-stylization, or reduced diversity. Our analysis demonstrates that this can be attributed to the inherent limitations of their regularization, which provides unreliable penalties. We introduce Data-regularized Diffusion Reinforcement Learning (DDRL), a novel framework that uses the forward KL divergence to anchor the policy to an off-policy data distribution. Theoretically, DDRL enables robust, unbiased integration of RL with standard diffusion training. Empirically, this translates into a simple yet effective algorithm that combines reward maximization with diffusion loss minimization. Extensive experiments and double-blind human votes demonstrate that DDRL significantly improves rewards while alleviating the reward hacking seen in baselines, achieving the highest human preference and establishing a robust and scalable paradigm for diffusion post-training.

Primary area:
generative models
Keywords:
diffusion model, video generation, reinforcement learning, theory, reward hacking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EAuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reward hacking in reinforcement learning for diffusion models. The authors argue that existing diffusion RL methods can become unreliable because their regularization is evaluated on samples from the current policy, precisely where the reward model and reference policy may be least trustworthy after aggressive optimization. The proposed method, DDRL, instead anchors the model to an off-policy data distribution. The paper connects this idea to forward-KL regularization and shows that, under its theoretical formulation, the regularizer can be implemented using the standard diffusion training loss. In practice, the resulting algorithm combines reward optimization on policy rollouts with diffusion training on either real or synthetic reference data. The method is evaluated on Cosmos2.5 2B and 14B models for text-to-video and image-to-video generation, with VideoAlign and VBench as optimization targets. The authors also report roughly 9,600 pairwise human judgments and additional SD3.5 text-to-image experiments using synthetic regularization data. Across the tested settings, DDRL generally improves reward while avoiding much of the preference degradation and visual artifacts observed with DanceGRPO and FlowGRPO. The paper also studies a reduced-cost version of DDRL and explores combining SFT and RL into a single post-training stage.

Critical strengths
  1. The paper addresses an important practical failure mode with unusually strong empirical evidence. Reward hacking is particularly difficult to evaluate using the optimized reward itself, so the large-scale human comparisons are important here. The video experiments cover two model scales, two generation settings, and two reward functions, and the qualitative and preference results consistently show the same failure pattern for the competing RL methods. This makes the empirical case much stronger than a standard reward-only comparison.

  2. The proposed regularization principle is simple and potentially quite useful. The connection between forward-KL regularization against an off-policy data distribution and the standard diffusion training loss gives a clean interpretation of why replaying high-quality data can prevent reward optimization from drifting away from the data manifold. I also appreciate that the method does not require keeping a reference model in memory during training, and that the paper shows a reduced-cost implementation with comparable performance.

Critical weaknesses
  1. The theoretical optimum is derived over a more general distribution class than the actual diffusion policy appears to support. The proof treats the full trajectory distribution as the optimization variable and derives an ideal reward-tilted solution. However, the actual model is restricted to a particular reverse diffusion parameterization with a fixed terminal prior and constrained transition family. The paper does not explain why the theoretically optimal trajectory distribution is guaranteed to be realizable by this model class.

  2. The practical training rule differs substantially from the objective for which the theoretical guarantee is established. The implementation drops several components of the theoretical objective, replaces the theoretical normalization with group-based reward normalization, removes the diffusion weighting, and trains only a subset of timesteps. These choices may be entirely reasonable in practice, but they make it unclear how much of the empirical behavior is actually explained by the formal analysis. The current wording sometimes presents the theory and the implemented algorithm as more tightly connected than the paper establishes.

  3. The appendix states that roughly half of the 15 raters are authors of the paper. Since human preference is central to the reward-hacking claim, I would like to see the results split between author and non-author raters.

Questions for authors
  1. The theorem optimizes over general trajectory distributions, while the implemented model belongs to a restricted reverse-diffusion family. Is the theoretical optimum guaranteed to be realizable by the actual policy class used in the experiments? If not, what is the intended interpretation of the theorem?

  2. Which parts of the theoretical guarantee still apply to the practical training rule after the modifications described in Section 3.3? In particular, should the theorem be understood as a motivation for DDRL, or as a guarantee for the actual algorithm evaluated in the paper?

  3. Do the human preference results remain similar when considering only the non-author raters?

  4. The main video experiments regularize on the same high-quality distribution used for SFT, and the appendix states that the evaluation set is i.i.d. with this training distribution. How sensitive is DDRL to mismatch between the regularization data and the evaluation distribution? This matters for understanding whether the method primarily prevents reward hacking or also benefits from having a matched high-quality replay distribution.

Reviewer_nBcQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a method to mitigate reward hacking in policy-gradient-based reward alignment of large-scale diffusion models by replacing conventional on-policy reverse-KL regularization with off-policy forward-KL regularization. In practice, this is implemented by incorporating a standard diffusion denoising loss on either real data or synthetic samples generated by a reference model. The authors provide a theoretical formulation intended to preserve the optimal policy distribution of conventional RL while arguing that forward-KL regularization offers greater robustness against reward hacking. Extensive experiments on large-scale video diffusion models (2B and 14B parameters), as well as text-to-image generation, demonstrate improved human preference and generation quality compared with existing diffusion RL baselines.

Critical strengths
  1. Extensive large-scale empirical evaluation with strong practical results. The paper evaluates DDRL on 2B and 14B video diffusion models across multiple reward functions and both text-to-video and image-to-video generation, with additional text-to-image experiments. The results suggest that DDRL can improve human preference and preserve perceptual quality while optimizing imperfect reward models, highlighting a practically important limitation of reward-only diffusion RL.
  2. Simple and practical implementation. The proposed approach combines reward optimization with a standard diffusion training loss, making it relatively straightforward to implement. The ability to use either real or synthetic reference data also provides flexibility when the original training dataset is unavailable.
Critical weaknesses
  1. The methodological novelty of the proposed algorithm is limited. The practical implementation essentially combines GRPO-based reward optimization with an auxiliary diffusion training loss on reference data. Similar approaches that combine diffusion RL with pretraining-data regularization to mitigate reward overoptimization and preserve generation quality have already been explored in prior work (e.g., [1]). Combining supervised training and RL is also well established in LLM post-training. Therefore, the main potential novelty of DDRL appears to lie in its forward-KL-based theoretical interpretation rather than the actual optimization algorithm.
  2. The theoretical formulation is disconnected from the practical implementation. Theorem 1 relies on a specific reward transformation, (λ(u)=−exp⁡(−u)\lambda(u)=-\exp(-u)), to recover the optimal distribution of the original RL objective. However, Section 3.3 explicitly removes this transformation and instead uses standard GRPO-style normalized advantages. Consequently, it is unclear whether the theoretical results apply to the algorithm used in the experiments. This substantially weakens the paper's theoretical contribution, which appears to be its primary methodological distinction from existing approaches.
  3. The claimed robustness of forward-KL regularization against reward hacking is not sufficiently justified. The paper attributes the failures of existing diffusion RL methods to reverse-KL regularization and argues that forward KL provides a more reliable alternative. However, forward KL's mode-covering behavior does not inherently guarantee the suppression of high-reward out-of-distribution generations. Conversely, reverse KL with an accurate reference distribution can strongly penalize such deviations. Theorem 1 establishes an optimal distribution under its assumptions, but does not demonstrate that forward KL is inherently more robust to reward hacking.
  4. The experiments do not isolate the benefits of the proposed formulation from additional supervised regularization. In the main video experiments, DDRL uses high-quality SFT videos for diffusion-loss regularization, whereas the compared RL baselines use only the corresponding prompts. This creates a potentially unfair comparison when attributing the improvements to the proposed forward-KL formulation, as DDRL benefits from additional supervised training signals that the baselines do not receive. Thus, the reported improvements may largely arise from retaining these supervised training signals rather than from the proposed formulation itself.

[1] Zhang, Yinan, et al. "Large-scale reinforcement learning for diffusion models." ECCV24.

Questions for authors
  1. [1] already combine PPO-based diffusion RL with an auxiliary pretraining diffusion loss to mitigate reward overoptimization and preserve generation quality. What is the substantive algorithmic difference between their approach and the practical DDRL implementation, beyond the use of GRPO instead of PPO?
  2. What theoretical evidence establishes that forward-KL regularization is inherently more robust to reward hacking than reverse KL? In particular, how does the proposed approach prevent high-reward OOD generations, given that forward KL primarily encourages coverage of the reference distribution?
  3. In the main video experiments, DDRL uses high-quality SFT videos that are not used for regularization by the baselines. Could the authors provide a controlled comparison in which the existing RL baselines receive the same auxiliary diffusion loss and training data, under a matched computational budget?
Reviewer_mcdjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies reward overoptimization in reinforcement learning for diffusion models, with a focus on high-resolution video generation. It argues that existing methods regularize with a reverse KL evaluated on the policy’s own samples, which cannot penalize drift into regions where the reward and reference models are unreliable. To address this, it proposes Data-regularized Diffusion Reinforcement Learning (DDRL), which combines on-policy reward optimization with a denoising loss on real or pre-generated reference data. The theoretical formulation uses a forward KL from the data-anchored noising process to the model over full diffusion trajectories, which reduces to the standard diffusion loss, together with an exponential reward transformation so that the optimum is the exponential tilted data distribution. The practical algorithm drops the exponential transform and loss weighting, yielding group-relative policy-gradient updates combined with standard diffusion training.

The evaluation covers Cosmos2.5 2B and 14B video models, VideoAlign and VBench rewards, text-to-video and image-to-video generation, and a synthetic-data experiment for OCR alignment with SD3.5-Medium. Baselines are DanceGRPO and FlowGRPO, plus an off-policy FlowGRPO variant and Diffusion-DPO in the image setting. Based on roughly 9,600 pairwise human judgments, the paper reports that DDRL is preferred over all compared methods and the base model, while baselines that reach higher rewards are often less preferred. Additional experiments examine longer training, a cheaper variant that computes the diffusion loss once per sample, and applying DDRL directly to a pretrained checkpoint in place of a separate SFT stage.

Critical strengths
  1. The paper investigates a consequential practical problem at a substantial scale. Reward improvement can conceal deteriorating generation quality, and the paper examines this issue across two video-model sizes, two reward families, and both text-to-video and image-to-video generation. Including human judgments and qualitative failure cases, rather than relying exclusively on the optimized rewards, makes the empirical investigation useful.

  2. The regularization recipe is simple and practically attractive. A diffusion loss on real or pre-generated reference samples is straightforward to integrate into existing training pipelines and avoids retaining a live reference model solely for regularization. The synthetic-data experiment and reduced-frequency regularization variant provide useful evidence about applicability when real training data or computation is constrained.

Critical weaknesses
  1. The theoretical justification does not establish the claimed properties of the evaluated algorithm. Theorem 1 relies on the exponential transformation λ(u)=−exp⁡(−u)\lambda(u)=-\exp(-u) and an exact reference-distribution log-partition constant. Section 3.3 explicitly removes this transformation, substitutes an on-policy group mean and standard deviation, and drops the KL-equivalent timestep weights wtw_t. These are substantive changes to the objective. The theorem cannot automatically justify the practical algorithm’s target-distribution or unbiasedness claims. The proof also optimizes over normalized trajectory densities without explicitly enforcing the fixed Gaussian noise marginal and reverse-transition restrictions introduced in Section 2 (see Appendix B, Eq (14)). Its realizability and noising assumptions therefore need clarification.

  2. The statistical reliability of the main human-preference results requires further validation. Human preference is the primary evidence that DDRL alleviates reward hacking. The authors provide a blinded evaluation protocol in Appendix C.1, including randomized presentation order, tie options, and evaluators unfamiliar with the paper. These are valuable safeguards. However, Table 2 and 4 report only differences in voting percentages, without per-comparison sample sizes, win/tie/loss breakdowns, or confidence intervals. Several reported advantages, particularly for the 14B model, are only 3–5 percentage points, making it difficult to assess their statistical reliability. Furthermore, approximately half of the 15 evaluators are authors. Although their participation does not invalidate the blinded evaluation, reporting results separately for non-author evaluators would help establish the robustness of the conclusions. Given that human preference is central to the paper's claims about mitigating reward hacking, these additional analyses would strengthen the empirical evidence.

Questions for authors
  1. Theoretical scope and implementation. Which guarantees are intended to hold for Algorithm 1 after removing the exponential reward transformation and changing the normalization and timestep weights? Can the authors justify this connection, or explicitly restrict Theorem 1 to the idealized objective? Please also state the assumptions under which the proposed optimal trajectory distribution is compatible with the fixed Gaussian noise marginal. When replacing reference-generated data with real data, how is the partition constant ZZ redefined?

  2. Human-evaluation reliability. Could the authors provide per-comparison sample sizes, win/tie/loss breakdowns, and confidence intervals for the main human-preference results in Table 2 and 4, accounting for repeated evaluations by the same raters and potentially repeated prompts? Could the authors also report results restricted to non-author evaluators to verify whether the conclusions remain consistent?

  3. Minor problem: Algorithm 1, line 6 uses an unsquared norm, which is likely a typo for the squared diffusion loss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.