- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies reward hacking in reinforcement learning for diffusion models. The authors argue that existing diffusion RL methods can become unreliable because their regularization is evaluated on samples from the current policy, precisely where the reward model and reference policy may be least trustworthy after aggressive optimization. The proposed method, DDRL, instead anchors the model to an off-policy data distribution. The paper connects this idea to forward-KL regularization and shows that, under its theoretical formulation, the regularizer can be implemented using the standard diffusion training loss. In practice, the resulting algorithm combines reward optimization on policy rollouts with diffusion training on either real or synthetic reference data. The method is evaluated on Cosmos2.5 2B and 14B models for text-to-video and image-to-video generation, with VideoAlign and VBench as optimization targets. The authors also report roughly 9,600 pairwise human judgments and additional SD3.5 text-to-image experiments using synthetic regularization data. Across the tested settings, DDRL generally improves reward while avoiding much of the preference degradation and visual artifacts observed with DanceGRPO and FlowGRPO. The paper also studies a reduced-cost version of DDRL and explores combining SFT and RL into a single post-training stage.
Critical strengths
-
The paper addresses an important practical failure mode with unusually strong empirical evidence. Reward hacking is particularly difficult to evaluate using the optimized reward itself, so the large-scale human comparisons are important here. The video experiments cover two model scales, two generation settings, and two reward functions, and the qualitative and preference results consistently show the same failure pattern for the competing RL methods. This makes the empirical case much stronger than a standard reward-only comparison.
-
The proposed regularization principle is simple and potentially quite useful. The connection between forward-KL regularization against an off-policy data distribution and the standard diffusion training loss gives a clean interpretation of why replaying high-quality data can prevent reward optimization from drifting away from the data manifold. I also appreciate that the method does not require keeping a reference model in memory during training, and that the paper shows a reduced-cost implementation with comparable performance.
Critical weaknesses
-
The theoretical optimum is derived over a more general distribution class than the actual diffusion policy appears to support. The proof treats the full trajectory distribution as the optimization variable and derives an ideal reward-tilted solution. However, the actual model is restricted to a particular reverse diffusion parameterization with a fixed terminal prior and constrained transition family. The paper does not explain why the theoretically optimal trajectory distribution is guaranteed to be realizable by this model class.
-
The practical training rule differs substantially from the objective for which the theoretical guarantee is established. The implementation drops several components of the theoretical objective, replaces the theoretical normalization with group-based reward normalization, removes the diffusion weighting, and trains only a subset of timesteps. These choices may be entirely reasonable in practice, but they make it unclear how much of the empirical behavior is actually explained by the formal analysis. The current wording sometimes presents the theory and the implemented algorithm as more tightly connected than the paper establishes.
-
The appendix states that roughly half of the 15 raters are authors of the paper. Since human preference is central to the reward-hacking claim, I would like to see the results split between author and non-author raters.
Questions for authors
-
The theorem optimizes over general trajectory distributions, while the implemented model belongs to a restricted reverse-diffusion family. Is the theoretical optimum guaranteed to be realizable by the actual policy class used in the experiments? If not, what is the intended interpretation of the theorem?
-
Which parts of the theoretical guarantee still apply to the practical training rule after the modifications described in Section 3.3? In particular, should the theorem be understood as a motivation for DDRL, or as a guarantee for the actual algorithm evaluated in the paper?
-
Do the human preference results remain similar when considering only the non-author raters?
-
The main video experiments regularize on the same high-quality distribution used for SFT, and the appendix states that the evaluation set is i.i.d. with this training distribution. How sensitive is DDRL to mismatch between the regularization data and the evaluation distribution? This matters for understanding whether the method primarily prevents reward hacking or also benefits from having a matched high-quality replay distribution.