- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies instability in diffusion reinforcement learning through an analysis of DiffusionNFT. It identifies a residual forward–reverse consistency force and examines the stabilizing role of a lagged reference. DiffusionART aggregates rewards and generated endpoints into a normalized covariance direction, adds this shared direction to cached rollout velocities, and performs two regression updates against frozen targets. A pairwise-feedback variant uses the same construction. Experiments cover text-to-image, text-to-video, and image-to-video generation, reporting improved reward optimization and training efficiency across several large diffusion models.
Critical strengths
-
The decomposition of NFT guidance into reward-related and forward–reverse consistency components is informative. Distinguishing the effects of rollout distribution and reference anchoring provides a useful perspective on instability and motivates interventions that can be tested directly.
-
The empirical scope is a substantial strength. Results include SD3.5-M, FLUX.1-dev, HunyuanVideo-13B, and Wan2.1-I2V-14B, including full-parameter video fine-tuning. The reported GPU-hour savings and matched reward-weight comparisons in Appendix A.5 support practical value beyond a small-model demonstration.
Critical weaknesses
-
The theoretical analysis does not sufficiently justify the implemented guidance or the claim that restoration compensates for its bias. Lemma 2 establishes the derivative of the endpoint distribution's mean under exponential reward tilting; it does not establish that applying the same normalized vector at every reverse-process state improves expected reward. With two unequal-reward samples, the guidance reduces to a normalized winner-minus-loser latent difference, which can mix reward-relevant changes with unrelated composition or appearance differences. Appendix B.3 explicitly acknowledges the discrepancy from state-conditional guidance, but its suggestion that restoration cancels the negative effects of this bias is not established. Writing d = v_theta - v_old, the objective expands as ||d-G||^2 = ||d||^2 - 2<d,G> + constant. The restoration is therefore a quadratic proximal penalty on velocity displacement, closely related to transition-KL regularization under equal-covariance Gaussian transitions. It limits displacement without establishing that the direction is correct. The important unresolved issue is why this shared guidance, combined with ordinary local regularization, provides the claimed mechanism of reward improvement.
-
The strongest video results lack independent quantitative validation of generation quality. Section 5.2 explicitly uses VideoReward for both optimization and evaluation. The reported acceleration to baseline plateau rewards therefore demonstrates efficient optimization of that evaluator, but does not establish an equivalent improvement in perceptual or motion quality. Appendix A.5 addresses reward-weight comparability, which I appreciate, but does not resolve this dependence. Given the large score increases and the paper's own discussion of reward hacking, an independent evaluator or a blinded human comparison is consequential to assessing whether the practical gains compensate for the incomplete theoretical justification.
Questions for authors
-
Beyond the endpoint-mean identity, what supports applying G to every state? Can a tractable example compare the resulting parameter update with the true expected-reward gradient or measure the reward change it induces?
-
Within ART, how do restoration strength and corrective masking separately affect stability when guidance, rollout data, and the two-update schedule are held fixed? This would help distinguish their contributions without comparing algebraically equivalent MSE and quadratic-penalty objectives.
7 comments
Response to Q3 and Q4
Q3. Do the video results demonstrate quality improvements beyond optimizing VideoReward itself?
A3. Based on the visual results in Figures 16, 17, and 18, we conclude that visual quality has indeed improved. More importantly, however, we view improvements in reinforcement learning algorithms and reward-model accuracy as two independent dimensions, and our focus is on improving the efficiency and stability of reinforcement learning algorithms.
More efficient optimization may exploit imperfect reward models more quickly, while open-source reward models for visual generation remain less than satisfactory. With better reward models, our algorithm would deliver better perceptual quality.
Q4. Can you separately isolate the effects of restoration strength and the correction mask while keeping the guidance and the two-update schedule fixed?
A4. The revised manuscript separately examines the effect of enabling versus disabling the mask while keeping the same covariance-guidance construction, timestep weighting, and two non-overlapping MSE updates (Figure 8, p. 10, lines 486–491). Video training without the mask exhibits a drop around step 200 and then recovers, whereas training with the mask improves steadily. Image training remains effective without the mask (Figure 12, p. 16, lines 812–816). Overall, we consider the trust-region mask a useful technique for further stabilizing DiffusionART.
We hope that all your concerns have been fully addressed and would be happy to discuss any further questions.
Thank you for the authors' responses and additional experiments. While I acknowledge the empirical improvements and the additional ablation results, my main concerns remain unresolved.
1. Insufficient justification of the core mechanisms (Q1).
The proposed method remains largely heuristic. The authors explain that applying a shared guidance direction to velocity predictions produces consistent shifts in endpoint predictions. However, this algebraic relationship does not justify why a global reward–endpoint covariance direction should be appropriate for every intermediate state, nor does it establish that such updates improve expected rewards.
The response clarifies how the method operates, but does not adequately justify why the proposed guidance should be effective.
2. Unconvincing explanation of the restoration mechanism (Q2).
The restoration term primarily constrains deviations from the old model, rather than correcting the bias introduced by global guidance. In particular, the restoring force does not provide information about the true state-dependent guidance direction. Therefore, improved training stability does not demonstrate that the proposed mechanism actually corrects guidance bias.
Furthermore, the reliability of covariance estimation with a small group size (e.g., K=2) remains insufficiently examined.
3. Insufficient validation of actual optimization effectiveness (Q3).
This remains my most significant concern.
My original question was whether improvements measured by VideoReward, which is also used for training, genuinely reflect better generation quality. However, the authors respond by
We view improvements in reinforcement learning algorithms and reward-model accuracy as two independent dimensions,
I find this response particularly concerning. Optimization efficiency and reward-model reliability cannot be treated as independent when assessing the actual effectiveness of an RL algorithm. Faster convergence to a poor or spurious local optimum is of little value, regardless of how efficiently the proxy reward is optimized.
Reward hacking is a widely recognized and extensively studied challenge in generative-model RL, including diffusion-based image generation. Numerous works, such as GRPO-Guard (CVPR 2026), AdaEnt (ICCV 2025), Aligning Text-to-Image Diffusion Models With Constrained Reinforcement Learning (TPAMI 2025), and Understanding Reward Hacking in Text-to-Image Reinforcement Learning (CVPR Findings 2026), have explicitly investigated reward overoptimization and its relationship with RL optimization mechanisms. These studies demonstrate that reward hacking is not merely an issue of reward-model accuracy, but is also closely intertwined with optimization dynamics and algorithmic design.
The reported optimization advantages may arise from the algorithm exploiting spurious shortcuts in the reward model, rather than learning a genuinely more effective optimization strategy. In such cases, higher proxy rewards may coexist with inferior results under independent evaluation metrics or actual perceptual quality.
I do not expect this work to solve reward hacking or achieve state-of-the-art performance across all metrics. However, I would expect a paper proposing a new RL optimization algorithm to recognize and critically examine whether its reported improvements reflect genuine gains or reward exploitation. The provided qualitative examples and evaluation using the training reward model are insufficient to establish this distinction.
4. Additional ablations (Q4).
I acknowledge that the revised manuscript provides a more controlled ablation of the correction mask, which partially addresses my concern. However, the effect of restoration strength and the specific contribution of the restoration mechanism remain insufficiently isolated.
Overall Assessment
Overall, I find the proposed method largely heuristic, with insufficient justification of its core mechanisms and inadequate validation of its actual effectiveness.
The paper demonstrates improvements in proxy reward optimization, but does not convincingly establish whether these gains translate into meaningful improvements in generation quality. More importantly, the authors' response to Q3 has further weakened my confidence in how they interpret and validate their empirical results.
I do not object to heuristic methods in principle, but when theoretical justification is limited, rigorous empirical validation becomes particularly important. Unfortunately, I find the evidence insufficient on both fronts.
Given these unresolved concerns, particularly those reinforced by the rebuttal, I am inclined to lower my original rating.
Response to Q1 and Q2
Thank you for your detailed review.
Q1. Despite the endpoint-mean identity, why can G be applied to every state? Why can this improve expected reward?
The rationale for applying G to every state is as follows:
For a fixed prompt c, consider the reward–endpoint covariance of endpoints generated by the old model:
C(c)=Covpold(R,x0∣c).
It combines deviations of rewards from their mean with deviations of endpoints from their mean. Intuitively, this vector points toward the weighted center of high-reward samples and away from the weighted center of low-reward samples, thereby characterizing endpoint changes associated with higher rewards in the current generation distribution.
ART estimates this covariance from endpoint samples for the same prompt and applies RMS normalization to obtain the shared direction G(c).
The direction above is defined in the space of actual endpoints, whereas the model must be trained at intermediate states. To connect the two, we use the one-step endpoint prediction at each state as a surrogate and take G(c) as the direction in which this prediction should move.
We neither re-estimate the reward covariance of the one-step prediction nor assume that it equals the actual endpoint covariance. Instead, we convert the reward-related direction extracted from endpoint samples into endpoint-prediction supervision that can be used at every state.
This approximation allows the algorithm to reuse reward information from existing endpoints without separately estimating posterior guidance at each intermediate state.
At fixed state st=(xt,t,c), x0=xt+tv⟹Δx0=tΔv.
With rollout-cached velocity prediction vθ(0)(st), ART constructs regression targets
vtgtART(st)=vθ(0)(st)+G(c),
x0,tgt(st)=x0(0)(st)+tG(c),
where x0(0)(st)=xt+tvθ(0)(st). For 0<t≤1, the velocity and endpoint-target shifts have the same direction. Despite different positions and velocity predictions, all states receive supervision in the same endpoint-space direction.
Q2. Can the restoration term really correct the bias in global covariance, especially when K=2?
A2. Shared guidance does introduce bias. This bias arises because the posterior distribution is difficult to access, so we use marginal covariance to approximate posterior covariance. It causes some training instability: as shown in Figure 6(b), ART exhibits a reward drop late in training without two-update restoration. Introducing two-update restoration, however, enables efficient and stable training.
Regarding whether the restoration mechanism can actually correct the direction, Figure 6(c) shows that ART can indeed maintain efficient and stable training with two-update restoration. From a theoretical perspective, the restoration mechanism can indeed correct the bias introduced by marginal covariance. As shown in Equation (6), ART involves two updates. The first update is driven solely by shared guidance, L0=2w/DG(c), without a restoration force. The second update is L(1)=2w/D(G(c)−d(st)) and introduces the restoration force −d(st)=vθ(0)(st)−vθ(st). This force grows with the displacement caused by the previous update. It can therefore correct the bias in shared guidance and the resulting training instability.