acceptodds
Under review as a conference paper at ICLR 2027

Diffusion-ART: Efficient, Stable and Scalable Reinforcement of Diffusion Model

20 views

Abstract

Reward-guided fine-tuning adapts diffusion models to preferences and task-specific objectives beyond pretraining. Yet scaling it remains difficult: reverse-process methods such as Flow-GRPO explore locally and have poor credit assignment, limiting their efficiency, while forward-process methods such as DiffusionNFT can be unstable without carefully tuned off-policy training. We show empirically and analytically that the instability comes from a hidden forward–reverse consistency force beyond the ideal NFT-style reward guidance, and that the restoration gradient is central to the stabilizing effect of off-policy updates. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), a simple yet effective method that systematically cancels the unstable self-consistency force and provides adaptive restoration. It requires minimal hyperparameter tuning, remains effective with fewer samples per prompt, and naturally supports pairwise reward models (DiffusionART-Pairwise). Experiments spanning text-to-image, text-to-video, and image-to-video generation show that DiffusionART is highly efficient and stable. For example, on HunyuanVideo-13B text-to-video generation, DiffusionART matches the plateau training rewards of DiffusionNFT and Flow-GRPO-Fast with 11.19× and 11.23× fewer GPU hours, respectively.

Primary area:
generative models
Keywords:
Diffusion Model, Post Training, Visual Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_upsFOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies instability in diffusion reinforcement learning through an analysis of DiffusionNFT. It identifies a residual forward–reverse consistency force and examines the stabilizing role of a lagged reference. DiffusionART aggregates rewards and generated endpoints into a normalized covariance direction, adds this shared direction to cached rollout velocities, and performs two regression updates against frozen targets. A pairwise-feedback variant uses the same construction. Experiments cover text-to-image, text-to-video, and image-to-video generation, reporting improved reward optimization and training efficiency across several large diffusion models.

Critical strengths
  1. The decomposition of NFT guidance into reward-related and forward–reverse consistency components is informative. Distinguishing the effects of rollout distribution and reference anchoring provides a useful perspective on instability and motivates interventions that can be tested directly.

  2. The empirical scope is a substantial strength. Results include SD3.5-M, FLUX.1-dev, HunyuanVideo-13B, and Wan2.1-I2V-14B, including full-parameter video fine-tuning. The reported GPU-hour savings and matched reward-weight comparisons in Appendix A.5 support practical value beyond a small-model demonstration.

Critical weaknesses
  1. The theoretical analysis does not sufficiently justify the implemented guidance or the claim that restoration compensates for its bias. Lemma 2 establishes the derivative of the endpoint distribution's mean under exponential reward tilting; it does not establish that applying the same normalized vector at every reverse-process state improves expected reward. With two unequal-reward samples, the guidance reduces to a normalized winner-minus-loser latent difference, which can mix reward-relevant changes with unrelated composition or appearance differences. Appendix B.3 explicitly acknowledges the discrepancy from state-conditional guidance, but its suggestion that restoration cancels the negative effects of this bias is not established. Writing d = v_theta - v_old, the objective expands as ||d-G||^2 = ||d||^2 - 2<d,G> + constant. The restoration is therefore a quadratic proximal penalty on velocity displacement, closely related to transition-KL regularization under equal-covariance Gaussian transitions. It limits displacement without establishing that the direction is correct. The important unresolved issue is why this shared guidance, combined with ordinary local regularization, provides the claimed mechanism of reward improvement.

  2. The strongest video results lack independent quantitative validation of generation quality. Section 5.2 explicitly uses VideoReward for both optimization and evaluation. The reported acceleration to baseline plateau rewards therefore demonstrates efficient optimization of that evaluator, but does not establish an equivalent improvement in perceptual or motion quality. Appendix A.5 addresses reward-weight comparability, which I appreciate, but does not resolve this dependence. Given the large score increases and the paper's own discussion of reward hacking, an independent evaluator or a blinded human comparison is consequential to assessing whether the practical gains compensate for the incomplete theoretical justification.

Questions for authors
  1. Beyond the endpoint-mean identity, what supports applying G to every state? Can a tractable example compare the resulting parameter update with the true expected-reward gradient or measure the reward change it induces?

  2. Within ART, how do restoration strength and corrective masking separately affect stability when guidance, rollout data, and the two-update schedule are held fixed? This would help distinguish their contributions without comparing algebraically equivalent MSE and quadratic-penalty objectives.

7 comments
AuthorsOpenReview ↗

Response to Q1 and Q2

Thank you for your detailed review.

Q1. Despite the endpoint-mean identity, why can GG be applied to every state? Why can this improve expected reward?

The rationale for applying GG to every state is as follows:

  1. We first extract a shared reward-related direction from actual endpoints.

For a fixed prompt cc, consider the reward–endpoint covariance of endpoints generated by the old model:

C(c)=Covpold(R,x0∣c).C(c)=Cov_{p_{\mathrm{old}}}(R,x_0\mid c).

It combines deviations of rewards from their mean with deviations of endpoints from their mean. Intuitively, this vector points toward the weighted center of high-reward samples and away from the weighted center of low-reward samples, thereby characterizing endpoint changes associated with higher rewards in the current generation distribution.

ART estimates this covariance from endpoint samples for the same prompt and applies RMS normalization to obtain the shared direction G(c)G(c).

  1. We then transfer this endpoint-level target direction to a one-step endpoint prediction.

The direction above is defined in the space of actual endpoints, whereas the model must be trained at intermediate states. To connect the two, we use the one-step endpoint prediction at each state as a surrogate and take G(c)G(c) as the direction in which this prediction should move.

We neither re-estimate the reward covariance of the one-step prediction nor assume that it equals the actual endpoint covariance. Instead, we convert the reward-related direction extracted from endpoint samples into endpoint-prediction supervision that can be used at every state.

This approximation allows the algorithm to reuse reward information from existing endpoints without separately estimating posterior guidance at each intermediate state.

  1. Finally, we use the linear relationship between the one-step endpoint prediction and velocity to construct directionally consistent regression targets.

At fixed state st=(xt,t,c)s _ t=(x _ t,t,c), x^0=xt+tv⟹Δx^0=tΔv.\widehat{x} _ 0=x _ t+t v \quad\Longrightarrow\quad \Delta\widehat{x} _ 0=t\Delta v.

With rollout-cached velocity prediction vθ(0)(st)v _ \theta^{(0)}(s _ t), ART constructs regression targets

vtgtART(st)=vθ(0)(st)+G(c),v _ {\mathrm{tgt}}^{\mathrm{ART}}(s _ t)=v _ \theta^{(0)}(s _ t)+G(c),

x^0,tgt(st)=x^0(0)(st)+tG(c),\widehat{x} _ {0,\mathrm{tgt}}(s _ t)=\widehat{x} _ 0^{(0)}(s _ t)+tG(c),

where x^0(0)(st)=xt+tvθ(0)(st)\widehat{x} _ 0^{(0)}(s _ t)=x _ t+t v _ \theta^{(0)}(s _ t). For 0<t≤10<t\leq1, the velocity and endpoint-target shifts have the same direction. Despite different positions and velocity predictions, all states receive supervision in the same endpoint-space direction.


Q2. Can the restoration term really correct the bias in global covariance, especially when K=2K=2?

A2. Shared guidance does introduce bias. This bias arises because the posterior distribution is difficult to access, so we use marginal covariance to approximate posterior covariance. It causes some training instability: as shown in Figure 6(b), ART exhibits a reward drop late in training without two-update restoration. Introducing two-update restoration, however, enables efficient and stable training.

Regarding whether the restoration mechanism can actually correct the direction, Figure 6(c) shows that ART can indeed maintain efficient and stable training with two-update restoration. From a theoretical perspective, the restoration mechanism can indeed correct the bias introduced by marginal covariance. As shown in Equation (6), ART involves two updates. The first update is driven solely by shared guidance, L0=2w/DG(c)L^{0} = 2w/D G(c), without a restoration force. The second update is L(1)=2w/D(G(c)−d(st))L^{(1)}=2w/D(G(c) - d(s_t)) and introduces the restoration force −d(st)=vθ(0)(st)−vθ(st)-d(s_t) = v_{\theta}^{(0)}(s_t) - v_{\theta}(s_t). This force grows with the displacement caused by the previous update. It can therefore correct the bias in shared guidance and the resulting training instability.

AuthorsOpenReview ↗

Response to Q3 and Q4

Q3. Do the video results demonstrate quality improvements beyond optimizing VideoReward itself?

A3. Based on the visual results in Figures 16, 17, and 18, we conclude that visual quality has indeed improved. More importantly, however, we view improvements in reinforcement learning algorithms and reward-model accuracy as two independent dimensions, and our focus is on improving the efficiency and stability of reinforcement learning algorithms.

More efficient optimization may exploit imperfect reward models more quickly, while open-source reward models for visual generation remain less than satisfactory. With better reward models, our algorithm would deliver better perceptual quality.


Q4. Can you separately isolate the effects of restoration strength and the correction mask while keeping the guidance and the two-update schedule fixed?

A4. The revised manuscript separately examines the effect of enabling versus disabling the mask while keeping the same covariance-guidance construction, timestep weighting, and two non-overlapping MSE updates (Figure 8, p. 10, lines 486–491). Video training without the mask exhibits a drop around step 200 and then recovers, whereas training with the mask improves steadily. Image training remains effective without the mask (Figure 12, p. 16, lines 812–816). Overall, we consider the trust-region mask a useful technique for further stabilizing DiffusionART.


We hope that all your concerns have been fully addressed and would be happy to discuss any further questions.

Thank you for the authors' responses and additional experiments. While I acknowledge the empirical improvements and the additional ablation results, my main concerns remain unresolved.

1. Insufficient justification of the core mechanisms (Q1).

The proposed method remains largely heuristic. The authors explain that applying a shared guidance direction to velocity predictions produces consistent shifts in endpoint predictions. However, this algebraic relationship does not justify why a global reward–endpoint covariance direction should be appropriate for every intermediate state, nor does it establish that such updates improve expected rewards.

The response clarifies how the method operates, but does not adequately justify why the proposed guidance should be effective.

2. Unconvincing explanation of the restoration mechanism (Q2).

The restoration term primarily constrains deviations from the old model, rather than correcting the bias introduced by global guidance. In particular, the restoring force does not provide information about the true state-dependent guidance direction. Therefore, improved training stability does not demonstrate that the proposed mechanism actually corrects guidance bias.

Furthermore, the reliability of covariance estimation with a small group size (e.g., K=2) remains insufficiently examined.

3. Insufficient validation of actual optimization effectiveness (Q3).

This remains my most significant concern.

My original question was whether improvements measured by VideoReward, which is also used for training, genuinely reflect better generation quality. However, the authors respond by

We view improvements in reinforcement learning algorithms and reward-model accuracy as two independent dimensions,

I find this response particularly concerning. Optimization efficiency and reward-model reliability cannot be treated as independent when assessing the actual effectiveness of an RL algorithm. Faster convergence to a poor or spurious local optimum is of little value, regardless of how efficiently the proxy reward is optimized.

Reward hacking is a widely recognized and extensively studied challenge in generative-model RL, including diffusion-based image generation. Numerous works, such as GRPO-Guard (CVPR 2026), AdaEnt (ICCV 2025), Aligning Text-to-Image Diffusion Models With Constrained Reinforcement Learning (TPAMI 2025), and Understanding Reward Hacking in Text-to-Image Reinforcement Learning (CVPR Findings 2026), have explicitly investigated reward overoptimization and its relationship with RL optimization mechanisms. These studies demonstrate that reward hacking is not merely an issue of reward-model accuracy, but is also closely intertwined with optimization dynamics and algorithmic design.

The reported optimization advantages may arise from the algorithm exploiting spurious shortcuts in the reward model, rather than learning a genuinely more effective optimization strategy. In such cases, higher proxy rewards may coexist with inferior results under independent evaluation metrics or actual perceptual quality.

I do not expect this work to solve reward hacking or achieve state-of-the-art performance across all metrics. However, I would expect a paper proposing a new RL optimization algorithm to recognize and critically examine whether its reported improvements reflect genuine gains or reward exploitation. The provided qualitative examples and evaluation using the training reward model are insufficient to establish this distinction.

4. Additional ablations (Q4).

I acknowledge that the revised manuscript provides a more controlled ablation of the correction mask, which partially addresses my concern. However, the effect of restoration strength and the specific contribution of the restoration mechanism remain insufficiently isolated.

Overall Assessment

Overall, I find the proposed method largely heuristic, with insufficient justification of its core mechanisms and inadequate validation of its actual effectiveness.

The paper demonstrates improvements in proxy reward optimization, but does not convincingly establish whether these gains translate into meaningful improvements in generation quality. More importantly, the authors' response to Q3 has further weakened my confidence in how they interpret and validate their empirical results.

I do not object to heuristic methods in principle, but when theoretical justification is limited, rigorous empirical validation becomes particularly important. Unfortunately, I find the evidence insufficient on both fronts.

Given these unresolved concerns, particularly those reinforced by the rebuttal, I am inclined to lower my original rating.

Reviewer_UFsdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DiffusionART, an online reward optimization method for diffusion and flow models. It first analyzes the discrepancy between the reverse sampling field and the ideal forward velocity field induced by generated endpoints, showing how this discrepancy enters the DiffusionNFT update. It then constructs a prompt-level guidance vector from the covariance between endpoint samples and rewards, applies this vector to cached rollout predictions, and performs two updates toward frozen targets to introduce a restoring effect without an EMA rollout policy. Experiments cover text-to-image, text-to-video, image-to-video, and simulated pairwise feedback.

Critical strengths
  1. The distinction between ideal forward velocities and the actual reverse sampling field is useful.
  2. The empirical evaluation has substantial breadth and includes meaningful controls.
Critical weaknesses
  1. The theoretical analysis does not fully justify the shared guidance update. Lemma 2 establishes how reward tilting changes the endpoint mean, but does not directly establish that applying this direction to every intermediate state improves expected reward. The authors acknowledge that prompt-level covariance is a biased substitute for state-conditional covariance. Its applicability and failure modes remain insufficiently characterized, and restoration limits displacement without necessarily correcting directional bias.

  2. Video quality and objective trade-offs are not fully evaluated. VideoReward supplies both training rewards and evaluation metrics for text-to-video, so the results establish reward optimization more directly than independent perceptual improvement. The authors also acknowledge that optimizing VQ/MQ can reduce text alignment and adjust reward weights accordingly, yet the main final-results table does not report quantitative TA. This leaves the quality–alignment trade-off unclear. Independent video evaluation or blinded human assessment would strengthen the conclusions.

Questions for authors
  1. Lemma 2 establishes a change in the endpoint mean under reward tilting. What additional justification supports applying the same covariance direction to all intermediate states? Is this step supported by further theoretical conditions, or primarily by empirical results? Clarifying this would help me assess the scope of the theoretical claims.

  2. The paper notes that optimizing VQ/MQ can reduce text alignment. Do you have quantitative TA results for the main and matched-weight comparisons? Is there any independent video evaluation or human assessment supporting the quality gains beyond VideoReward? These results would help determine whether the improvements preserve text alignment and reflect broader video quality.

3 comments
AuthorsOpenReview ↗

Response to Q1

Q1. Beyond the endpoint-mean identity, what supports applying shared covariance guidance to every state?

A1. The rationale for applying GG to every state is as follows:

  1. We first extract a shared reward-related direction from actual endpoints.

For a fixed prompt cc, consider the reward–endpoint covariance of endpoints generated by the old model:

C(c)=Covpold(R,x0∣c).C(c)=Cov_{p_{\mathrm{old}}}(R,x_0\mid c).

It combines deviations of rewards from their mean with deviations of endpoints from their mean. Intuitively, this vector points toward the weighted center of high-reward samples and away from the weighted center of low-reward samples, thereby characterizing endpoint changes associated with higher rewards in the current generation distribution.

ART estimates this covariance from endpoint samples for the same prompt and applies RMS normalization to obtain the shared direction G(c)G(c).

  1. We then transfer this endpoint-level target direction to a one-step endpoint prediction.

The direction above is defined in the space of actual endpoints, whereas the model must be trained at intermediate states. To connect the two, we use the one-step endpoint prediction at each state as a surrogate and take G(c)G(c) as the direction in which this prediction should move.

We neither re-estimate the reward covariance of the one-step prediction nor assume that it equals the actual endpoint covariance. Instead, we convert the reward-related direction extracted from endpoint samples into endpoint-prediction supervision that can be used at every state.

This approximation allows the algorithm to reuse reward information from existing endpoints without separately estimating posterior guidance at each intermediate state.

  1. Finally, we use the linear relationship between the one-step endpoint prediction and velocity to construct directionally consistent regression targets.

At fixed state st=(xt,t,c)s _ t=(x _ t,t,c), x^0=xt+tv⟹Δx^0=tΔv.\widehat{x} _ 0=x _ t+t v \quad\Longrightarrow\quad \Delta\widehat{x} _ 0=t\Delta v.

With rollout-cached velocity prediction vθ(0)(st)v _ \theta^{(0)}(s _ t), ART constructs regression targets

vtgtART(st)=vθ(0)(st)+G(c),v _ {\mathrm{tgt}}^{\mathrm{ART}}(s _ t)=v _ \theta^{(0)}(s _ t)+G(c),

x^0,tgt(st)=x^0(0)(st)+tG(c),\widehat{x} _ {0,\mathrm{tgt}}(s _ t)=\widehat{x} _ 0^{(0)}(s _ t)+tG(c),

where x^0(0)(st)=xt+tvθ(0)(st)\widehat{x} _ 0^{(0)}(s _ t)=x _ t+t v _ \theta^{(0)}(s _ t). For 0<t≤10<t\leq1, the velocity and endpoint-target shifts have the same direction. Despite different positions and velocity predictions, all states receive supervision in the same endpoint-space direction.

AuthorsOpenReview ↗

Response to Q2 and Q3

Q2. What bias might shared guidance introduce? Can the restoration term really correct its direction?

A2. Shared guidance does introduce bias. This bias arises because the posterior distribution is difficult to access, so we use marginal covariance to approximate posterior covariance. It causes some training instability: as shown in Figure 6(b), ART exhibits a reward drop late in training without two-update restoration. Introducing two-update restoration, however, enables efficient and stable training.

Regarding whether the restoration mechanism can actually correct the direction, Figure 6(c) shows that ART can indeed maintain efficient and stable training with two-update restoration. From a theoretical perspective, the restoration mechanism can indeed correct the bias introduced by marginal covariance. As shown in Equation (6), ART involves two updates. The first update is driven solely by shared guidance, L0=2w/DG(c)L^{0} = 2w/D G(c), without a restoration force. The second update is L(1)=2w/D(G(c)−d(st))L^{(1)}=2w/D(G(c) - d(s_t)) and introduces the restoration force −d(st)=vθ(0)(st)−vθ(st)-d(s_t) = v_{\theta}^{(0)}(s_t) - v_{\theta}(s_t). This force grows with the displacement caused by the previous update. It can therefore correct the bias in shared guidance and the resulting training instability.


Q3. What trade-offs arise in text alignment? Have the video gains been independently validated?

A3. The revised manuscript now reports TA, VQ, and MQ together under matched weights of TA:VQ:MQ=7:2:1{TA}:{VQ}:{MQ}=7:2:1 in Table 3 (p. 14, lines 714–726):

MethodVQ ↑MQ ↑TA ↑
Pretrained model5.1951.9431.118
Flow-GRPO-Fast3.1491.8741.667
DiffusionNFT (EMA)6.8272.6832.096
DiffusionART8.9704.2242.111

Figure 7(a–c) also shows all three training curves (p. 9, lines 459–485). These results show that, under this evaluator, ART achieves higher VQ/MQ while its TA is comparable to NFT's. The comparison with VQ:MQ=1:1{VQ}:{MQ}=1:1 in Appendix A.5 neither includes TA in the optimization objective nor reports a corresponding quantitative TA comparison; we do not claim that this setting preserves text alignment.

VideoReward is used for both optimization and evaluation. We therefore interpret these gains as efficient and stable reward optimization, rather than independent evidence of improved perceptual quality. Faster optimization may also accelerate the exploitation of weaknesses in the reward model. We view improvements in reinforcement learning algorithms and reward-model accuracy as two independent dimensions, and our focus is on improving the efficiency and stability of reinforcement learning algorithms.


Thank you again for your constructive comments. We hope that all your concerns have been addressed.

Area_Chair_m23YOpenReview ↗

rebuttal response

The rebuttal period is quickly approaching the end. Please go over the authors’ rebuttal and revision, and add acknowledgment and further comments if needed.

Reviewer_og7TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzed DiffusionNFT, identifying a hidden consistency force(Proposition 1) and stabilizing role of EMA based restoraiton(Proposition2). Based on these insights, aythors propse DiffusionART, which uses reward–endpoint covariance for velocity guidance and two updates for restoration without EMA tuning. Experiments demonstrate promising efficiency and performance, espicially in video generation.

Critical strengths
  1. Insightful analysis of DiffusionNFT: In section 3, authors investigate the roles of joint positive–negative training and EMA through theoretical analysis and empirical ablations.

  2. Strong video-generation results: DiffusionART demonstrates impressive training efficiency and reward improvements espicially on large-scale text-to-video and image-to-video tasks.

Critical weaknesses
  1. Instability analysis: In Section 3.1, the authors identify the consistency force as a key source of training instability of DiffusionNFT. However, the results in Figure 2(a) do not directly establish a causal relationship between the consistency force and the instability.

  2. Misalignment between motivation and method: One of the motivation of the paper, stated in the abstract and introduction, is the poor credit assignment of Flow-GRPO. However, it remains unclear how DiffusionART addresses this issue, as the method applies the same prompt-level guidance direction to all reverse-process states.

Questions for authors
  1. Consistency force (Section 3.1): To better support the claim that the consistency force causes training instability, could the authors provide an ablation controlling its contribution in Eq. (3), e.g. Fλ=2g+λ(2q−1)eF_\lambda=2g+\lambda(2q-1)e?

  2. Credit assignment (Section 4.1): One of motivations is the poor credit assignment of Flow-GRPO. How does DiffusionART address this issue? In particular, what justifies applying the same guidance direction to all reverse-process states?

  3. Training efficiency (Figure 5): Since training efficiency is a key contribution in the paper, could the authors additionally report reward curves against GPU hours or the number of reward evaluations, rather than only training steps?

  4. Reward over-optimization (Figure 5): An important goal of reward alignment is to improve reward while avoiding reward over-optimization. Could the authors report diversity metrics in Figure 5 to assess whether reward over-optimization occurred?

8 comments
AuthorsOpenReview ↗

Response tp Q1

Thank you for your constructive comments.

Q1. What evidence connects the consistency force to NFT's instability?

A1. We agree that Figure 2(a) alone cannot establish causality. We added a goodness-center ablation in Figure 2(c). In DiffusionNFT,

ri:=r(x0(i),c)=12[1+clip⁡(Ri−RˉcZc,−1,1)]∈[0,1].r_i:=r(x_0^{(i)},c)=\frac{1}{2}\left[1+\operatorname{clip}\left(\frac{R_i-\bar R_c}{Z_c},-1,1\right)\right]\in[0,1].

Shifting the original center from 0.50 to 0.49 or 0.51 sharply reduces reward, while training at 0.50 remains stable.

Proposition 1 decomposes NFT's force as 2g+(2q−1)e2g+(2q-1)e, where

g&=\operatorname{Cov}(r,u_t\mid s_t),\quad q=\mathbb{E}[r\mid s_t], e=v_{\mathrm{fwd}}^{\mathrm{old}}(s_t)-v_{\mathrm{rvs}}^{\mathrm{old}}(s_t). \end{aligned}$$ Ignoring clipping at the $[0,1]$ boundaries, a uniform shift $r\mapsto r+\delta$ leaves $g$ unchanged and sends $q$ to $q+\delta$. Thus, reward guidance $2g$ is unchanged, while the consistency force gains $2\delta e$. Together with Figure 2(c), this suggests that even a small increase in consistency force can destabilize NFT and is a key factor in its instability.
AuthorsOpenReview ↗

Response to Q2-Q4

Q2. Why apply shared guidance to every state? How does this address Flow-GRPO's credit-assignment problem?

A2. Flow-GRPO gives every stochastic transition the same terminal advantage, so an action can receive a strong signal for rewards largely due to its inherited state rather than its own contribution. This is its credit-assignment problem.

ART instead constructs shared endpoint-prediction supervision:

  1. For each prompt cc, estimate $ \begin{aligned} C(c)=Cov_{p_{\mathrm{old}}}(R,x_0\mid c). \end{aligned} $ from actual endpoints sampled by the old model, then RMS-normalize it to obtain G(c)G(c). This covariance combines centered rewards and endpoints: intuitively, it points toward the weighted center of high-reward samples and away from that of low-reward samples, capturing endpoint changes associated with higher rewards under the current generation distribution.

  2. At each intermediate state, use the one-step endpoint prediction x^0\widehat{x}_0 as a surrogate and G(c)G(c) as its target direction. This transfers actual-endpoint reward information into per-state supervision; it neither re-estimates the prediction's reward covariance nor assumes equality with the actual-endpoint covariance. This approximation reuses sampled rewards without separately estimating posterior guidance at each state.

  3. At fixed state st=(xt,t,c)s _ t=(x _ t,t,c), x^0=xt+tv⟹Δx^0=tΔv.\widehat{x} _ 0=x _ t+t v \quad\Longrightarrow\quad \Delta\widehat{x} _ 0=t\Delta v.

With rollout-cached velocity prediction vθ(0)(st)v _ \theta^{(0)}(s _ t), ART constructs regression targets

vtgtART(st)=vθ(0)(st)+G(c),v _ {\mathrm{tgt}}^{\mathrm{ART}}(s _ t)=v _ \theta^{(0)}(s _ t)+G(c),

x^0,tgt(st)=x^0(0)(st)+tG(c),\widehat{x} _ {0,\mathrm{tgt}}(s _ t)=\widehat{x} _ 0^{(0)}(s _ t)+tG(c),

where x^0(0)(st)=xt+tvθ(0)(st)\widehat{x} _ 0^{(0)}(s _ t)=x _ t+t v _ \theta^{(0)}(s _ t). For 0<t≤10<t\leq1, the velocity and endpoint-target shifts have the same direction. Despite different positions and velocity predictions, all states receive supervision in the same endpoint-space direction.


Q3. Could you report reward curves against GPU-hours rather than only training steps?

A3. We added Figure 15 (Appendix A.10, p. 18, lines 929–931), replotting all five Figure 7 panels against GPU-hours with the same runs and reward weights: TA:VQ:MQ=7:2:1\mathrm{TA}:\mathrm{VQ}:\mathrm{MQ}=7:2:1 (T2V) and VQ:MQ=1:1\mathrm{VQ}:\mathrm{MQ}=1:1 (I2V). The additional T2V VQ:MQ=1:1\mathrm{VQ}:\mathrm{MQ}=1:1 comparison remains in Figure 10 (p. 15, lines 780–781).


Q4. Does faster reward optimization reduce generation diversity?

A4. We measured within-prompt LPIPS diversity on 200 DrawBench prompts with 8 matched initial noise seeds per prompt across Base, NFT, and ART (512×512512\times512, 40 sampling steps, CFG=1\mathrm{CFG}=1). We average LPIPS-AlexNet over each prompt's 28 image pairs, then equally across prompts. LPIPS (Learned Perceptual Image Patch Similarity; Zhang et al., 2018) measures perceptual distance via deep features calibrated to human judgments; higher within-prompt mean pairwise LPIPS indicates more variation across seeds, not better image quality.

ModelLPIPS ↑Decrease vs. base
Base SD3.5-M0.8303—
DiffusionNFT0.510438.52%
DiffusionART0.534435.64%

Both fine-tuned checkpoints show lower LPIPS diversity, with a slightly smaller decrease for ART. Thus, in this checkpoint comparison, ART shows no greater LPIPS reduction. We believe all RL algorithms align models toward higher rewards, inevitably reducing diversity.


We hope this addresses your concerns and welcome further questions.

Thank you for the clarification. I have two follow-up question:

  1. I understand that G(c)G(c) represents the mean-shift direction under reward tilting. However, does applying the shared mean-shift guidance G(c)G(c) to all reverse-process states guarantee an improvement in expected reward?
  2. Could the authors also provide the results in Fig 5 with GPU hours or the number of reward evaluations on the x-axis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.