acceptodds
Under review as a conference paper at ICLR 2027

Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

Abstract

Reward-based fine-tuning of diffusion models has largely inherited likelihood-based policy optimization developed for autoregressive large language models (LLMs). Diffusion models, however, are natively trained through velocity regression and do not directly provide the likelihood of a generated sample. Existing methods address this using transition likelihoods along stochastic denoising trajectories or evidence lower bounds (ELBOs) to approximate generated-sample likelihoods. These approximations arise from applying likelihood-based updates to models whose native training and generation operate through velocity fields. We instead take velocity matching as the starting point for reward fine-tuning. We propose **reward-based velocity matching (RVM)**, which weights the velocity-matching loss by reward with an additional anchor regression term, requiring neither likelihood estimation nor likelihood ratios. RVM recovers Reinforce Adjoint Matching at the update level and contains DiffusionNFT as a special case, while ELBO-based methods are closely related through the same velocity-regression structure. This unified view isolates reward design and anchor velocity as principal design choices in velocity-based fine-tuning. Across text-to-image, text-to-video, and image-to-video generation, RVM matches or outperforms the evaluated trajectory-based methods. On Wan2.1-T2V-1.3B, it achieves the highest VBench Overall among the evaluated methods, with substantially lower estimated training costs than the trajectory-based approaches. For video fine-tuning, we introduce a dynamic-tracking reward that provides explicit motion feedback and improves both Dynamic Degree and overall VBench performance.

Primary area:
generative models
Keywords:
Reinforcement learning, diffusion models, video diffusion models, velocity matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hDuwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes reward-based velocity matching (RVM), which combines reward-weighted velocity matching with an anchor regression term, requiring neither likelihood estimation nor likelihood ratios. The framework unifies RAM, DiffusionNFT, and AWM through a common velocity-regression structure. It also introduces a dynamic-tracking reward to improve video motion. Experiments across text-to-image, text-to-video, and image-to-video generation demonstrate competitive or superior performance, with lower estimated training costs than the evaluated trajectory-based methods on Wan2.1.

Critical strengths
  1. RVM uses a simple reward-weighted velocity-matching objective and demonstrates competitive performance across text-to-image, text-to-video, and image-to-video generation, supporting its practical applicability.
  2. The framework connects RAM, DiffusionNFT, and AWM through a common velocity-regression formulation, clarifying their relationships and highlighting reward weighting and anchor selection as key design choices.
Critical weaknesses

Prior work, including DiffusionNFT and RAM, already uses forward noising and velocity regression without trajectory-based likelihood estimation. While RVM offers a useful unified perspective, its additional theoretical contribution relative to these methods is somewhat unclear. A more explicit discussion of the new insights would strengthen the paper.

Questions for authors
  1. Does “Scaling RL” in the title primarily refer to efficient application across generation tasks, or also to performance improvements with increasing training compute? Is there evidence of further gains with larger GPU-hour budgets, or have the reported runs already plateaued?
  2. In Table 2, DiffusionNFT and RAM use the same reward mixture as RVM, including DT2. Could the authors explain RVM’s substantially higher Dynamic Degree and provide further analysis of the factors contributing to this difference?
2 comments

We thank the reviewer for the careful feedback on what RVM adds beyond DiffusionNFT and RAM, whether Wan2.1 benefits from longer training, and why Dynamic Degree differs across methods on SkyReels when all three use the same reward mixture. We address each point below.

$ $


W1. Insight beyond DiffusionNFT and RAM

Our main contribution is a unified framework for velocity-based reward fine-tuning (Theorem 3.1). RVM is a new formulation that (i) is simple and native to the velocity field, (ii) shows that the three recent methods are special cases of this simple form, and (iii) makes the design choices explicit.

  • (i) Simple and native to the velocity field. RVM is a simple objective with two terms: reward-weighted velocity matching and regression toward an anchor velocity field (Section 3.1). It is written directly on the velocity field and needs no likelihood or likelihood-ratio estimation. It provides clear and intuitive interpretation: a positive reward pulls the predicted velocity toward the sample's target velocity. A negative reward pushes it away. The anchor term keeps the model close to a reference.
  • (ii) It unifies methods with different motivations. DiffusionNFT [1] was derived from contrastive flow matching, RAM [2] from stochastic control, and AWM [3] from ELBO-based policy gradients. Theorem 3.1 unifies all three in a single family of velocity-based methods by identifying the reward weights and anchor configurations under which the RVM objective recovers each method’s update (Section 3.2). This gives all three updates the common interpretation in (i).
  • (iii) The framework makes the design choices explicit and guides our experiments. The common formulation in (ii) identifies how each method weights its samples and anchors its velocity update. We use this framework to compare the RVM family with trajectory-based methods, compare members of the RVM family, and test the effects of individual design choices.
    • Trajectory-based versus velocity-based: on video, every evaluated velocity-based method exceeds every evaluated trajectory-based method in VBench Overall (Tables 1–2). Its estimated training cost is also lower (Figures 4–5, W4).
    • Within velocity-based methods: RVM is comparable to or outperforms DiffusionNFT and RAM.
    • Reward weight and anchor: RVM enables a systematic analysis of these design choices through ablations (W2).

Beyond the formulation, the paper makes two experimental contributions: a dynamic-tracking reward that keeps videos from becoming static (Figure 6), and an evaluation across text-to-image, text-to-video, and image-to-video generation, including SkyReels-V1 at its native 544×960544\times960 resolution (Table 6).

Manuscript revision. Section 3.2 now states, after Theorem 3.1, that the three methods come from different motivations and that this connection defines the velocity-based family, as opposed to trajectory-based methods (p. 6, lines 321–323, and p. 7, lines 363–364).

$ $


Q1. Meaning of "scaling" and the effect of longer training

(a) Meaning of “scaling.” By “scaling RL,” we mean making reward fine-tuning computationally practical across generation tasks, including high-resolution video generation with large diffusion models. Training a fixed model for longer by allocating more GPU-hours falls outside what we mean by “scaling” in the title. Specifically, we mean the following:

  • Efficient training and lower GPU cost. At a batch size of 256 videos on Wan2.1, the estimated cost per training iteration is 5.8 GPU-hours for RVM, 13.5 for FlowGRPO, and 14.5 for DanceGRPO (Figure 5).
  • High-resolution video. On SkyReels image-to-video, we trained on two resolutions, 400×640400\times640 and the model's native 544×960544\times960, about twice the pixels per frame. At this high resolution, RVM reaches a VBench-I2V Overall score of 87.58, which improves on its base model (84.12 with CFG and 82.80 without CFG; Table 6).
  • Larger model. The video models range from Wan2.1-T2V-1.3B (1.3B parameters) to SkyReels-I2V-V1 (13B parameters), ten times larger, and RVM improves both (Tables 1–2).

$ $

(b) Longer training. Increasing the training budget for a fixed model falls outside what we mean by “scaling” in the title. To address the reviewer’s question, we extended the Wan2.1-T2V-1.3B run from the reported 90 rollouts to 130. VBench Overall was 84.13 at both 90 and 110 rollouts and rose to 84.49 at 130, a gain of 0.36 points over the additional 40 rollouts (Table 1).

Tab. 1. Longer training on Wan2.1-T2V-1.3B (VBench).

RolloutsOverallQualitySemantic
7083.2684.7777.24
90 (current report)84.1386.0976.28
11084.1385.7777.55
13084.4986.6176.01

Manuscript revision. The Introduction and Section 4 state what we mean by scaling (p. 2, lines 105–107, and p. 7, lines 372–374).

Q2. RVM's higher Dynamic Degree in Table 2

On SkyReels, the three methods use the same reward mixture. RVM uses no anchor, RAM uses a frozen reference, and DiffusionNFT uses an EMA anchor. To test whether anchoring could contribute to the difference in Dynamic Degree, we compared these anchor choices within RVM on Wan2.1 while holding the reward weighting fixed.

Tab. 2. Anchor choice on Wan2.1-T2V-1.3B (VBench).

AnchorOverallQualitySemanticDynamic Degree
None84.1386.0976.2875.00
πema\pi_{\mathrm{ema}}83.8085.5476.8776.39
πref\pi_{\mathrm{ref}}82.1083.5876.1848.61
πref\pi_{\mathrm{ref}} (CFG)84.1785.4978.8870.83

The frozen anchor reduced Dynamic Degree from 75.00 to 48.61, whereas the EMA anchor yielded 76.39. The Wan2.1 result therefore suggests that frozen anchoring may contribute to RAM’s lower Dynamic Degree on SkyReels. The cause of DiffusionNFT’s lower score on SkyReels remains open.

Manuscript revision. The anchor experiment is in Appendix G.1.1 (pp. 26–27, lines 1382–1406, Table 9), and Section 4.3 discusses the anchor choice (pp. 9–10, lines 482–489, Figure 7).

References

[1] DiffusionNFT: Online Diffusion Reinforcement with Forward Process

[2] Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

[3] Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models

Reviewer_A4K3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates reward fine-tuning as signed, reward-weighted velocity regression with an optional anchor term. It shows that particular choices of rewards and anchors recover the updates of RAM, DiffusionNFT, and AWM. In video experiments, it also introduces a dynamic-tracking reward after finding that preference and visual-quality rewards can favor nearly static outputs. Results span image, text-to-video, and image-to-video generation.

Critical strengths
  1. The update-level equivalences clarify that several differently motivated methods share a velocity-regression structure. This makes reward weighting and anchor choice easier to analyze.
  2. The motion ablation identifies a specific reward failure: VideoAlign and HPSv3 mixtures can raise aggregate scores while nearly eliminating motion. Adding dynamic tracking reverses that effect.
  3. The paper compares RVM with RAM and DiffusionNFT under the same video reward mixture, giving a more relevant test than the published trajectory-baseline scores alone.
Critical weaknesses
  1. The main RVM form substantially overlaps with existing velocity-based updates. The paper’s independent algorithmic contribution is narrower than its broad framing suggests.
  2. For Wan2.1, the RVM, RAM, and DiffusionNFT runs differ in anchor and reward-weight handling as well as objective form. Their score differences do not isolate one design decision.
  3. The theoretical link to a KL-regularized reward optimum requires positive exponential reward weights and strong realizability assumptions. The video experiments use signed, group-centered weights, so that result does not justify the implemented update directly.
  4. The 11.8× total-cost comparison extrapolates DanceGRPO to a different number of iterations. The paper does not establish time to a matched quality threshold.
  5. On SkyReels-I2V, RVM improves Dynamic Degree and Quality but scores below RAM on Overall.
Questions for authors

Please refer to weakness

3 comments

We thank the reviewer for the detailed feedback on RVM’s relationship to prior velocity-based methods, the assumptions behind our KL-optimality argument, and the interpretation of our video results and training-cost comparisons. We address each point below.

$ $


W1. Overlap with existing velocity-based methods

Our main contribution is a unified framework for velocity-based reward fine-tuning (Theorem 3.1). RVM is a new formulation that (i) is simple and native to the velocity field, (ii) shows that the three recent methods are special cases of this simple form, and (iii) makes the design choices explicit.

  • (i) Simple and native to the velocity field. RVM is a simple objective with two terms: reward-weighted velocity matching and regression toward an anchor velocity field (Section 3.1). It is written directly on the velocity field and needs no likelihood or likelihood-ratio estimation. It provides clear and intuitive interpretation: a positive reward pulls the predicted velocity toward the sample's target velocity. A negative reward pushes it away. The anchor term keeps the model close to a reference.
  • (ii) It unifies methods with different motivations. DiffusionNFT [1] was derived from contrastive flow matching, RAM [2] from stochastic control, and AWM [3] from ELBO-based policy gradients. Theorem 3.1 unifies all three in a single family of velocity-based methods by identifying the reward weights and anchor configurations under which the RVM objective recovers each method’s update (Section 3.2). This gives all three updates the common interpretation in (i).
  • (iii) The framework makes the design choices explicit and guides our experiments. The common formulation in (ii) identifies how each method weights its samples and anchors its velocity update. We use this framework to compare the RVM family with trajectory-based methods, compare members of the RVM family, and test the effects of individual design choices.
    • Trajectory-based versus velocity-based: on video, every evaluated velocity-based method exceeds every evaluated trajectory-based method in VBench Overall (Tables 1–2). Its estimated training cost is also lower (Figures 4–5, W4).
    • Within velocity-based methods: RVM is comparable to or outperforms DiffusionNFT and RAM.
    • Reward weight and anchor: RVM enables a systematic analysis of these design choices through ablations (W2).

Beyond the formulation, the paper makes two experimental contributions: a dynamic-tracking reward that keeps videos from becoming static (Figure 6), and an evaluation across text-to-image, text-to-video, and image-to-video generation, including SkyReels-V1 at its native 544×960544\times960 resolution (Table 6).

Manuscript revision. Section 3.2 now states, after Theorem 3.1, that the three methods come from different motivations and that this connection defines the velocity-based family, as opposed to trajectory-based methods (p. 6, lines 321–323, and p. 7, lines 363–364).

W2. Effect of reward weighting and anchor choice

RVM expresses the updates of RAM and DiffusionNFT in a common velocity-regression form (Theorem 3.1), and our ablations examine how (a) reward composition, (b) reward weighting, and (c) anchoring affect performance. Our new experiments vary the anchor configuration at fixed reward weighting on Wan2.1 and the reward weight at a fixed pretrained anchor on SD3.5-M OCR.

(a) Reward composition (Figure 6). Within RVM on Wan2.1, VideoAlign alone gives an Overall score of 71.04 and a Dynamic Degree of 2.78. Replacing its visual-quality component with HPSv3 raises Overall to 80.39, while Dynamic Degree remains low at 5.56. Adding the dynamic-tracking reward to that same mixture raises Overall to 84.13 and Dynamic Degree to 75.00.

(b) Reward weight (new, SD3.5-M OCR). Rows 2 and 3 are new RVM runs that differ only in the reward weight. We ran this pair on SD3.5-M OCR, where controlled runs are inexpensive; the anchor comparison in (c) is on Wan2.1. Row 4 is DiffusionNFT itself (NFT-based weight with the EMA anchor), taken from Table 3; it is a reference and not a controlled change of the anchor alone.

Tab. 1. Reward weight and anchor on SD3.5-M OCR.

Reward weightAnchorOCRPickScoreCLIP ScoreHPSv2.1ImageRewardAesthetic
base model—0.12120.190.2720.188−0.30-0.305.27
Group-centeredπref\pi_{\mathrm{ref}}0.95822.070.3030.2690.795.22
rnftir^i_{\mathrm{nft}}πref\pi_{\mathrm{ref}}0.94221.800.3020.2290.325.19
rnftir^i_{\mathrm{nft}} (DiffusionNFT, Table 3)πema\pi_{\mathrm{ema}}0.9322.090.3070.2770.975.17

(c) Anchor. We fix the reward weight to rgrpoir^i_{\mathrm{grpo}} and compare four anchors. All rows share the RVM objective, the training data, the effective reward scale, and the evaluation protocol. The no-anchor row sets β=0\beta=0 and is the RVM result in Table 1. For each anchor, we selected the better β\beta from {10−3,10−2}\{10^{-3},10^{-2}\} and report all metrics at that value.

Tab. 2. Anchor choice on Wan2.1-T2V-1.3B (VBench).

AnchorOverallQualitySemanticDynamic Degree
None84.1386.0976.2875.00
πema\pi_{\mathrm{ema}}83.8085.5476.8776.39
πref\pi_{\mathrm{ref}}82.1083.5876.1848.61
πref\pi_{\mathrm{ref}} (CFG)84.1785.4978.8870.83

Takeaways.

  • Reward composition. On Wan2.1, adding the dynamic-tracking reward to the HPSv3 mixture substantially improves motion within RVM.
  • Reward weight. With the same frozen anchor, group-centered and NFT-based weights give similar OCR accuracy. Group-centered weighting scores higher on ImageReward.
  • Anchor choice. Relative to unanchored RVM, Dynamic Degree changes little with the EMA anchor and drops with the unguided frozen reference. The CFG-guided reference gives the highest Semantic score.

Manuscript revision. Both experiments are in Appendix G.1.1 with settings and full results (pp. 26–27, lines 1372–1406, Tables 8–9). Section 4.3 discusses the anchor choice (pp. 9–10, lines 482–489, Figure 7), and Appendix C.1 gives the anchor evaluation protocol (p. 22, lines 1150–1155).

W4. Fair comparison of training efficiency

The 2.32.3–2.5×2.5\times and the 11.8×11.8\times are not the same comparison under different settings. They measure two different things: the first is the cost of one iteration, and the second is the total cost of a full run.

  • 2.32.3–2.5×2.5\times (Figure 5): the cost of one iteration. At the same batch of 256 videos, one iteration costs 5.8 GPU-hours for RVM, 13.5 for FlowGRPO, and 14.5 for DanceGRPO. This is the saving from the method itself: RVM samples with a CFG-free ODE and trains on fewer noise levels.
  • 11.8×11.8\times (Figure 4): the total cost of the reported runs. RVM uses 525 GPU-hours (90 iterations of 256 videos), and DanceGRPO an estimated 6,171 GPU-hours (220 iterations of 512 videos). This number also contains the difference between the two schedules, so it does not show the cost to reach the same quality.

For this DanceGRPO configuration, only the final VBench score is available, so the 11.8×11.8\times estimate compares the costs of the two full training schedules. To assess training efficiency more fairly, we added a comparison between RVM and FlowGRPO on SD3.5-M OCR, where we can match key training settings and compare the cost of reaching a common test accuracy threshold of 0.92. Both methods use the same prompt set and reward, with 48 prompts per rollout, 24 images per prompt, and 10 sampling steps. We report both costs in GPU-hours on the same hardware.

Tab. 3. Training cost on SD3.5-M OCR. OCR is the accuracy on test prompts.

MethodOCRRolloutsGPU-hours per rolloutGPU-hoursReduction
FlowGRPO0.92∼600\sim6001.12∼670\sim6701×1\times
RVM0.958700.53∼37\sim37∼18×\sim18\times
RVM, first 28 rollouts0.93280.53∼15\sim15∼45×\sim45\times

At the 28-rollout checkpoint, RVM reached 0.93 test OCR accuracy, exceeding the 0.92 target at a measured cost of about 15 GPU-hours. We estimate that FlowGRPO required about 670 GPU-hours to reach the same target, roughly 45 times the cost of the RVM checkpoint. For FlowGRPO, we used the rollout count from its published learning curve [4] and estimated the per-rollout cost on our hardware.

Manuscript revision. Section 4.2 states that the Wan2.1 costs are estimates (p. 9, lines 449–458; captions of Figures 4–5, p. 8) and reports the OCR cost comparison next to the Wan2.1 results (p. 9, lines 458–461). Appendix G.1.2 gives the matched settings, the test evaluation, the per-rollout costs, and the cost table (p. 27, lines 1408–1436, Table 10), and Appendix C.2 the sampling protocol (p. 22, lines 1170–1173).

$ $


W3. Theoretical link to a KL-regularized reward optimum

We thank the reviewer for pointing this out. The link to a KL-regularized reward optimum requires exponential reward weights and strong realizability assumptions, so it does not directly justify the implemented update. We therefore removed this content from the manuscript.

Manuscript revision. The appendix section on unanchored matching with exponential reward weights (Appendix B.6 of the submission) is removed, together with the part of the next section (Appendix B.7 of the submission) that referred to it. The remaining small-step expansion of that section is merged into Appendix B.5 as its last paragraph (p. 21, lines 1092–1100).

$ $


W5. Interpreting similar performance on SkyReels

On SkyReels-I2V, RVM performs similarly to DiffusionNFT and RAM: the three methods reach VBench-I2V Overall scores of 86.27, 86.16, and 86.61 (Table 2). On Wan2.1, RVM outperforms both, with an Overall score of 84.13, compared with 82.95 and 81.95 (Table 1). Our main contribution lies elsewhere: the unified framework that connects these methods, the easier interpretation it gives, the systematic analysis of design choices it enables, and the evaluation on video generation (W1).

References

[1] DiffusionNFT: Online Diffusion Reinforcement with Forward Process

[2] Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

[3] Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models

[4] Flow-GRPO: Training Flow Matching Models via Online RL

Reviewer_pdMTOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

To bypass the complexities of adapting LLM-style likelihood optimization to diffusion models, the authors propose Reward-Based Velocity Matching (RVM). By framing fine-tuning directly around velocity regression rather than likelihood estimation, RVM weights the velocity loss using rewards and an anchor term, unifying methods like Reinforce Adjoint Matching and DiffusionNFT. Across text-to-image, text-to-video, and image-to-video tasks, RVM achieves competitive or superior performance while substantially reducing training costs. Furthermore, it introduces a dynamic-tracking reward that enhances motion feedback in video fine-tuning.

Critical strengths
  1. I really appreciate the authors' insight in Fig. 1 regarding the two generative paradigms. I have also thought about this perspective for some time as well, and I agree that it very natural to fine-tune diffusion models directly in their native velocity space.
  2. Empirical results across text-to-image, text-to-video, and image-to-video tasks show strong performance together with substantial reductions in training cost.
Critical weaknesses
  1. The title highlights scaling, yet the experiments lack systematic studies over model size, resolution, or compute. Additionally, the reported 11.8×11.8\times cost reduction relies on mismatched settings, whereas matched comparisons show a smaller 2.3–2.5×2.3\text{–}2.5\times drop, and it remains unexplained why RVM scales while its special case DiffusionNFT does not.
  2. Because RVM unifies prior methods like DiffusionNFT and RAM as special cases, and top video results depend on a new dynamic-tracking reward, it is unclear whether performance gains stem from the RVM objective itself or from specific design choices. More controlled ablations are needed to isolate the core contribution.
Questions for authors

see weaknesses.

5 comments

We thank the reviewer for the positive assessment and constructive feedback on what we mean by scaling, how differences in training budgets affect the cost comparison, and how reward design contributes to our video results. We address each point below.

$ $


W1-(1). Meaning of scalability

By scaling, we mean making reward fine-tuning computationally practical across generation tasks: it trains efficiently at a lower GPU cost (see W1-(2)) and works at larger scales. We use "scaling" in this sense and do not claim a scaling law. Specifically, we mean the following:

  • Efficient training and lower GPU cost. At a batch size of 256 videos on Wan2.1, the estimated cost per training iteration is 5.8 GPU-hours for RVM, 13.5 for FlowGRPO, and 14.5 for DanceGRPO (Figure 5).
  • High-resolution video. On SkyReels image-to-video, we trained on two resolutions, 400×640400\times640 and the model's native 544×960544\times960, about twice the pixels per frame. At this high resolution, RVM reaches a VBench-I2V Overall score of 87.58, which improves on its base model (84.12 with CFG and 82.80 without CFG; Table 6).
  • Larger model. The video models range from Wan2.1-T2V-1.3B (1.3B parameters) to SkyReels-I2V-V1 (13B parameters), ten times larger, and RVM improves both (Tables 1–2).

Manuscript revision. The Introduction now states what we mean by scalability (p. 2, lines 105–107), and Section 4 states what we mean by scaling (p. 7, lines 372–374).

$ $


W1-(2). Cost reduction under mismatched training budgets

The 2.32.3–2.5×2.5\times and the 11.8×11.8\times are not the same comparison under different settings. They measure two different things: the first is the cost of one iteration, and the second is the total cost of a full run.

  • 2.32.3–2.5×2.5\times (Figure 5): the cost of one iteration. At the same batch of 256 videos, one iteration costs 5.8 GPU-hours for RVM, 13.5 for FlowGRPO, and 14.5 for DanceGRPO. This is the saving from the method itself: RVM samples with a CFG-free ODE and trains on fewer noise levels.
  • 11.8×11.8\times (Figure 4): the total cost of the reported runs. RVM uses 525 GPU-hours (90 iterations of 256 videos), and DanceGRPO an estimated 6,171 GPU-hours (220 iterations of 512 videos). This number also contains the difference between the two schedules, so it does not show the cost to reach the same quality.

For this DanceGRPO configuration, only the final VBench score is available, so the 11.8×11.8\times estimate compares the costs of the two full training schedules. To assess training efficiency more fairly, we added a comparison between RVM and FlowGRPO on SD3.5-M OCR, where we can match key training settings and compare the cost of reaching a common test accuracy threshold of 0.92. Both methods use the same prompt set and reward, with 48 prompts per rollout, 24 images per prompt, and 10 sampling steps. We report both costs in GPU-hours on the same hardware.

Tab. 3. Training cost on SD3.5-M OCR. OCR is the accuracy on test prompts.

MethodOCRRolloutsGPU-hours per rolloutGPU-hoursReduction
FlowGRPO0.92∼600\sim6001.12∼670\sim6701×1\times
RVM0.958700.53∼37\sim37∼18×\sim18\times
RVM, first 28 rollouts0.93280.53∼15\sim15∼45×\sim45\times

At the 28-rollout checkpoint, RVM reached 0.93 test OCR accuracy, exceeding the 0.92 target at a measured cost of about 15 GPU-hours. We estimate that FlowGRPO required about 670 GPU-hours to reach the same target, roughly 45 times the cost of the RVM checkpoint. For FlowGRPO, we used the rollout count from its published learning curve [4] and estimated the per-rollout cost on our hardware.

Manuscript revision. Section 4.2 states that the Wan2.1 costs are estimates (p. 9, lines 449–458; captions of Figures 4–5, p. 8) and reports the OCR cost comparison next to the Wan2.1 results (p. 9, lines 458–461). Appendix G.1.2 gives the matched settings, the test evaluation, the per-rollout costs, and the cost table (p. 27, lines 1408–1436, Table 10), and Appendix C.2 the sampling protocol (p. 22, lines 1170–1173).

$ $


W1-(3). Scalability of RVM and its special cases

We do not claim that DiffusionNFT does not work on video. DiffusionNFT and RAM also ran successfully on both video models we tested. They reach VBench Overall scores of 82.95 and 81.95 on Wan2.1 (Table 1), and 86.16 and 86.61 on SkyReels (Table 2).

Our contribution is therefore not that RVM alone scales. It is the unified framework itself. RVM is a simple objective that contains these methods as special cases (Theorem 3.1). This gives an easier interpretation of them and allows a systematic analysis of the design choices (W2). In our large-scale experiments on image and video generation, RVM is comparable to or outperforms these methods (Tables 1–3).

W2. Gains from the objective or from design choices?

RVM unifies RAM and DiffusionNFT within a single family of velocity-based methods by showing how specific choices of reward weight and anchor recover their updates from a common regression objective (Theorem 3.1). For video training, we use the simplest form of this objective: removing the anchor leaves only reward-weighted velocity matching. When all three methods use the same reward mixture, this unanchored RVM achieves a higher Overall score than both RAM and DiffusionNFT on Wan2.1 and a comparable score on SkyReels (Tables 1–2).

To examine these design choices more systematically within RVM, we complement our reward-composition ablation on Wan2.1 (Figure 6) with two new comparisons. On SD3.5-M OCR, we vary the reward weight while holding the pretrained anchor fixed. On Wan2.1, we compare anchor configurations under the same reward weighting.

(a) Reward composition (Figure 6). Within RVM on Wan2.1, VideoAlign alone gives an Overall score of 71.04 and a Dynamic Degree of 2.78. Replacing its visual-quality component with HPSv3 raises Overall to 80.39, while Dynamic Degree remains low at 5.56. Adding the dynamic-tracking reward to that same mixture raises Overall to 84.13 and Dynamic Degree to 75.00.

(b) Reward weight (new, SD3.5-M OCR). Rows 2 and 3 are new RVM runs that differ only in the reward weight. We ran this pair on SD3.5-M OCR, where controlled runs are inexpensive; the anchor comparison in (c) is on Wan2.1. Row 4 is DiffusionNFT itself (NFT-based weight with the EMA anchor), taken from Table 3; it is a reference and not a controlled change of the anchor alone.

Tab. 1. Reward weight and anchor on SD3.5-M OCR.

Reward weightAnchorOCRPickScoreCLIP ScoreHPSv2.1ImageRewardAesthetic
— (base model)—0.12120.190.2720.188−0.30-0.305.27
Group-centeredπref\pi_{\mathrm{ref}}0.95822.070.3030.2690.795.22
rnftir^i_{\mathrm{nft}}πref\pi_{\mathrm{ref}}0.94221.800.3020.2290.325.19
rnftir^i_{\mathrm{nft}} (DiffusionNFT, Table 3)πema\pi_{\mathrm{ema}}0.9322.090.3070.2770.975.17

(c) Anchor. We fix the reward weight to rgrpoir^i_{\mathrm{grpo}} and compare four anchors. All rows share the RVM objective, the training data, the effective reward scale, and the evaluation protocol. The no-anchor row sets β=0\beta=0 and is the RVM result in Table 1. For each anchor, we selected the better β\beta from {10−3,10−2}\{10^{-3},10^{-2}\} and report all metrics at that value.

Tab. 2. Anchor choice on Wan2.1-T2V-1.3B (VBench).

AnchorOverallQualitySemanticDynamic Degree
None84.1386.0976.2875.00
πema\pi_{\mathrm{ema}}83.8085.5476.8776.39
πref\pi_{\mathrm{ref}}82.1083.5876.1848.61
πref\pi_{\mathrm{ref}} (CFG)84.1785.4978.8870.83

Takeaways.

  • Reward composition. On Wan2.1, adding the dynamic-tracking reward to the HPSv3 mixture substantially improves motion within RVM.
  • Reward weight. With the same frozen anchor, group-centered and NFT-based weights give similar OCR accuracy. Group-centered weighting scores higher on ImageReward.
  • Anchor choice. Relative to unanchored RVM, Dynamic Degree changes little with the EMA anchor and drops with the unguided frozen reference. The CFG-guided reference gives the highest Semantic score.

Manuscript revision. The reward-weight and anchor experiments are in Appendix G.1.1 with settings and full results (pp. 26–27, lines 1372–1406, Tables 8–9). Section 4.3 discusses the anchor choice (pp. 9–10, lines 482–489, Figure 7), and Appendix C.1 gives the anchor evaluation protocol (p. 22, lines 1150–1155).

References

[1] DiffusionNFT: Online Diffusion Reinforcement with Forward Process

[2] Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

[3] Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models

[4] Flow-GRPO: Training Flow Matching Models via Online RL

I appreciate the authors’ explanations and clarifications.

I believe the term “scaling” may lead to potential misunderstandings among readers, particularly when used in the title. I wonder whether the authors would consider revising the terminology or clarifying its intended meaning to avoid such ambiguity.

My other concerns have been adequately addressed. Thank you again for the thoughtful responses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We summarize the manuscript revisions, additional experiments, and clarifications below.

1. Clarifying the overlap with prior methods and related ablations [pdMT, A4K3, hDuw]

  • Our main contribution is a unified velocity-matching objective that recovers the independently derived updates of RAM, DiffusionNFT, and AWM under the conditions of Theorem 3.1 (Secs. 3.1–3.2). RVM makes reward weighting and anchoring explicit as separate design choices. These were intertwined in the original RAM and DiffusionNFT objectives, and we can now vary each independently to test its effect.

  • We added controlled ablations of reward weighting and anchor choice (Tabs. 8–9). On Wan2.1, a CFG-guided frozen velocity anchor improved semantic alignment relative to the unanchored default (Fig. 7, Tab. 9).

  • With the same reward mixture and ODE sampler, our simple default configuration (group-centered weights and constant anchor) achieved higher overall scores than RAM and DiffusionNFT on Wan2.1 and comparable scores on SkyReels (Tabs. 1–2).

2. Training efficiency [pdMT, A4K3]

  • We clarified what the Wan2.1 cost estimates measure per iteration and over the full reported training runs (Figs. 4–5).
  • A comparison at a common OCR accuracy target showed that RVM reached the target at substantially lower training cost than the FlowGRPO estimate (Sec. 4.2, App. G.1.2, Tab. 10).

3. Meaning of scaling [pdMT, hDuw]

  • We clarified that scaling refers to making reward fine-tuning computationally practical across image and video tasks, including high-resolution video generation (Sec. 1 and Sec. 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.