- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes reward-based velocity matching (RVM), which combines reward-weighted velocity matching with an anchor regression term, requiring neither likelihood estimation nor likelihood ratios. The framework unifies RAM, DiffusionNFT, and AWM through a common velocity-regression structure. It also introduces a dynamic-tracking reward to improve video motion. Experiments across text-to-image, text-to-video, and image-to-video generation demonstrate competitive or superior performance, with lower estimated training costs than the evaluated trajectory-based methods on Wan2.1.
Critical strengths
- RVM uses a simple reward-weighted velocity-matching objective and demonstrates competitive performance across text-to-image, text-to-video, and image-to-video generation, supporting its practical applicability.
- The framework connects RAM, DiffusionNFT, and AWM through a common velocity-regression formulation, clarifying their relationships and highlighting reward weighting and anchor selection as key design choices.
Critical weaknesses
Prior work, including DiffusionNFT and RAM, already uses forward noising and velocity regression without trajectory-based likelihood estimation. While RVM offers a useful unified perspective, its additional theoretical contribution relative to these methods is somewhat unclear. A more explicit discussion of the new insights would strengthen the paper.
Questions for authors
- Does “Scaling RL” in the title primarily refer to efficient application across generation tasks, or also to performance improvements with increasing training compute? Is there evidence of further gains with larger GPU-hour budgets, or have the reported runs already plateaued?
- In Table 2, DiffusionNFT and RAM use the same reward mixture as RVM, including DT2. Could the authors explain RVM’s substantially higher Dynamic Degree and provide further analysis of the factors contributing to this difference?
2 comments
Q2. RVM's higher Dynamic Degree in Table 2
On SkyReels, the three methods use the same reward mixture. RVM uses no anchor, RAM uses a frozen reference, and DiffusionNFT uses an EMA anchor. To test whether anchoring could contribute to the difference in Dynamic Degree, we compared these anchor choices within RVM on Wan2.1 while holding the reward weighting fixed.
Tab. 2. Anchor choice on Wan2.1-T2V-1.3B (VBench).
| Anchor | Overall | Quality | Semantic | Dynamic Degree |
|---|---|---|---|---|
| None | 84.13 | 86.09 | 76.28 | 75.00 |
| 83.80 | 85.54 | 76.87 | 76.39 | |
| 82.10 | 83.58 | 76.18 | 48.61 | |
| (CFG) | 84.17 | 85.49 | 78.88 | 70.83 |
The frozen anchor reduced Dynamic Degree from 75.00 to 48.61, whereas the EMA anchor yielded 76.39. The Wan2.1 result therefore suggests that frozen anchoring may contribute to RAM’s lower Dynamic Degree on SkyReels. The cause of DiffusionNFT’s lower score on SkyReels remains open.
Manuscript revision. The anchor experiment is in Appendix G.1.1 (pp. 26–27, lines 1382–1406, Table 9), and Section 4.3 discusses the anchor choice (pp. 9–10, lines 482–489, Figure 7).
References
[1] DiffusionNFT: Online Diffusion Reinforcement with Forward Process
[2] Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models
[3] Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
We thank the reviewer for the careful feedback on what RVM adds beyond DiffusionNFT and RAM, whether Wan2.1 benefits from longer training, and why Dynamic Degree differs across methods on SkyReels when all three use the same reward mixture. We address each point below.
$ $
W1. Insight beyond DiffusionNFT and RAM
Our main contribution is a unified framework for velocity-based reward fine-tuning (Theorem 3.1). RVM is a new formulation that (i) is simple and native to the velocity field, (ii) shows that the three recent methods are special cases of this simple form, and (iii) makes the design choices explicit.
Beyond the formulation, the paper makes two experimental contributions: a dynamic-tracking reward that keeps videos from becoming static (Figure 6), and an evaluation across text-to-image, text-to-video, and image-to-video generation, including SkyReels-V1 at its native 544×960 resolution (Table 6).
Manuscript revision. Section 3.2 now states, after Theorem 3.1, that the three methods come from different motivations and that this connection defines the velocity-based family, as opposed to trajectory-based methods (p. 6, lines 321–323, and p. 7, lines 363–364).
$ $
Q1. Meaning of "scaling" and the effect of longer training
(a) Meaning of “scaling.” By “scaling RL,” we mean making reward fine-tuning computationally practical across generation tasks, including high-resolution video generation with large diffusion models. Training a fixed model for longer by allocating more GPU-hours falls outside what we mean by “scaling” in the title. Specifically, we mean the following:
$ $
(b) Longer training. Increasing the training budget for a fixed model falls outside what we mean by “scaling” in the title. To address the reviewer’s question, we extended the Wan2.1-T2V-1.3B run from the reported 90 rollouts to 130. VBench Overall was 84.13 at both 90 and 110 rollouts and rose to 84.49 at 130, a gain of 0.36 points over the additional 40 rollouts (Table 1).
Tab. 1. Longer training on Wan2.1-T2V-1.3B (VBench).
Manuscript revision. The Introduction and Section 4 state what we mean by scaling (p. 2, lines 105–107, and p. 7, lines 372–374).