- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies motion collapse during video reward optimization and proposes GraftOOD, which combines reward-rectified online samples with external video examples in a shared velocity-regression objective. Specifically,online rollouts are trained with reward-rectified velocity targets, while external exemplars are re-noised and assigned standard conditional flow-matching targets, allowing both sources to be optimized through a shared regression objective without access to the exemplars’ original generation trajectories.
Critical strengths
1.The paper is motivated by a practically important failure mode. As a user of video benchmarks and reward models, I find the observation that proxy-reward optimization can reduce video dynamics both credible and relevant. The paper further demonstrates this behavior with well constructed experiments.
2.The proposed formulation offers a clean and practically convenient way to incorporate terminal-only exemplars into reward-guided velocity regression without requiring their original generation trajectories.
Critical weaknesses
-
The claim that GraftOOD avoids an additional SFT-loss coefficient is somewhat overstated. The relative strength of exemplar supervision is instead controlled by the empirically selected and , making their selection analogous to tuning an explicit loss coefficient rather than eliminating the balancing problem. Moreover, changing simultaneously changes both exemplar exposure and the number of online rollouts used for advantage estimation, making their individual effects difficult to isolate.
-
The practical cost of constructing GraftSet-Video is not reported. The dataset relies on 9,588 videos collected from or generated by multiple models, yet the paper does not provide the generation time and computational or API cost. Since acquiring these examples is a central component of the method, this omission makes its practical feasibility and reproducibility difficult to assess.
-
The motivation for the unified framework, and its advantages over simply combining FlowAWR with exemplar-based SFT, are not sufficiently clear or empirically isolated (see Questions).
Questions for authors
-
Online rollouts provide full generation trajectories, yet GraftOOD discards this information and trains from re-noised terminal samples. Could the authors clarify the advantages and limitations of velocity regression relative to trajectory-based optimization?
-
What benefit does the unified regression objective provide over a naïve combination of FlowAWR and a separately weighted exemplar SFT loss?
9 comments
Our Response to Reviewer sxz4 (2/2) [Q1, W2]: Terminal Regression, Trajectory-Based Optimization, and Dataset Cost
Q1: What are the advantages and limitations of velocity regression from re-noised terminal samples relative to trajectory-based optimization?
A3: The reviewer is correct that GraftOOD trains from re-noised terminal samples rather than the sampled denoising transitions, for both online videos and exemplars. This is inherited from FlowAWR. For online samples, terminal rewards define group-relative advantages, which rectify the reference velocity at the re-noised state. Retaining this reward-guided target does not mean retaining the original trajectory in the training objective.
- Why this interface. Both sources provide terminal video latents. Re-noising them gives inputs in the same velocity space, allowing online reward rectification and exemplar supervision to enter one regression objective. External generators need not expose denoising trajectories, transition likelihoods, or their velocity fields.
- What it trades off. Trajectory-based optimization assigns terminal reward through likelihood terms along sampled transitions. Terminal regression does not use that transition-level information. Its practical advantage coefficients also approximate state-conditional quantities using finite-group reward normalization. Our pointwise-optimum result characterizes the regression target; it does not establish equivalence to a full trajectory policy gradient or guarantee better video quality.
- Empirical comparison. FlashOOD provides the trajectory-based alternative: it combines Flash-GRPO's single-SDE-transition update with exemplar SFT at the matched noise level. With the same exemplar pool and main allocation, GraftOOD attains DD/VR/HPSv3 of 44.41/5.08/7.32, versus 58.26/1.65/5.31 for FlashOOD. Thus regression supports a useful dynamics–reward balance in this comparison; we do not claim that it universally dominates trajectory-based optimization.
W2: The practical cost of constructing GraftSet-Video is not reported.
A4:
- Collection and selection. We retain videos from public video-reward-model resources, keeping only sources whose mean HPSv3 is above 7, then supplement prompts with fewer than four exemplars. All nine sources in the final corpus are generators substantially stronger than the base policy (source HPSv3 means 8.656–10.155). The final corpus contains 9,588 videos for 2,397 prompts. Supplementary generation fills the missing slots; the reported API spending is for this stage, not for regenerating every video or reproducing the source datasets.
- Acquisition cost. HPSv3 scoring for selection took under one day on 8 RTX PRO 6000 GPUs. Supplementary generation cost under 2,000 USD in API fees and took about three days in parallel. These figures cover compute and API fees, not manual effort.
- Training cost. The main Wan2.1-1.3B RL comparisons use 160 optimizer steps. Normalized to 24 RTX PRO 6000 GPUs (96 GB), including evaluation, FlowAWR takes about 29 h (≈700 GPU·h); GraftOOD uses ≈700/660/650 GPU·h at . These are approximate measured costs, with wall-clock variation from reward-service waiting. In the main setting, each round contains 48 groups: eight groups replace four rollouts with exemplars, reducing generation from 384 to 352 videos. Exemplars are not scored by the reward model during training.
- Release. The de-anonymized version will link the prompts and videos, subject to source-dataset licenses and generation-service terms. Where video redistribution is not permitted, we will provide prompts, source indices, and preprocessing scripts.
- Exemplar statistics. On the same GraftSet prompts, exemplar DD is 80.14, versus 53.61 for the base Wan2.1-1.3B (0–100 scale); per-source DD ranges from 58.96 to 95.02. The GraftSet mean HPSv3 is 9.735; downstream TA is −0.45 for GraftOOD, versus −0.65 for FlowAWR and −0.66 for the base model.
In summary, we agree that and set exemplar strength. The new controls isolate the rollout-count confound, and at the same relative loss weight the append variant ends below GraftOOD on HPSv3, VR, and DD. Supplementary generation cost under 2,000 USD in API fees. We hope these results address your concerns and are happy to clarify any remaining points.
Our Kind Reminder to Reviewer sxz4: Summary of Our Response (Exemplar Strength, Unified Objective, Terminal Regression, Cost)
Dear Reviewer sxz4,
Thank you again for your careful review. As the discussion period is nearing its end, we briefly summarize our response to your questions. The General Response outlines the starting point of GraftOOD and collects the additional results.
- Exemplar strength and rollout count (W1; Response (1/2), A1). and set exemplar strength through data composition. To separate exposure from the online rollout count at , GraftSet prompts alone give DD 10.21, reducing those groups to four rollouts gives 6.27, and adding four exemplars at that fixed online count gives 44.41. Across all nine positive settings, DD stays within 30.58–48.68 (FlowAWR: 4.86).
- Unified objective versus FlowAWR + SFT (W3 & Q2; Response (1/2), A2). The exemplar target, the exclusion of exemplars from reward normalization, and their group-fraction weight follow from one dual-source group definition, so GraftOOD regresses toward the in-distribution reward-rectified target and the out-of-distribution exemplar field within one group. An append control (FlowAWR + ·SFT) instead keeps all eight rollouts and adds the same exemplars as a separate SFT loss at GraftOOD's relative weight. Although it generates more videos per round (384 vs. 352), it ends below GraftOOD on HPSv3, VR, and DD (6.96/4.89/36.58 vs. 7.32/5.08/44.41).
- Terminal regression versus trajectory-based optimization (Q1; Response (2/2), A3). Training from re-noised terminal samples, inherited from FlowAWR, lets online videos and exemplars without denoising trajectories enter one regression objective; it does not use transition-level likelihoods, and we do not claim equivalence to a full trajectory policy gradient. Against FlashOOD, the trajectory-based alternative, GraftOOD attains DD/VR/HPSv3 of 44.41/5.08/7.32 versus 58.26/1.65/5.31, a better dynamics–reward balance in this comparison.
- Construction and training cost (W2; Response (2/2), A4). HPSv3 scoring for selection took under one day on 8 RTX PRO 6000 GPUs, and supplementary generation cost under 2,000 USD in API fees over about three days in parallel. Training takes about 700 GPU·h for FlowAWR and 650–700 GPU·h for GraftOOD.
We hope that our response resolves your concerns, and we would be grateful if you could take it into account when reconsidering your final recommendation.
Our Response to Reviewer sxz4 (1/2) [W1, W3, Q2]: Exemplar Strength, Rollout Controls, and the Unified Objective
We thank the reviewer for the careful and constructive review. We respond to each point below.
A1: We agree that "without an additional supervised-loss coefficient" overstates the point: ρ and nood do set exemplar strength. The formulation uses unit-weight residuals in the same velocity space; it makes supervision strength a data-composition choice, but does not eliminate balancing.
Changing nood also changes the number of online rollouts and the statistics used for reward normalization. We separate these factors at ρ=0.167, 160 steps:
Same evaluation set and protocol as the main table.
Introducing GraftSet prompts alone gives DD 10.21; reducing those groups to four rollouts gives 6.27; adding four exemplars at that fixed online count gives 44.41.
The existing sweep shows that the benefit is not restricted to one ratio. Across all nine positive settings, DD is 30.58–48.68 (FlowAWR: 4.86) and HPSv3 is 6.54–7.74 (base: 4.70). At ρ=0.167, nood=1, exemplars constitute about 2% of training samples, yet DD reaches 30.58 and VR 5.52 (FlowAWR: 5.11). The ratios select different points on the dynamics–reward trade-off.
A2: GraftOOD applies advantage-weighted regression to a group containing online rollouts and terminal-only exemplars. Its decomposition into FlowAWR and SFT terms follows from this formulation. For exemplars, an uncentered unit coefficient cancels the unavailable reference field and yields a computable flow-matching target; exemplars are excluded from online reward normalization; and their relative weight is the group fraction. These choices define how external videos enter reward-relative learning.
We compare against FlowAWR + λ·SFT with the same GraftSet prompts and exemplars: four exemplars are appended to eight online rollouts in selected groups, with λ=1 and all other settings matched, including prompt mixing, shared timestep, and online-only reward normalization. At nood=4, G=8, this matches GraftOOD's relative online/exemplar loss weight; the comparison is append versus replace, including the resulting rollout allocation. It generates more videos per round (384 vs. 352), yet ends below GraftOOD on HPSv3, VR, and DD (6.96/4.89/36.58, versus 7.32/5.08/44.41). The full comparison, including SFT only (3 epochs) and SFT → FlowAWR (160 further RL steps):