- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SAGE-GRPO combines an integrated exploration-noise variance, timestep gradient equalization, and a dual KL constraint using periodically refreshed and stepwise reference policies to improve reinforcement learning alignment of video generators. Experiments primarily use HunyuanVideo 1.5, with additional reward models, image backbones, and human preference evaluation. The paper interprets these components as stable exploration along the data manifold.
Critical strengths
(1)The work addresses a valuable practical tension between rollout quality, reward reliability, and training stability in video generation. (2) It studies sampling noise, timestep update scales, and reference-policy constraints within a unified, testable training recipe. (3)The evaluation includes sampling baselines, component ablations, different rewards and backbones, and human preference evidence beyond automatic reward scores. The alignment-focused video setting is particularly promising.
Critical weaknesses
(1)The variance correction controls isotropic noise magnitude but does not establish tangent-space exploration. Similarly, refreshing the reference policy does not ensure that successive anchors remain near a valid video distribution. The claimed improvement in reward reliability also lacks direct validation. (2)Under Setting B with Fixed KL, SAGE-GRPO scores 0.2857 versus CPS’s 0.3705. Its strongest result, 0.8066, additionally uses Dual Moving KL, which is not evaluated with competing samplers. Matched comparisons are needed to distinguish sampler improvements from regularization benefits. (3)Dual Moving KL substantially improves Setting B but reduces Overall relative to unregularized SAGE-GRPO in Setting A. Human evaluation covers only Setting B, leaving this trade-off unresolved. The limited video prompt set and lack of reported independent training repeats further constrain claims of broad, sustained stability.
Questions for authors
(1)How do CPS and FlowGRPO perform with the same Dual Moving KL? (2)Does the lower reward in Setting A correspond to better human-perceived quality? What evidence supports improved reward reliability and prevention of cumulative drift beyond training-reward curves?
3 comments
W2 / Q1. Matched Dual Moving KL on the full Setting A/B table
As required, DanceGRPO, FlowGRPO, and CPS are now evaluated with the same gradient equalization, Dual Moving KL, coefficient schedule, and training and tuning budget as SAGE-GRPO. The table below keeps the submitted main results and adds those three Dual Moving KL rows in Setting B. PickScore uses the paper's scale.
| Method | Setting | Configuration | Overall ↑ | VQ ↑ | MQ ↑ | TA ↑ | CLIPScore ↑ | PickScore ↑ |
|---|---|---|---|---|---|---|---|---|
| HunyuanVideo 1.5 | — | Original | 0.0654 | -0.7539 | -0.5870 | 1.4063 | 0.5409 | 19.2322 |
| DanceGRPO | A | No KL | 0.2768 | -0.7589 | -0.3852 | 1.4209 | 0.5386 | 19.1828 |
| DanceGRPO | A | Fixed KL | 0.0979 | -0.8077 | -0.5091 | 1.4147 | 0.5403 | 19.1230 |
| FlowGRPO | A | No KL | 0.2733 | -0.7151 | -0.5286 | 1.5170 | 0.5443 | 19.2244 |
| FlowGRPO | A | Fixed KL | 0.1880 | -0.6771 | -0.5912 | 1.4563 | 0.5431 | 19.2582 |
| CPS | A | No KL | 0.6343 | -0.4855 | -0.4021 | 1.5219 | 0.5479 | 19.2712 |
| CPS | A | Fixed KL | 0.0928 | -0.7156 | -0.5825 | 1.3908 | 0.5479 | 19.1594 |
| SAGE-GRPO | A | No KL | 0.4859 | -0.6104 | -0.4141 | 1.5104 | 0.5423 | 19.1360 |
| SAGE-GRPO | A | Fixed KL | 0.2244 | -0.7438 | -0.5320 | 1.5001 | 0.5446 | 19.1932 |
| SAGE-GRPO | A | Dual Moving KL | 0.2173 | -0.7881 | -0.4249 | 1.4303 | 0.5430 | 19.3752 |
| DanceGRPO | B | No KL | -0.2172 | -0.8854 | -0.6218 | 1.2901 | 0.5439 | 19.1152 |
| DanceGRPO | B | Fixed KL | 0.1290 | -0.7739 | -0.5083 | 1.4112 | 0.5452 | 18.9176 |
| DanceGRPO | B | Dual Moving KL (added) | 0.2668 | -0.6016 | -0.4328 | 1.3012 | 0.5462 | 19.0824 |
| FlowGRPO | B | No KL | 0.4773 | -0.5671 | -0.4731 | 1.5175 | 0.5403 | 19.1074 |
| FlowGRPO | B | Fixed KL | 0.2103 | -0.6654 | -0.5506 | 1.4263 | 0.5427 | 19.2608 |
| FlowGRPO | B | Dual Moving KL (added) | 0.4839 | -0.3022 | -0.1806 | 0.9667 | 0.5468 | 19.2627 |
| CPS | B | No KL | 0.3694 | -0.6650 | -0.5325 | 1.5669 | 0.5479 | 19.0086 |
| CPS | B | Fixed KL | 0.3705 | -0.6121 | -0.4787 | 1.4613 | 0.5458 | 19.1464 |
| CPS | B | Dual Moving KL (added) | 0.3768 | -0.6650 | -0.5325 | 1.5743 | 0.5477 | 19.1807 |
| SAGE-GRPO | B | No KL | -0.1222 | -0.8720 | -0.6046 | 1.3544 | 0.5404 | 19.1282 |
| SAGE-GRPO | B | Fixed KL | 0.2857 | -0.7062 | -0.4425 | 1.4344 | 0.5414 | 19.1802 |
| SAGE-GRPO | B | Dual Moving KL | 0.8066 | -0.4765 | -0.2384 | 1.5216 | 0.5484 | 19.2920 |
The Fixed KL gap cited in the review remains in this table: under Setting B, SAGE-GRPO Overall is 0.2857, below CPS at 0.3705. The added rows place the competing SDEs under the same Dual Moving KL. SAGE-GRPO then obtains Overall 0.8066, compared with 0.4839 for FlowGRPO, 0.3768 for CPS, and 0.2668 for DanceGRPO. FlowGRPO still has higher VQ and MQ, and CPS still has higher TA. The highest SAGE-GRPO Overall is therefore the precise SDE together with Dual Moving KL in Setting B, not the SDE under every KL configuration.
W3 / Q2.1. Setting A user study
The lower Setting A Overall does not mean better human-perceived quality. We ran a user study with the same 29 evaluators and the same 32 prompts as the Setting B study in Table 4. Entries are pairwise win rates of SAGE-GRPO against the named baseline, shown to one decimal percent. A − B is the Setting A rate minus the Setting B rate in the paper.
| SAGE-GRPO vs. | A VQ | A MQ | A SA | B VQ | B MQ | B SA | A − B VQ | A − B MQ | A − B SA |
|---|---|---|---|---|---|---|---|---|---|
| DanceGRPO | 62.9% | 65.0% | 55.9% | 85.9% | 75.8% | 79.2% | −23.0 | −10.8 | −23.3 |
| FlowGRPO | 55.0% | 71.0% | 54.0% | 83.8% | 79.2% | 71.9% | −28.8 | −8.2 | −17.9 |
| CPS | 46.0% | 61.0% | 51.9% | 80.2% | 70.8% | 67.9% | −34.2 | −9.8 | −16.0 |
Setting A is the averaged reward. SAGE-GRPO Overall there falls from 0.4859 without KL to 0.2173 with Dual Moving KL, and it is also below CPS without KL at 0.6343. The human rates do not follow that ranking. SAGE-GRPO is preferred on motion against all three baselines (65.0%, 71.0%, and 61.0%). It is also preferred on semantic alignment, and on visual quality against DanceGRPO and FlowGRPO. The exception is visual quality against CPS, at 46.0%. A lower Setting A metric does not mean that human viewers prefer the baselines; on motion, SAGE-GRPO is preferred.
Setting B should still be the operating point. Its VideoAlign Overall is higher once KL is used, and every human rate above is higher than Setting A, by 8.2 to 34.2 percentage points. Setting B has no Dual Moving KL row for DanceGRPO, FlowGRPO, or CPS in the submitted table, so those three deltas use No KL and Fixed KL only.
| Method | Configuration | Setting A | Setting B | B − A |
|---|---|---|---|---|
| DanceGRPO | No KL | 0.2768 | -0.2172 | −0.4940 |
| DanceGRPO | Fixed KL | 0.0979 | 0.1290 | +0.0311 |
| FlowGRPO | No KL | 0.2733 | 0.4773 | +0.2040 |
| FlowGRPO | Fixed KL | 0.1880 | 0.2103 | +0.0223 |
| CPS | No KL | 0.6343 | 0.3694 | −0.2649 |
| CPS | Fixed KL | 0.0928 | 0.3705 | +0.2777 |
| SAGE-GRPO | No KL | 0.4859 | -0.1222 | −0.6081 |
| SAGE-GRPO | Fixed KL | 0.2244 | 0.2857 | +0.0613 |
| SAGE-GRPO | Dual Moving KL | 0.2173 | 0.8066 | +0.5893 |
Without KL, Setting B is not uniformly higher. With Fixed KL, Setting B is higher for every method. For SAGE-GRPO with Dual Moving KL, Overall rises from 0.2173 in Setting A to 0.8066 in Setting B. Setting B with Dual Moving KL is the operating point we use, and it is the clear gain over the averaged Setting A.
We thank the reviewer for recognizing the practical tension between rollout quality, reward reliability, and training stability. The concerns address tangent-space exploration and cumulative drift (W1/Q2.2), matched Dual Moving KL comparisons (W2/Q1), and the Setting A user study (W3/Q2.1). We respond to each below.
W1 / Q2.2. Additional evidence on exploration and drift
We respectfully clarify that the variance correction integrates a scalar diffusion variance over each interval, and it does not estimate a tangent basis or project noise onto a tangent space. To provide further evidence, we extract the first frame of a real video and obtain its caption using Qwen-VL. With frozen HunyuanVideo 1.5 I2V as the backbone, we apply the SDE of SAGE-GRPO or FlowGRPO for the first N∈{1,5,10,20} steps, followed by deterministic ODE sampling for the remaining 20−N steps. Each entry is the mean over 4 seeds. Dist2Real is the mean frame-aligned CLIP-ViT-L distance to the real video, computed as one minus cosine similarity. LatDrift is the relative latent deviation from the pure-ODE endpoint, ∥z−zODE∥2/∥zODE∥2.
At every tested N, SAGE-GRPO is closer to the real video than FlowGRPO on both measurements. The Dist2Real gaps, FlowGRPO minus SAGE-GRPO, are 0.0081, 0.0057, 0.0106, and 0.0137. The LatDrift gaps are 0.0145, 0.1018, 0.1449, and 0.1288. These distances do not construct a tangent basis.
Figure 7 of the revised paper shows the long-run anchor comparison. With the anchor refreshed, policy–anchor KL stays near its initial value, from 0.0063 to 0.0060 (0.9×) over 532 steps, and the mean training reward improves from −0.48 to −0.09. With the anchor frozen, KL grows from 0.0070 to 0.0448 (6.4×) over 502 steps, while the mean training reward changes from −0.41 to −0.29. This measures drift from the reference anchor beyond the reward curve. It is not an independent human validation of reward reliability. The image evaluation uses 256 prompts across three adjacent checkpoints, which are not independent training seeds. The video evaluation uses 32 prompts.