acceptodds
Under review as a conference paper at ICLR 2027

Manifold-Aware Exploration for Reinforcement Learning in Video Generation

11 views

Abstract

Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Guided Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using VideoAlign and HPSV3 as reward models, and generalize to images on SD3.5-medium and FLUX.1-dev with HPSV3. We observe consistent gains in VQ, MQ, TA, and visual metrics, demonstrating superior performance in reward maximization and overall generation quality. Our code will be released to the public.

Primary area:
reinforcement learning
Keywords:
video generation, reinforcement learning, precise sde, dynamic KL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_RBHNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SAGE-GRPO combines an integrated exploration-noise variance, timestep gradient equalization, and a dual KL constraint using periodically refreshed and stepwise reference policies to improve reinforcement learning alignment of video generators. Experiments primarily use HunyuanVideo 1.5, with additional reward models, image backbones, and human preference evaluation. The paper interprets these components as stable exploration along the data manifold.

Critical strengths

(1)The work addresses a valuable practical tension between rollout quality, reward reliability, and training stability in video generation. (2) It studies sampling noise, timestep update scales, and reference-policy constraints within a unified, testable training recipe. (3)The evaluation includes sampling baselines, component ablations, different rewards and backbones, and human preference evidence beyond automatic reward scores. The alignment-focused video setting is particularly promising.

Critical weaknesses

(1)The variance correction controls isotropic noise magnitude but does not establish tangent-space exploration. Similarly, refreshing the reference policy does not ensure that successive anchors remain near a valid video distribution. The claimed improvement in reward reliability also lacks direct validation. (2)Under Setting B with Fixed KL, SAGE-GRPO scores 0.2857 versus CPS’s 0.3705. Its strongest result, 0.8066, additionally uses Dual Moving KL, which is not evaluated with competing samplers. Matched comparisons are needed to distinguish sampler improvements from regularization benefits. (3)Dual Moving KL substantially improves Setting B but reduces Overall relative to unregularized SAGE-GRPO in Setting A. Human evaluation covers only Setting B, leaving this trade-off unresolved. The limited video prompt set and lack of reported independent training repeats further constrain claims of broad, sustained stability.

Questions for authors

(1)How do CPS and FlowGRPO perform with the same Dual Moving KL? (2)Does the lower reward in Setting A correspond to better human-perceived quality? What evidence supports improved reward reliability and prevention of cumulative drift beyond training-reward curves?

3 comments

We thank the reviewer for recognizing the practical tension between rollout quality, reward reliability, and training stability. The concerns address tangent-space exploration and cumulative drift (W1/Q2.2), matched Dual Moving KL comparisons (W2/Q1), and the Setting A user study (W3/Q2.1). We respond to each below.

W1 / Q2.2. Additional evidence on exploration and drift

We respectfully clarify that the variance correction integrates a scalar diffusion variance over each interval, and it does not estimate a tangent basis or project noise onto a tangent space. To provide further evidence, we extract the first frame of a real video and obtain its caption using Qwen-VL. With frozen HunyuanVideo 1.5 I2V as the backbone, we apply the SDE of SAGE-GRPO or FlowGRPO for the first N∈{1,5,10,20}N\in\{1,5,10,20\} steps, followed by deterministic ODE sampling for the remaining 20−N20-N steps. Each entry is the mean over 4 seeds. Dist2Real is the mean frame-aligned CLIP-ViT-L distance to the real video, computed as one minus cosine similarity. LatDrift is the relative latent deviation from the pure-ODE endpoint, ∥z−zODE∥2/∥zODE∥2\|z-z_{\mathrm{ODE}}\|_2/\|z_{\mathrm{ODE}}\|_2.

NNSAGE-GRPO Dist2Real ↓FlowGRPO Dist2Real ↓SAGE-GRPO LatDrift ↓FlowGRPO LatDrift ↓
10.08930.09740.51980.5343
50.09730.10300.58630.6881
100.09880.10940.52470.6696
200.09050.10420.57510.7039

At every tested NN, SAGE-GRPO is closer to the real video than FlowGRPO on both measurements. The Dist2Real gaps, FlowGRPO minus SAGE-GRPO, are 0.0081, 0.0057, 0.0106, and 0.0137. The LatDrift gaps are 0.0145, 0.1018, 0.1449, and 0.1288. These distances do not construct a tangent basis.

Figure 7 of the revised paper shows the long-run anchor comparison. With the anchor refreshed, policy–anchor KL stays near its initial value, from 0.0063 to 0.0060 (0.9×) over 532 steps, and the mean training reward improves from −0.48 to −0.09. With the anchor frozen, KL grows from 0.0070 to 0.0448 (6.4×) over 502 steps, while the mean training reward changes from −0.41 to −0.29. This measures drift from the reference anchor beyond the reward curve. It is not an independent human validation of reward reliability. The image evaluation uses 256 prompts across three adjacent checkpoints, which are not independent training seeds. The video evaluation uses 32 prompts.

W2 / Q1. Matched Dual Moving KL on the full Setting A/B table

As required, DanceGRPO, FlowGRPO, and CPS are now evaluated with the same gradient equalization, Dual Moving KL, coefficient schedule, and training and tuning budget as SAGE-GRPO. The table below keeps the submitted main results and adds those three Dual Moving KL rows in Setting B. PickScore uses the paper's ×26\times 26 scale.

MethodSettingConfigurationOverall ↑VQ ↑MQ ↑TA ↑CLIPScore ↑PickScore ↑
HunyuanVideo 1.5—Original0.0654-0.7539-0.58701.40630.540919.2322
DanceGRPOANo KL0.2768-0.7589-0.38521.42090.538619.1828
DanceGRPOAFixed KL0.0979-0.8077-0.50911.41470.540319.1230
FlowGRPOANo KL0.2733-0.7151-0.52861.51700.544319.2244
FlowGRPOAFixed KL0.1880-0.6771-0.59121.45630.543119.2582
CPSANo KL0.6343-0.4855-0.40211.52190.547919.2712
CPSAFixed KL0.0928-0.7156-0.58251.39080.547919.1594
SAGE-GRPOANo KL0.4859-0.6104-0.41411.51040.542319.1360
SAGE-GRPOAFixed KL0.2244-0.7438-0.53201.50010.544619.1932
SAGE-GRPOADual Moving KL0.2173-0.7881-0.42491.43030.543019.3752
DanceGRPOBNo KL-0.2172-0.8854-0.62181.29010.543919.1152
DanceGRPOBFixed KL0.1290-0.7739-0.50831.41120.545218.9176
DanceGRPOBDual Moving KL (added)0.2668-0.6016-0.43281.30120.546219.0824
FlowGRPOBNo KL0.4773-0.5671-0.47311.51750.540319.1074
FlowGRPOBFixed KL0.2103-0.6654-0.55061.42630.542719.2608
FlowGRPOBDual Moving KL (added)0.4839-0.3022-0.18060.96670.546819.2627
CPSBNo KL0.3694-0.6650-0.53251.56690.547919.0086
CPSBFixed KL0.3705-0.6121-0.47871.46130.545819.1464
CPSBDual Moving KL (added)0.3768-0.6650-0.53251.57430.547719.1807
SAGE-GRPOBNo KL-0.1222-0.8720-0.60461.35440.540419.1282
SAGE-GRPOBFixed KL0.2857-0.7062-0.44251.43440.541419.1802
SAGE-GRPOBDual Moving KL0.8066-0.4765-0.23841.52160.548419.2920

The Fixed KL gap cited in the review remains in this table: under Setting B, SAGE-GRPO Overall is 0.2857, below CPS at 0.3705. The added rows place the competing SDEs under the same Dual Moving KL. SAGE-GRPO then obtains Overall 0.8066, compared with 0.4839 for FlowGRPO, 0.3768 for CPS, and 0.2668 for DanceGRPO. FlowGRPO still has higher VQ and MQ, and CPS still has higher TA. The highest SAGE-GRPO Overall is therefore the precise SDE together with Dual Moving KL in Setting B, not the SDE under every KL configuration.

W3 / Q2.1. Setting A user study

The lower Setting A Overall does not mean better human-perceived quality. We ran a user study with the same 29 evaluators and the same 32 prompts as the Setting B study in Table 4. Entries are pairwise win rates of SAGE-GRPO against the named baseline, shown to one decimal percent. A − B is the Setting A rate minus the Setting B rate in the paper.

SAGE-GRPO vs.A VQA MQA SAB VQB MQB SAA − B VQA − B MQA − B SA
DanceGRPO62.9%65.0%55.9%85.9%75.8%79.2%−23.0−10.8−23.3
FlowGRPO55.0%71.0%54.0%83.8%79.2%71.9%−28.8−8.2−17.9
CPS46.0%61.0%51.9%80.2%70.8%67.9%−34.2−9.8−16.0

Setting A is the averaged reward. SAGE-GRPO Overall there falls from 0.4859 without KL to 0.2173 with Dual Moving KL, and it is also below CPS without KL at 0.6343. The human rates do not follow that ranking. SAGE-GRPO is preferred on motion against all three baselines (65.0%, 71.0%, and 61.0%). It is also preferred on semantic alignment, and on visual quality against DanceGRPO and FlowGRPO. The exception is visual quality against CPS, at 46.0%. A lower Setting A metric does not mean that human viewers prefer the baselines; on motion, SAGE-GRPO is preferred.

Setting B should still be the operating point. Its VideoAlign Overall is higher once KL is used, and every human rate above is higher than Setting A, by 8.2 to 34.2 percentage points. Setting B has no Dual Moving KL row for DanceGRPO, FlowGRPO, or CPS in the submitted table, so those three deltas use No KL and Fixed KL only.

MethodConfigurationSetting ASetting BB − A
DanceGRPONo KL0.2768-0.2172−0.4940
DanceGRPOFixed KL0.09790.1290+0.0311
FlowGRPONo KL0.27330.4773+0.2040
FlowGRPOFixed KL0.18800.2103+0.0223
CPSNo KL0.63430.3694−0.2649
CPSFixed KL0.09280.3705+0.2777
SAGE-GRPONo KL0.4859-0.1222−0.6081
SAGE-GRPOFixed KL0.22440.2857+0.0613
SAGE-GRPODual Moving KL0.21730.8066+0.5893

Without KL, Setting B is not uniformly higher. With Fixed KL, Setting B is higher for every method. For SAGE-GRPO with Dual Moving KL, Overall rises from 0.2173 in Setting A to 0.8066 in Setting B. Setting B with Dual Moving KL is the operating point we use, and it is the clear gain over the averaged Setting A.

Reviewer_k9X8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SAGE-GRPO for reinforcement learning post-training of flow-based video generation models. The method combines a closed-form computation of the diffusion variance, timestep-dependent gradient reweighting, and a dual KL regularizer involving a periodically refreshed reference policy and the rollout policy. Experiments evaluate these components on video generation under different reward configurations, with additional evaluations across reward models and image generation backbones. The paper reports improvements in its preferred configuration and favorable human preference results.

Critical strengths

(1) Computing the integrated diffusion variance rather than using a first-order approximation provides a clear motivation for reducing excessive noise injection. The analysis identifies a relevant numerical issue in stochastic sampling for flow-based models. (2) The evaluation includes multiple reward configurations, additional reward models and backbones, and a human preference study assessing visual quality, motion quality, and text alignment. These results provide evidence of the practical potential of the complete training recipe.

Critical weaknesses

(1) The interpretation of Dual Moving KL is insufficiently supported. The distinction between a recent rollout policy and a less frequently updated reference provides a reasonable stabilization heuristic. However, periodically copying the current policy into the reference does not establish that the new anchor is closer to the data manifold or represents a more valid distribution. It looks like some ad-hoc tricks without enough justifications. (2) The experiments do not sufficiently isolate the contributions of the proposed components. Although the paper includes component ablations, the main comparisons do not establish whether the gains arise from the new SDE formulation or from the accompanying stabilization mechanisms. For example, under Setting B with fixed KL, SAGE achieves an Overall score of 0.2857, below CPS at 0.3705. Section 4.3 subsequently invokes SAGE’s Dual Moving KL result of 0.8066 in discussing the SDE comparison, changing the regularization configuration. A comparison equipping competing SDE formulations with the same equalizer, dual KL mechanism, and tuning budget is missing. (3) Under the stated interpolation, the score expression in Eq. (7) appears to omit the (1−σt)(1-\sigma_t) factor multiplying the clean-sample estimate. In Eqs. (11) and (13), the Gaussian policy is defined with covariance ΣtI\Sigma_t I, but the KL denominator is written as 2Σt22\Sigma_t^2, rather than 2Σt2\Sigma_t. Furthermore, Eq. (48), presented as a minimization loss, subtracts the positive KL regularizer. These may be presentation errors rather than implementation errors, but they affect central expressions and require correction and reconciliation (4) The adaptive KL update appears inconsistent with its stated control objective. Equation (47) decreases the KL penalty coefficient when the observed KL exceeds the target and increases it when the KL falls below the target. Under the usual interpretation of a positive KL penalty, this weakens regularization precisely when the constraint is exceeded. This direction appears inconsistent with the stated goal of keeping the realized KL near a target value. The manuscript does not provide an adequate explanation for this choice or evidence distinguishing its effect from that of the other stabilization components.

Questions for authors

(1) What evidence supports the claim that periodically refreshed anchors remain valid or move closer to the data manifold? Could the authors directly measure cumulative drift and generation quality over longer training runs? (2) Could the authors compare the SDE formulations under an identical configuration of gradient equalization, Dual Moving KL, coefficient schedules, and training/tuning budgets? (3) Could the authors reconcile Eqs. (7), (11), (13), and (48) with consistent definitions of the clean-sample estimate, variance, and optimization direction? Which expressions were actually used in the reported experiments, and are the discrepancies confined to the manuscript?

4 comments

We thank the reviewer for recognizing the motivation for a precise SDE that directs exploration toward the video manifold, and for the practical potential supported by the broader evaluation. The concerns address the validity of refreshed anchors (W1/Q1), isolation of sampler and regularizer contributions (W2/Q2), equation conventions (W3/Q3), and the role of adaptive KL weighting (W4). We respond to each below, including the additional baseline comparisons.

Q1.1. Evidence of manifold-proximate exploration

Thank you for raising the question that is also mentioned by reviewer RBHN and b1xQ. We add a real-video-conditioned probe to examine how far stochastic exploration moves from a real video and from a deterministic sampling endpoint. We extract the first frame of a real video and obtain its caption using Qwen-VL. With frozen HunyuanVideo 1.5 I2V as the backbone, we apply the SDE of SAGE-GRPO or FlowGRPO for the first N∈{1,5,10,20}N\in\{1,5,10,20\} steps, followed by deterministic ODE sampling for the remaining 20−N20-N steps. This comparison is inference only, and each entry below is the mean over 4 seeds.

We measure two complementary quantities: (i) Dist2Real, the mean frame-aligned CLIP-ViT-L distance to the real video, computed as one minus cosine similarity, and (ii) LatDrift, the relative latent deviation from the pure-ODE endpoint under the same conditioning, ∥z−zODE∥2/∥zODE∥2\|z-z_{\mathrm{ODE}}\|_2/\|z_{\mathrm{ODE}}\|_2. Lower values indicate a smaller departure from the real video and from the deterministic endpoint.

NNSAGE-GRPO Dist2Real ↓FlowGRPO Dist2Real ↓SAGE-GRPO LatDrift ↓FlowGRPO LatDrift ↓
10.08930.09740.51980.5343
50.09730.10300.58630.6881
100.09880.10940.52470.6696
200.09050.10420.57510.7039

At every tested NN, SAGE-GRPO is closer to the real video than FlowGRPO on both measurements. The Dist2Real gaps, FlowGRPO minus SAGE-GRPO, are 0.0081, 0.0057, 0.0106, and 0.0137. The LatDrift gaps are 0.0145, 0.1018, 0.1449, and 0.1288. The ordering does not change as the number of stochastic steps increases from 1 to 20. This supports more localized exploration by the integrated-variance SDE around the real-video reference.

Q1.2 / W1. Anchor refresh and long-run training.

We further evaluate the anchor-refresh strategy over two SAGE-GRPO training runs of more than 500 steps. Both use the same Dual KL setup, with the reference anchor either periodically refreshed or kept frozen.

MeasurementRefreshed anchorFrozen anchor
Policy–anchor KL, initial → final0.0063 → 0.00600.0070 → 0.0448
Fitted KL slope per step ↓1.8×10−61.8\times 10^{-6}7.4×10−57.4\times 10^{-5}
Mean reward, first 100 steps → last 100 steps−0.48 → −0.09−0.41 → −0.29

With periodic refresh, the policy–anchor KL remains close to its initial level, whereas it increases by 6.4× with a frozen anchor. Meanwhile, mean reward improves from −0.48 to −0.09, compared with −0.41 to −0.29 for the frozen-anchor run. The corresponding KL and reward trajectories are shown in Figure 6 of the revised paper.

These results support the practical role of reference refresh in Dual Moving KL: it maintains a recent regularization reference while allowing continued reward improvement, rather than keeping the policy tied to an increasingly distant initial anchor.

W2 / Q2. Isolating sampler and regularizer contributions

As required, we add the fair comparison results for DanceGRPO, FlowGRPO, and CPS with Dual Moving KL, under the same gradient equalization, coefficient schedule, and training and tuning budget as SAGE-GRPO. The SDE is the only component that changes, and the additional experimental results are:

MethodKL configurationOverall ↑VQ ↑MQ ↑TA ↑CLIPScore ↑PickScore ↑
DanceGRPODual Moving KL0.2668-0.6016-0.43281.30120.546219.0824
FlowGRPODual Moving KL0.4839-0.3022-0.18060.96670.546819.2627
CPSDual Moving KL0.3768-0.6650-0.53251.57430.547719.1807
SAGE-GRPODual Moving KL0.8066-0.4765-0.23841.52160.548419.2925

This table places DanceGRPO, FlowGRPO, CPS, and SAGE-GRPO under Dual Moving KL with the same experimental conditions. The precise SDE of SAGE-GRPO attains the best VideoAlign Overall, 0.8066, above FlowGRPO at 0.4839, CPS at 0.3768, and DanceGRPO at 0.2668. It also has the highest CLIPScore and PickScore. The next table separates the two KL transitions. Fixed − No KL and Dual − Fixed are the successive changes in VideoAlign Overall.

SamplerNo KL OverallFixed KL OverallDual Moving KL OverallFixed − No KLDual − Fixed
DanceGRPO-0.21720.12900.2668+0.3462+0.1378
FlowGRPO0.47730.21030.4839−0.2670+0.2736
CPS0.36940.37050.3768+0.0011+0.0063
SAGE-GRPO-0.12220.28570.8066+0.4079+0.5209

We acknowledge SAGE-GRPO needs the full method to reach the best performance. Under Fixed KL, its Overall is 0.2857, below CPS at 0.3705. DanceGRPO and SAGE-GRPO then improve at both transitions: Fixed KL raises Overall by 0.3462 and 0.4079, and Dual Moving KL raises it further by 0.1378 and 0.5209. FlowGRPO drops by 0.2670 under Fixed KL, and Dual Moving KL recovers that loss by 0.2736. CPS changes by only 0.0011 and 0.0063, so its Overall stays within 0.3694–0.3768. CPS is therefore not sensitive to the KL choice.

Based on these additional experiments, we conduct further analysis that the difference follows from the SDE. Let σ′<σ\sigma'<\sigma denote the next noise level and z∼N(0,I)z\sim\mathcal N(0,I). With ϕ=πη/2\phi=\pi\eta/2, CPS updates the latent as

x′=(1−σ′)x^0+σ′cos⁡ϕ ϵ^+σ′sin⁡ϕ z.x'=(1-\sigma')\hat x_0+\sigma'\cos\phi\,\hat\epsilon+\sigma'\sin\phi\,z.

The squared coefficients of the predicted noise and the fresh noise sum to σ′2\sigma'^2, so CPS adds stochasticity inside the scheduled noise budget at each step. This can stabilize rollouts, which fits both its lead under Fixed KL and its narrow Overall range. The same fixed budget also limits how far a KL change can raise the score.

SAGE-GRPO instead integrates the diffusion variance over the denoising interval and uses it to scale both the score correction and the injected noise:

x'=x-(\sigma-\sigma')v_\theta+\tfrac12 V\hat s_\theta+\sqrt{V}\,z,$$ where $\hat s_\theta$ is the score estimate at the current noise level. **This score-corrected update directs exploration toward the video manifold, and Dual Moving KL limits policy drift during training. Their combination reaches the highest VideoAlign Overall, 0.8066, compared with 0.3768 for CPS under the same KL strategy.**

W3 / Q3. Score, variance and optimization conventions

Thank you for pointing out the notation issue. The (1−σt)(1-\sigma_t) factor in Eq. 7 is kept consistent with our baseline FlowGRPO. Eq. 7 corresponds directly to Eq. 9 of FlowGRPO. We derived this factor carefully and checked that the code implements the same expression.

Under the interpolation xt=(1−σt)x0+σtz1x_t=(1-\sigma_t)x_0+\sigma_t z_1, the conditional mean given x0x_0 is (1−σt)x0(1-\sigma_t)x_0. Our scheduler computes the clean estimate x^0=xt−σtvθ(xt,t)\hat x_0=x_t-\sigma_t v_\theta(x_t,t) and uses

sθ(xt)=−xt−(1−σt)x^0σt2=−xt+(1−σt)vθ(xt,t)σt,s_\theta(x_t)=-\frac{x_t-(1-\sigma_t)\hat x_0}{\sigma_t^2}=-\frac{x_t+(1-\sigma_t)v_\theta(x_t,t)}{\sigma_t},

which is the velocity form used in FlowGRPO, and it enters the drift in its Eq. 9. SAGE-GRPO retains this score parameterization and uses the integrated variance Σt\Sigma_t to scale both the score correction and the injected noise. We will align Eq. 7 and Appendix A with the implementation by restoring (1−σt)(1-\sigma_t) and writing the denoising step with the signed increment δσ=σt+1−σt<0\delta\sigma=\sigma_{t+1}-\sigma_t<0.

KL notation and reference strategies. Thank you for pointing out this inconsistency. As defined in the main text, Σt\Sigma_t denotes the integrated transition variance, rather than the standard deviation. For two conditional Gaussian policies with the same covariance ΣtI\Sigma_t I, the KL divergence is ∥μθ−μr∥22/(2Σt)\|\mu_\theta-\mu_r\|_2^2/(2\Sigma_t). The denominators in Eqs. 11 and 13 should therefore be 2Σt2\Sigma_t, or equivalently 2st22s_t^2 when st=Σts_t=\sqrt{\Sigma_t} denotes the standard deviation. We will correct the equations and make the distinction explicit.

To clarify the reference strategies, let π0\pi_0 denote the initial policy, πold\pi_{\mathrm{old}} the policy used to generate the current rollout batch, and πr(k)\pi_{r(k)} the latest periodically refreshed reference. The four regularizers are:

StrategyRegularizerReference
Fixed KLDKL(πθ∥π0)D_{\mathrm{KL}}(\pi_\theta\Vert\pi_0)Frozen initial policy
Stepwise KLDKL(πold∥πθ)D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta)Rollout policy
Moving KLDKL(πθ∥πr(k))D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{r(k)})Periodically refreshed policy
Dual Moving KLβposDKL(πθ∥πr(k))+βvelDKL(πold∥πθ)\beta_{\mathrm{pos}} D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{r(k)})+\beta_{\mathrm{vel}} D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta)Both references

Each term above is evaluated conditionally on the rollout state and averaged over sampled states and latent coordinates in the released implementation. The fixed and moving terms use the transition variance in the denominator, with a small numerical stabilizer. The stepwise term averages log⁡πold−log⁡πθ\log\pi_{\mathrm{old}}-\log\pi_\theta on transitions sampled from πold\pi_{\mathrm{old}}; its expectation therefore corresponds to DKL(πold∥πθ)D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta). We have already clarified this direction in Eq. 15, together with the reference-refresh schedule and normalization in the revision.

Moreover, we acknowledge the sign inconsistency in Eq. 48. We carefully checked the implementation and confirmed that the minimized loss should be L=−Jpolicy+λKLLKL\mathcal L=-J_{\mathrm{policy}}+\lambda_{KL}\mathcal L_{KL}. The implementation adds the weighted KL term to the minimized policy loss. We have corrected Eq. 48 and clarified the optimization convention in the revision.

W4. Adaptive KL weighting

Thank you for asking about the adaptive weighting mechanism. As clarified in W3/Q3, we minimize L=−Jpolicy+λKLLKL\mathcal L=-J_{\mathrm{policy}}+\lambda_{\mathrm{KL}}\mathcal L_{\mathrm{KL}}, where λKL\lambda_{\mathrm{KL}} scales the gradient of the KL penalty. When the mean of the last ten KL observations exceeds the upper threshold, Eq. (47) lowers the coefficient within its prescribed bounds. This reduces the influence of the KL penalty on the update and gives the policy objective more room to improve reward. When the mean falls below the lower threshold, the rule increases the coefficient, strengthening the penalty on further departures from the reference policies. We will clarify in Appendix F that these thresholds govern when to adjust the strength of regularization to balance policy improvement and retention.

Furthermore, figure 10 compares three coefficient schedules within SAGE-GRPO. The fixed 10−510^{-5} and two-stage 10−7→10−510^{-7}\to10^{-5} schedules compare applying the higher weight from the start with introducing it gradually. The 10−7→10−510^{-7}\to10^{-5} and 10−7→10−610^{-7}\to10^{-6} schedules start at the same weight and differ in the strength of later regularization. The 10−7→10−510^{-7}\to10^{-5} schedule yields higher VQ, MQ, and TA rewards in the later part of training than either alternative. This supports our choice of a schedule that starts with weak regularization and increases its strength as training progresses.

Comments on the paper

2 comments

We thank the reviewer for your positive score and recognizing that stable GRPO for video generation is an important practical problem, and for the motivation and ablations on noise correction and gradient balancing. The concerns address the KL sign and adaptive coefficient (W1/Q1), noise magnitude versus manifold-aware exploration (W2), and whether Dual Moving KL also benefits the other samplers (Q2). We respond to each below.

W1 / Q1. KL sign and coefficient update.

We thank the reviewer for identifying the KL sign and the coefficient update. In Eq. 48, We minimize (\mathcal L=-J_{\mathrm{policy}}+\lambda_{\mathrm{KL}}\mathcal L_{\mathrm{KL}}). The reported experiments already use this objective, and we have corrected the sign in Eq. (48). After warm-up, Eq. 47 multiplies (\lambda_{\mathrm{KL}}) by 0.9 when the mean of the last ten KL observations exceeds (1.5D_{\mathrm{target}}), and by 1.1 when it falls below (0.5D_{\mathrm{target}}). The coefficient is clipped to its prescribed range and stays constant between the thresholds. Lowering the coefficient reduces the influence of the reference policies on the current update, while raising it strengthens the penalty on further departures. We will clarify that the thresholds govern the strength of regularization during reward optimization. Figure 10 compares the complete coefficient schedules. The (10^{-7}\to10^{-5}) schedule yields higher VQ, MQ, and TA later in training than either a fixed (10^{-5}) weight or the (10^{-7}\to10^{-6}) schedule.

Moreover, we respectfully declare the coefficient update in Eq. 47, which is also questioned by Reviewer k9X8. At a given coefficient, a larger KL already incurs a larger penalty. When the mean of the last ten KL observations is high, Eq. 47 reduces the coefficient within its prescribed bounds, giving the policy objective more room to improve reward. When recent KL is low, the rule increases the coefficient, making further departures more costly. In the Appendix, Figure 10 compares this schedule with a fixed 10−510^{-5} weight and a milder 10−7→10−610^{-7}\to10^{-6} schedule. The 10−7→10−510^{-7}\to10^{-5} schedule yields higher VQ, MQ, and TA rewards later in training. This is the update used in the experiments. It adjusts the strength of regularization; it is not a solver that forces the KL back to a fixed target.

W2. Noise magnitude versus manifold-aware exploration

We would like to clarify the scope of the manifold-aware framing, which is the same question also raised by Reviewers k9X8 and RBHN. The real-video-conditioned probe below is our answer. We extract the first frame of a real video and obtain its caption using Qwen-VL. With frozen HunyuanVideo 1.5 I2V as the backbone, we apply the SDE of SAGE-GRPO or FlowGRPO for the first N∈{1,5,10,20}N\in\{1,5,10,20\} steps, followed by deterministic ODE sampling for the remaining 20−N20-N steps. Each entry is the mean over 4 seeds. Dist2Real is the mean frame-aligned CLIP-ViT-L distance to the real video, computed as one minus cosine similarity. LatDrift is ∥z−zODE∥2/∥zODE∥2\|z-z_{\mathrm{ODE}}\|_2/\|z_{\mathrm{ODE}}\|_2.

NNSAGE-GRPO Dist2Real ↓FlowGRPO Dist2Real ↓SAGE-GRPO LatDrift ↓FlowGRPO LatDrift ↓
10.08930.09740.51980.5343
50.09730.10300.58630.6881
100.09880.10940.52470.6696
200.09050.10420.57510.7039

At every tested NN, SAGE-GRPO is closer to the real video than FlowGRPO on both measurements. The integrated-variance SDE therefore keeps exploration more local than the first-order noise used by FlowGRPO. These distances do not construct a tangent basis.

Q2. Dual Moving KL across different RL methods.

Relative to Fixed KL, Dual Moving KL raises Overall by 0.1378 for DanceGRPO, 0.2736 for FlowGRPO, 0.0063 for CPS, and 0.5209 for SAGE-GRPO. FlowGRPO mainly recovers the score lost under Fixed KL, while CPS changes little across all three settings. SAGE-GRPO trails CPS under Fixed KL (0.2857 vs. 0.3705) and leads under the shared Dual Moving KL configuration (0.8066 vs. 0.3768). These comparisons show that the benefit of switching to Dual Moving KL varies across samplers, with the largest observed gain and highest Overall score obtained with the SAGE-GRPO SDE.

MethodOverall ↑VQ ↑MQ ↑TA ↑CLIPScore ↑PickScore ↑
DanceGRPO0.2668-0.6016-0.43281.30120.546219.0824
FlowGRPO0.4839-0.3022-0.18060.96670.546819.2627
CPS0.3768-0.6650-0.53251.57430.547719.1807
SAGE-GRPO0.8066-0.4765-0.23841.52160.548419.2925

To separate the Dual Moving KL gain from the SDE, we also report Overall under each KL configuration. Dual − Fixed is the change within the same sampler.

SamplerNo KLFixed KLDual Moving KLFixed − No KLDual − Fixed
DanceGRPO-0.21720.12900.2668+0.3462+0.1378
FlowGRPO0.47730.21030.4839−0.2670+0.2736
CPS0.36940.37050.3768+0.0011+0.0063
SAGE-GRPO-0.12220.28570.8066+0.4079+0.5209

Under this shared configuration, SAGE-GRPO also has the highest CLIPScore and PickScore, while FlowGRPO has higher VQ and MQ and CPS has higher TA. The largest gain and the highest Overall occur when Dual Moving KL is used with the SAGE-GRPO SDE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.